Error: Macro BackLinksMenu(None) failed

compressed data is corrupt

Error: Macro TicketQuery(summary=TEXT_LAYOUT_LAZY, format=table, col=summary|owner|status|type|component|priority|effort|importance, rows=description|analysis_owners|analysis_reviewers|analysis_score|design_owners|design_reviewers|design_score|implementation_owners|implementation_reviewers|implementation_score|test_owners|test_reviewers|test_score|) failed

current transaction is aborted, commands ignored until end of transaction block

Analysis

Overview

The current text layout mechanism re-flows the text every time the content or the areas are changed. This is quite ineffective for large chains, so it causes performance problems.

Task requirements

• Make the layout as lazy as possible:
  • When something is changed, all the area layouts before the modified one should be reused (it should currently be that way, just ensure and test that everything is working)
  • When something is changed, all the area layouts after the last visible one could wait.
  • Laziness should not break the auto-chaining.
• Provide basics for segment layout caching. Caching should not necessarily work after this task is completed, since the text hash function may not be correct. However, make sure that if the text is OK, the layout segments will be cached.

Task result

Better text layout performance.

Implementation idea

• Introduce hot layout status and define it. Possible statuses may be valid, invalid, partially valid.
• Introduce area layout status - valid or invalid.
• Create a background thread, which validates the layout area by area when the user is not working.

Related

None.

How to demo

Measure the layout performance in different scenarios with and without lazy approach (this could auto-tested).

Design

First of all, create class VisibleArea - composition of ImmArea and visibility flag. It will be immutable, with only one constructor and getters for the area and the visibility value. It will be used for the layout purposes, to it should be in that package.

Split the anonymous SceneTextView from the HeadTextFrameView, since the class has become too large.

In the SceneTextView's TextViewFlow, except the method getAreas(), add getVisibleAreas(). It will get the areas from the chain and the visible() value of the corresponding frame, and construct a VisibleArea from both. Then the getAreas() method will use getVisibleAreas() to create a list of ImmArea-s.

In RootPageView, override the computeVisible() method so that it returns true when the view is current in its parent book view. According to the default implementation of BaseVisualElement.visible(), this will cause a frame view to be visible() only when its page is current. Note, that the BaseVisualElement's visibility is not the same as the frame model's visibility channel.

• As a result, the getVisibleAreas() will collect information about available areas for a chain as well as information about the current areas - the ones that the user works with.

In HotAreaLayout, put a flag valid and create a constant HotAreaLayout.INVALID. It represents a layout with empty text and empty area.

Introduce HotLayoutStatus. This way the logic will know how to handle the current layout. A HotLayoutStatus is defined for a concrete HotTextLayout and given ImmHotText and a list of VisibleArea-s as follows:

• INVALID, if any of these is true:
  • The given text is NOT the same as the layout's;
  • The ImmArea-s from the given list are not the same as the ImmAreas used for the layout;
  • Not all the area layouts before the last visible area in the HotTextLayout are HotAreaLayout.isValid().

• VALID, if:
  • The status is not INVALID, and
  • All the area layouts in the HotTextLayout are HotAreaLayout.isValid().

• PARTIALLY_VALID, if:
  • The status is not INVALID,
  • The status is not VALID.

In HotTextLayout, create method update(ImmHotText, ImmList<VisibleArea>, int). It returns a new HotTextLayout and its body is almost the same as the static create(..) method, except that it does not take argument prevLayout - the previous layout is the layout itself. This lets us remove some null-checks and other if-s.

The last argument in the update() is a lazy factor. It indicates how lazy the algorithm should be. If you pass 0, it will be as lazy as possible. If out pass Integer.MAX_INT, it should work as the current layout. I think this factor will be useful for future testing and for dry-runs.

The main difference between the 2 algorithms is, that the update method creates/reuses area layouts only to the last visible area index + the lazy factor. The rest layouts are HotAreaLayout.INVALID. As a result, this method will generate:

• PARTIALLY_VALID layout (according the the given text and areas), if (last visible index + lazy factor) < The area list's size;
• VALID layout (according the the given text and areas), otherwise.

After this, the create() method will look like HotTextLayout.EMPTY.update(..).

Create method validateNextArea() in HotTextLayout, which returns a new HotTextLayout with just one more valid area. Everything else will be re-used, so this method is as quick as HotAreaLayout permits.

So, now we have to define who and how should treat the current HotLayout. There are generally 2 different approaches for the layout : The auto-chaining's and the TextFlow's.

• Auto chaining. The common between the auto-chaining and the layout is only the dryRun() method - the one which tells how many areas to add/remove from the current model. Its logic will be like this:
  • If the current layout's status is not VALID according to the current text and areas, does nothing.
  • Otherwise, acts as it currently does.

• Text flow. This is about the textLayout() property in the text flow. It will:
  • If the last layout is VALID with the current text and areas, does nothing.
  • If the last layout is PARTIALLY_VALID with the current text and areas, does nothing.
  • Otherwise, calls update() to the last layout.

Obviously, in general, the text flow makes INVALID layouts PARTIALLY_VALID ones, and the auto-chaining invalidates VALID ones. So, we need someone to completely validate PARTIALLY_VALID layouts. Here comes the

• Background thread. This should be a daemon thread, which runs with lowest priority and the only thing it does is
  • If the current layout is PARTIALLY_VALID, calls its validateNextArea() method with the current text and areas. The result is saved to a RwProp backgroundLayout() in the text flow.

This result should somehow be used, so we add a new rule:

• The text flow tracks the background layout and if it is not INVALID with the current text and areas, uses it instead of the last computed layout.

After a number of iterations, the background thread will completely validate the layout, which means that the auto-chaining will still work, it will just be delayed in time.

There is a problem with the HotLayout.dryRun() method - it can only return non-negative values or -1. The bad effect here is that the autochaining can remove only 1 area at a time, then everything must be revalidated. So, if you have a chain with 300 frames, select all the text and delete it, you will wait several tens of minutes before all the areas are removed. It can easily be fixed with little modification of the algorithm. So, do it :)

Another issue appears with the search - it cannot search in invalid areas, since they have no consumed text. The only solution I can propose is to force everyone that is interested in the consumed text to validate the layout first. (Since the layout is immutable, it cannot validate itself. If we make it mutable, we will have troubles with repainting, I guess).

The determination of the visible areas appeared to be a slow operation in large books. A possible optimization could be to hold the list of areas in auto-prop (HeadSceneTextView.visibleAreas() ), not in method.

Segment layout caching was inspected, and the results are:

• Caching the styled text throws exceptions with the HotLineLayout.getPosPlace() method - the text is visually the same, but does not contain the same positions. Make method getPosPlace(int index) in HotSegmentLayout, so that the position will be turned into real index before calling this method.
• Caching cannot currently work, because getStyledHash() in ImmHotText does not work correctly. It will be fixed with TEXT_MODEL_INTERNAL task. Until then, the written lines about segment caching could stay commented.

Unit tests:

• HotLayout.getStatus() is tested in HotLayoutTest.
• BackgroundThread is tested in BackgroundThreadTest.

Implementation

• The task is implemented according the design in branches/private/kyli/2331.

• The problem with the broken search still persists, I hope it will be fixed in near future. This task is intended to be integrated in out private branch for now, so the broken search will not be a problem until our merge date. I take the responsibility not to forget about this issue until then.

• The PagePreviewPalette has some problems with repainting. However, this does not concern the text layout, nor the text layout is the reason for this behavior. The problem is that when an area layout is recomputed, it is not redrawn. I doubt this is an issue, since we do not guarantee the PPP will display the exact content of a page.

• The background thread only calls a HotLayout's method. The only performance problems it could have (in my opinion) is, that it could iterate infinitely. Therefore I created a test (linked in the design section), which asserts the thread does not deliver unneeded results. When no work is required, the thread waits blocked. On the other side, every head text frame has exactly one background thread (not like in the auto-chaining), Which I think is OK.

• The auto-chain invocation and background-thread one have been put in a ResourceProperty, I think this makes the design more clear and it is now visible when and why these thread work.

• Just a note, please do not forget that this ticket should be integrated in the ​svn://sophie2.org/sophie2/branches/private/kyli/sophie2-text branch.

Testing

^{(Place the testing results here.)}

Comments

^{(Write comments for this or later revisions here.)}