client resources upload

[dido]

The goal of this task is to allow the user to upload books freely to the Sophie server via Sophie 2 Author.

• The upload should not disturb the user work and should be in background. That means that after a user clicked 'save on server' button, he can continue working on the book.
• However we need to make sure that the user will be aware for the success of the upload or for eventual failure. Notification dialog should pop-up after success or failure with description what had happened and close button in it. (Note that the button is "close" not "ok", it is always annoying to click "ok" on error massage).
• [OPTIONAL]In order to notificate the user that somthing is happening and his commands are not ignored, we could do the following:
  • Immediately after "save on server" is clicked - the server tab is opened and inside the "server resources" palette new item is added. It should have icon on the left side (animated gif) that indicates that it's currently uploading.

[meddle]

• The task is more low-level upload logic connected, so this analysis is perfect for it at the current moment.
• The first two points will be implemented for sure.

[meddle]

Write yourself as analysis owner, Dido!

[meddle]

Overview

• The 'upload large resources' task is not a trivial one, all the improvements I made to the large resources in Sophie 2 were because of this task and the next one, the 'download large resources'.
  • So I will work using the test-driven development approach, shaping the classes and the interfaces using tests.
  • Because of that new base test classes will be needed for working with server through the facade.
  • Another thing is that the test servers should listen on alternative ports than the default one to not disturb a working server instance.
  • Another thing for this design is that it will describe more abstract ideas and because of that will be written iteratively and part of it will be added at the implementation phase.
• The task can be divided to three parts:
  • First the most difficult one - Uploading models containing large resources, storing them on the server and converting them for the client.
  • Actual upload of large resources in the background, tricky one using Threading and streams.
  • User notifications -> The most annoying one, but the most important from user point of view...

Improvements on the 'Uploading models' part (#2376)

• Not only BinDatas will be converted but all the holders of binary data like the ImmImage will be converted to contain chunked ones, when uploading.
  • For this there will be new interface ImmReferrer -> it will represent something that refers immutables of a given kind that can be changed into the future.
    • It has two methods:
      • List<I> getReferedImmutables() -> retrieves all of the immutables of type 'I' referred by the implementation.
      • T modifyRefferedImmutables(Map<I, I> replaces) -> modifies the keys of the passed map that are referred in the implementation with their corresponding values.
    • It has two extender interfaces:
      • The old ResourceReferrer, that will continue to work the same.
      • A new one BinDataReferrer that represents referrers to bin datas.
        • The BinData itself, ImmImage, ResourceModel and ResourceImportInfo are the current implementations.
  • FacadeBinDataUtil will now work with the BinDataReferrer instead with BinDatas, it will have special thread with the {{{ResourceModel}}s because for downloading the precise keys pointing to the referrers are needing. For uploading they are not, though.
  • When uploading/downloading everything will work with referrers.
  • ServerBinaryModifier now is simplified to change only the AutoAction fields of the change, not the AutoAction this way the changes will be again model independent.
• The Server Test API is better, there was a research how the Facade works and soon there will be MockFacade for controlling the server and RMI module for controlling clients.
• The Author and Reader fake runners are improved too. The code is similar to the code that Kyli used for faster starting on faster machines... That way the fakes should be faster.

Server Side Uploading

• On the server side the things are simple, there is registered chunks directory already, so every chunk should go there. If the chunk already exists - good, skip it.
• In the FacadeR4 interface will be added new method Response<Void> uploadBinPart(String sessionId, String partName, BinData part); That handles uploading a chunk packet as BinData to be handled by the persistence serialization while going to the server.
• The implementation ServerFacade the method works that way:
  • Gets the chunk dir on the server.
  • Creates a File object to a file with name as the chunk's one in that folder.
    • If the file exists, OK, return.
    • If not, create the file and fill it with the binary data from the chunk.
      • If there is a problem - ResponseException to be handled by the client
      • If not, success.

Client side Uploading

• The FacadeBinData will be able to register itself for uploading when created. This will be a choice, because when such a data is created when there is download there will be no need for that.
• The ChunkedBinData will have 'equals' method and 'hashCode' method, that don't take in mind the transient for the running Sophie instance chunks dir. The FacadeBinDatas 'equals' and 'hashCode' will call the one of their chunk datas.
  • These changes will be used when uploading to not upload the same data twice.
• UploadManager<D> interface will be introduced for uploading data.
  • It has one method void registerUpload(D data), that register some data for uploading to remote place.
  • There will be a extension point for such manager in the base.connectivity module.
  • They will be retrieved from the point by their data type. So the interface may need getDataType method.
  • For now there will be one implementation -> BinaryUploadManager, working with FacadeBinDatas
    • This manager will have one thread (daemon) for every server session that will know which datas are uploaded and which not and will not upload the same data again.
    • This threads will queue datas and chunks and will upload them one by one, when everything is uploaded they will sleep for more time.
    • If something is not uploaded right (there was ResponseException) there will some tries for that thing to be uploaded and if there is problems after that some alert.
• The facaDeBinDatas before uploading will register themselves in the BinaryUploadManager registered in the current instance of Sophie 2

User Notifications

• The UploadManagers will dispatch events when they succeed in uploading everything or there was some error
• There will be logic in main.ws_connector that will handle such events and will show to the user dialogs with the specified by dido messages.
• For the download task the error manipulation views will be able to have time out to try again to refresh themselves so if the bin data is retrieved from the server the user should be able to see the content.
• (Optional for this task) The users should be able to click 'try again' when there was an error uploading.

Tests

• Checks and little fixes:
  • All the tests in org.sophie2.base.commons module work.
  • All the tests in org.sophie2.base.connectivity module work.
  • All the tests and demos in org.sophie2.base.layout module work.
  • TemplatingTest -> Something from the trunk has broke it, so may be the 'Page templates' task should fix it.
  • All the tests in org.sophie2.base.persistence module work.
  • Most of test/demos org.sophie2.dev are trash... May be we need task for cleaning them out...
  • In the org.sophie2.extra.func.annotations module, deleted the commented tests, in the future the annotations will be reimplemented...
  • Fixed the NativeBrowserBridgeTest, but the browserr demo is completely out of date...
  • Fixed HttpServiceTest.
• New tests:
  • UploadBinDataTest -> tests the uploading process.
  • The FacadeBinDataTest works with Binary referrers now and tests the new logic.
  • New persistence was implemented for the ImmImages so there is test for it ImmImagePersisterTest that tests it, the last of the test methods will fail until the immutable collections and their persisters are placed in the wrong modules, but in the author/server/reader the logic works well. The ImmImage is persisted corresponding to what binary data it contains, there is backward compatibility.

Design Related Code

• Branch : ​svn://sophie2.org/sophie2/branches/private/meddle/upload2
• In track : branches/private/meddle/upload2

[pap]

• I pass it.
• In UploadState I think START_UPLOAD should better be named UPLOAD_STARTED or UPLOAD_STARTING
• As these UploadManagers are parametrized by two arguments it may be better to mention both of them in the name. Or as you proposed make them not generic on the group.

[meddle]

• The Group is not generic parameter anymore, it is Object.
• I will rename it to UPLOAD_STARTED.
• In the implementation phase I will provide more User related changes and developments I did on the design.

[meddle]

UploadManager changes

• The UploadManager<D> interface has the following changes:
  • The manager has no generic type of for the group.
  • There are two new methods:
    • void initialize(Object initContext) and boolean isInitialized() which are used for initializing some context to be dispatched with the events dispatched from the implementations. For example visuals.
  • The EventIds of the events dispatched by default by the UploadManagers are here; 'UPLOAD_UPDATE' is the only one for now.
  • The enum UploadState is here too, maybe with the 'download' task the states will be pulled out and will be renamed, will have 'DOWNLOAD' state... Or may be there will be another enum for the download state (The user will have 'upload and download' icon).
  • There is a cache for the registered managers to not go over the extensions every time. It is UploadManagerCache and has method public static <UM extends UploadManager<D>, D> UM getUploadManager(Class<D> clazz) for retrieving cached managers.
• BinaryUploadManager is more client based one so it is moved to org.sophie2.main.func.servers -> org.sophie2.main.func.servers.resources package.

User Notifications

• UploadDownloadLogic -> The logic that handles the events for upload and download. For now has two handlers:
  • ON_UPLOAD_STATE_UPDATE -> Handles updates of the upload state of a resource. Updates the states of the ServerResourcesPalette items (the icons of the items are changes corresponding to the given state). If the state is error, a dialog is shown (talked with dido about that)
  • ON_ERROR_REFRESH -> Tries to upload a resource that had upload error. The event is dispathed by the items of the ServerResourcesPalette
• The ServerResourcesPalette is updated its items are cached and have state (UploadState for now).
  • There is a new renderer that renders the items with their state. It works the following way:

    	if (value instanceof Component) {
    		return (Component) value;
    	}
    			
    	super.getListCellRendererComponent(list, value, index, isSelected, cellHasFocus);
    			
    	if (value instanceof ServerResourceItem) {
    		ServerResourceItem item = (ServerResourceItem) value;
    		
    		ResourceRefR4 ref = item.resourceRef().get();
    
    		setText(ref.getLocation());
    
    		UploadState state = item.state().get();
    		if (state == UploadState.ERROR) {
    			String toolTip = SkinUtil.getSkinValue(ServerResourcesPalette.class,
    					ERROR_TOOL_TIP_ID, String.class);
    			setToolTipText(toolTip);
    		} else {
    			setToolTipText(list.getToolTipText());
    		}
    		
    		ImmImage icon = getStateIcon(state);
    
    		if (icon != null) {
    			setIcon(icon.toIcon());
    		}
    	}
    			
    	return this;
    
    

  • Every icon and tooltip is retrieved through the skin.
  • The items are made cleverly (even if the resource is not rendered in the palette its item has the right state... uploading a whole book is example)
• Fixed bug -> The ServerResourcesPalette is updated when a new book is uploaded. Just called the updateResources method...
• When connecting for the first time the UploadManager is initialized.

Tests

• Added base test for immutability ImmutabilityTestBase
• The ImmImageTest -> updated to be ImmutabilityTestBase
• ChunkedBinDataTest -> To test the ChunkedBinData

Code

• Branch : ​svn://sophie2.org/sophie2/branches/private/meddle/upload2
• In track : branches/private/meddle/upload2

[pap]

• Commited in [9031]

[meddle]

Closing all the tickets before M Y1