Client download large resources

[dido]

The goal of this task is to allow the user to download large resources from the server on background without exceptions.

• [OPTIONAL]To indicate that the application is currently downloading resources change the book icon inside the resource palette. In case the book is uploading resources use proper icon to indicate both.
• If the resource is not yet downloaded or missing, use the refresh button (missing resource button, that we already have) inside the containing frame. When the user press it, the application should try to download the resource again.
• Implement auto refresh for not yet downloaded ones resources. Every 5sec Sophie should try to download the resources again.
  • Note if the resource is missing don't use auto refresh.
  • Auto refresh should stop after the fifth unsuccessful try

[deyan]

It was not very clear to me about what resources is this task about. After a clarification, is seems fine.

[meddle]

• Yes, the resources of both the upload and the download tasks at this point are Binary ones.

[meddle]

Model updates needed by the download

• ChunkedBinData
  • Added new method public List<String> getChunksInInterval(int begin, int end), which retrieves the chunks in a given interval defined by beginning and the end in the byte content.
  • public List<File> getChunks() -> Caches its files, so only the first time, objects are constructed.
  • public static File retrieveChunksDir() -> Now is public.
• FacadeBinData
  • public byte[browser:] getBytes(int begin, int end) -> It uses the method ChunkedBinData#getChunksInInterval(int begin, int end) to retrieve the chunks containing the wanted bytes.
    • If some of these chunks are not in the local cache, downloads them.
    • If there is some error, notifies the user.
    • If no, returns the bytes from the inner bin data.
  • The other getByte* methods call the above one.
  • public List<File> getChunks() -> Stores all the chunks in the local cache if some of them is absent.
• ResourceFilesUtil -> For all the instances of sophie the temp files are in special instance folder, this preserves two users with different rights to write in the same folder, also deleting after exit is improved.

Server side implementation

• FacadeR4s methods for downloading will be reduced to only one:
  • Response<ImmList<BinData>> getBinChunks(String sessionId, ImmList<String> chunkNames) -> Retrieves the chunks with the wanted chunk names, the size is kept in the model, so method for it is not needed.
• ServerFacade
  • The new method is implemented on the server side here.
  • The algorithm is the following:
    • For all the names
      • If the chunk with the wanted name does not exist, throw ResponseError.
      • If it exist get it and create RawBinData from it, add it into the result list.
        • If there is problem reading the chunk -> throw ResponseError

User Notifications

• BUG : The refresh logic was deleted:
  • FIX : I added it in FrameLogic
• Introduce new interface Expirable, representing something that's meaning or function expire after given amount of time.
  • Methods:
    • void setTimeOut(long timeOut) -> Sets the time out for expiration.
    • long getTimeOut() -> The getter of the time out
  • Time out under zero means 'never expire', and zero means 'expire at this very moment!'
  • Implementors :
    • ExpirableWrappingException -> Something like a base implementation for exceptions.
    • BinSourceNotFoundException -> Now implements it with default time out '-1', because it is caught on too many places and it is bad to put another catch for Expirable at all of them...
• So the FacadeBinData will set the expiration of the BinSourceNotFoundExceptions before rethrow. The time out will be 5 seconds, and after five unsuccessfully tries the exception will be rethrown without timeout modification (Will not expire), if the user tries manual refresh, the five ties will be renewed...
• Palette notifications:
  • They will be shown for background downloads.
  • There will be a DownloadManager that will have registerDownload(ResourceRefR4, ImmList<String>) method that all the FacadeBinDatas will call after download for some forward downloading (which size will be specified by the size of the bin data).
    • When download begins, a DOWNLOAD_UPDATE event will be dispatched, with state DOWNLOAD_STARTED
    • When download ends for ref SUCCESS DOWNLOAD_UPDATE event will be dispatched, and on error ERROR DOWNLOAD_UPDATE
    • The downloads will be grouped by refs and the most recent call of the registerDownload(ResourceRefR4, ImmList<String>) will be the one with priority...
  • The items of the ServerResourcesPalette will have two state properties, downloadState and uploadState.
  • UPLOAD_STATUS enum will be pulled out of the UploadManager and will be renamed to TRANSFER_STATUS. The download states will be added there.
  • The icon of the palette will be computed by the two statuses of the item.
• If a chunk is not present at the server a special ResponseException will be thrown, and the FacadeBinDatas will not refresh themselves automatically (see the analysis)

Additional optimizations (Optional)

• Update the UploadManager and the FacadeR4 in such a way to be possible to upload a list of chunks, this way the nubmer of the requests to the server will be decreased. (From the download I saw that this increases the speed of the transfer).
• The chunks should be stored into Sets (repeating chunks will be removed automatically).
• It is best to have a chunk pool with the uploaded chunks, so already uploaded chunks to not be uploaded again.

Current problems

• There is a strange bug with the media now, when I added the refresh logic again, the media resources want to be refresh initially, even if they are linked from the disk in a local book.
  • I think it is something connected to the change in the MediaFrameView, I made, something connected to the NotAvailableException.
  • May be make the exception put into the error property ExpirableWrappingException with small time out?

Tests

• New tests:
  • ImmTreeListTest -> Tests the immutability of the ImmTreeList.
  • RawBinDataTest -> The RawBinData had no 'equals' and 'hashCode' methods, added them and this test assures that they work properly.
  • DownloadBinDataTest -> Tests the downloading processes.

Code

• Branch : ​svn://sophie2.org/sophie2/branches/private/meddle/download
• In track : branches/private/meddle/download

[pap]

• OK :)

[meddle]

Design changes:

• There will be no DownloadManager, the UploadManager will become TransferManager<D>
  • It will have the new method for downloading <SD> void registerDownload(Object group, D data, TransferFilter<D, SD> filter); Which downloads grouped data. The downloaded data can be filtered with the help of a TransferFilter<D, SD>. This way the method will be able to download not the whole data but a part of it, which helps downloading on portions. The TransferFilter<D, SD> is inner interface of the TransferManager.
    • There is an implementation for the FacadeBinData that filters a list of chunk names for a given data.
• The update of the palette is implemented but it can be seen rarely, because maybe the download is too fast? I don't know.
• The upload is improved in a way that more than one chunk can be uploaded at a time.
• Everything that is related to both download and upload is renamed to 'transfer', check the JavaDoc and var/method names, it is possible something to be missed.

[pap]

• Commited in [9080].
• Created a default page template in author 1 and created few pages using it in author 2. Templates do throw error in server books
• When the resource is already downloaded, the icon is not always refreshed
• Autochaining is really slow:)
• No coding remarks

[meddle]

Closing all the tickets before M Y1