large resources exception

[dido]

At this point exception is thrown when the user link a large resources inside book, that have been deployed on Sophie server. The goal for this task is to mute the exception.

[meddle]

• The true goal of the task is to implement the new model of the BinDatas stored on the server
• With this model the exceptions will cease to be thrown.
• Additional thing is to add a Server Testing API for unit tests through the HTTP Facade.
• Fix the exception when inserting frames on a Server book.

[meddle]

Overview

• The 'upload large resources' task is not a trivial one, all the improvements I made to the large resources in Sophie 2 were because of this task and the next one, the 'download large resources'.
  • So I will work using the test-driven development approach, shaping the classes and the interfaces using tests.
  • Because of that new base test classes will be needed for working with server through the facade.
  • Another thing is that the test servers should listen on alternative ports than the default one to not disturb a working server instance.
  • Another thing for this design is that it will describe more abstract ideas and because of that will be written iteratively and part of it will be added at the implementation phase.
• The task can be divided to three parts:
  • First the most difficult one - Uploading models containing large resources, storing them on the server and converting them for the client.
  • Actual upload of large resources in the background, tricky one using Threading and streams.
  • User notifications -> The most annoying one, but the most important from user point of view...

Uploading Large Resources And Exceptions

• In this ticket I'll implement the first task of the three above:
  • If the model of the uploaded/downloaded datas is updated, the exceptions will cease to be thrown.
  • The client uploading will be able to work with the binary resources, and the clients downloading will have 'resource not found frames'.
  • The server will contain all the binary datas in one format, there will be no embedded or stored in files ones.
• The datas for the server will be kept in chunks, that will be cacheable on the clients and easy to find/transfer from the server.
  • When client uploads a binary resource, its bin data will be uploaded in the following way:
    • It will be divided into chunks and stored in a temp 'chunk_cache' folder.
    • The chunks will be named with the MD5 sum of their content (unique for the content).
    • The list of chunks and the size of the whole data will be uploaded to the server.
    • The server will be transferring to the clients this list of chunk names and size of the datas.
    • When asking for bytes, the client firstly will seek in the chunks cache, and then ask the server.
      • If the data is not in the chunk cache and there is no chunks on the server/no connection there will be no exceptions, it will behave as the current FileDatas.
• Classes and Interfaces, introduced for the model:
  • ChunkedDataHolder -> Holder of chunked bin datas, its content can be persisted to be transfered to the server.
    • Methods:
      • ImmList<String> getChunksList() -> Retrieves the MD5 names of the chunks (persistable data).
      • int getChunksCount() -> Retrieves the number of the chunks.
      • String getChunksDir() -> Gets the current chunk directory for the instance of Sophie 2 running.
      • ChunkedBinData getChunkedBinData() -> Gets the hold bin data. Used for the persistence mostly and for wrapping chunked data.
    • Implementors:
      • ChunkedBinData -> The base implementation, it uses FileBinDatas for access to the chunks, so the exceptions, when the chunk is not fount is automatically handled by the frames.
        • Extends BinData
        • Persistable immutable.
        • Factory methods:
          • public static ChunkedBinData create(File file) -> Creates the data from a File, splitting it into chunks in the chunks dir of the currently running instance of Sophie 2.
          • public static ChunkedBinData create(BinData data) -> Creates the bin data from another bin data.
          • public static ChunkedBinData create(ImmList<String> chunksList, int size) -> Creates the data from a list of chunk names and size of data, used by the persistence for loading.
        • public ChunkedBinData getChunkedBinData() -> For this implementation simply returns 'this'.
      • RemoteBinData -> A wrapper to a chunked bin data stored on a remote place, containing all the necessary information and tools to connect when a chunk that is not locally stored is wanted.
        • It is a BinData
        • Factory methods:
          • public static RemoteBinData create(FacadeR4 facade, String sessionId, ResourceRefR4 location, Key<BinData> key, BinData originalData) -> Creates the data with facade to connect to, session id for the current session, location of the remote resource containing the wrapped data, and the actual key holding it.
        • It is meant to be used only on the clients
        • It is persisted to a normal ChunkedBinData
        • The public ChunkedBinData getChunkedBinData() returns the wrapped data.
• Persistence:
  • In the ResourceDataPersistType there is a new enum constant 'CHUNKED'. A data is chunked if it is implementation of ChunkedDataHolder
  • In the ResourceFilesUtil, there is a new check if the data to persist is 'CHUNKED'.
    • If it is chunked, the ChunkedDataHolderPersister is called, which stores the hold data as described above, and loads alway ChunkedBinData, which can be transformed/wrapped later.
• Transferring:
  • When uploading a resource all the bin datas in it are wrapped in RemoteBinDatas (and in the remote bin datas are transformed into ChunkBinDatas), and when downloaded again are transformed to remote ones.
  • RemoteBinDataUtil -> The util which wraps the BinDatas into RemoteBinDatas. It can transform all the datas contained into a ResourceModel.
    • public static RemoteBinData convertBinDataToRemote(ResourceRefR4 location, FacadeR4 facade, String sessionId, Key<BinData> key, BinData data) -> Converts a single bin data to remote one.
    • public static ResourceModel convertBinDatasToRemote(FacadeR4 facade, ResourceRefR4 remoteLocation, String sessionId, ResourceModel original) -> Converts the binary datas in a whole model to remote ones.
    • Used when uploading books, when executing changes to remote resource, when extracting resources from the facade.
  • When executing changes to remote resources, they must be updated to contain writes only with chunked bin datas:
    • This is possible if when registering a change to a ResourceModel we modify the writes and reads (The ChangeEffect) in special way, and then convert the Change itself.
    • So ResourceRevision's applyChange method signature will change. It will now take a special modifier of changes.
      • There will be a method applyChange with the old signature, that will pass the default modifier, that saves the old effect and change.
    • ChangeEffectModifier -> Used to modify change effect and change in a special way. Interface.
      • ChangeEffect modifyEffect(ChangeEffect effect) -> Modifies a change effect.
      • Change modifyChange(Change original, Map<Key<?>, Object> extractedWrites) -> Updates a change to have the modified writes of the effect get from the {modifyEffect(ChangeEffect)} method.
      • Default implementation that preservers the effect and the change -> public static final ChangeEffectModifier DEFAULT_MODIFIER
      • ServerBinaryModifier -> Modifies the changes to the server to have only RemoteBinDatas.
        • ChangeEffect modifyEffect(ChangeEffect effect) -> uses the RemoteBinDataUtil to convert all the writes containing binary datas to remote.
        • Change modifyChange(Change original, Map<Key<?>, Object> extractedWrites) -> If the Change is ModelChange and contains writes with binary datas, creates a new one with a special AutoAction that applies all the writes of the modified change effect from the ChangeEffect modifyEffect(ChangeEffect effect) method.

Directories for chunks

• On the clients these directories will be temporary, and on the server not
• The directories on the different instances will be different
• So there will be extensions with the directories, that will be registered with the Serve, Author and Reader modules.
• There will be a extension point for them in the base.commons module.
• The ChunkedBinDatas will work with these extensions when getting their chunk folders for search/store.
• The fake starters will start the Author/Reader modules and will use the AppMainWindow created by these modules to add test books.

Uploading Frames and exceptions in the trunk

• When the last 'Large resources' task was integrated into the trunk, the ResourceImportInfo class was updated to have import type enumeration for linking resource models.
• This enumeration has to be persisted with the info, but there is no enum persister registered for it, so I registered one in the main.resources module.
• When the info was persisted the value ref, containing this enum was set wrongly to the model data, itself, fixed that too.

Server testing API

• There is a module for testing with a mock server
• In it there is a real mock server runner that can be run freely, it is lighter than the full-fledged one.
• MockServerMain -> A normal fake runner for development.
  • The main method can take additional names of modules to start.
• MockServerModule -> Registers a chunk directory for testing with the server.
• ServerModelTestBase -> The base test for all the special server tests. It starts a mock server in a different JVM and passes to it some additional modules, given by the developer.
  • Has some helper methods for the developers.
  • There are problems some times killing the server process, I don't know why :/

Tests

• RemoteBinDataTest -> Tests uploading/downloading resources with bin data to the server. Test the RemoteBinDataUtil as well.
• ChunkedBinDataTest -> Tests working with chunked datas.
• ChunkedBinDataPersisterTest -> Tests persisting ChunkedBinDatas
• Fixed all the tests in the base.resources module to run.
• There can be some tests that I forgot to add :)

Problems

• Some things that need to be converted are not just bin datas, and holders of bin datas (Only the ImmImage for now). So in the next task the convert methods will work with BinDataRefferer, something like the ResourceReferrer but for bin datas. I can do that for that task, but maybe it will take time, so I suggest it to be for the next one. There are no exceptions with the ImmImage when uploading it, don't know why though :)
• The embedded resources are uploaded to chunks too, so when downloaded they will be empty as well, till the next task is finished, if you want the embedded resources will be not converted in the end of this task. (The RemoteBinDataTest will fail in the end of this task if we do that...)
• The Fake reader, don't use the ReaderModule, I'll fix that.

[pap]

• It is nice to mention what "large resource" means. It is an important definition as this term is used alot.
• In the JavaDoc of ChunkDataHolder.getChunksList please correct the word "hold" to "held". Other methods in the same interface have the same problem.
• ChunkedDataHolder hierachy is a bit confusing ... at first glance.
• getChunksDir() - why is this method needed in the interface.
• The way you determine the deletOnExit in ChunkedBinDate.create(BinData) is not nice as it somehow breaks the idea of the directory providers. Maybe those providers should tell whether we should delete on exit.
• I think it is better to put the three create methods one after another.
• RemoteBinData sounds a bit too general. This one uses our facade. But there might be a RemoteBinData that would use another communication protocol.
• Wouldn't it be better if the instanceof check was done in the ChunkedBinData create method instead of the constructor of RemoteBinData.
• Possible problems with the Changes modifications. Mira told you about them.
• As this is to be integrated into the trunk I think it is better not to degrade in user experience and so not to make embedded resources to chunks.

[pap]

• OK
• Commited in [8940].

[meddle]

No comments :(