Timelines

This document describes the concept of Timelines in Sophie2. Here you will find a description of what timelines, channels and timeline entries are, how are timelines and links working together, what are the most popular uses of timelines and other useful information related to this key part of the project.

Design

Here you will find a detailed description of the internal and external design of timelines and some draft diagrams of the user interface. This is based on the discussion about timelines held on 2009-04-24 (see the discussion log). Timelines in Sophie 2.0 have a lot of differences from the timelines in Sophie 1. They are meant to be simple and powerful at the same time.

User interactions

A book in Sophie2 cannot have timelines than span all over it. Instead, each page of the book has its own timeline (that can be only one). Each group of objects on the page can also have its own single timeline. This separation ensures that there won't be situations when the timeline says the user is simultaneously on more than one page and prevents having actions involving a frame that is not on the current page. It is a substitute to the single/multiple page timelines in Sophie 1.0. The timelines of the grouped objects provide more interactive content. For example, a movie playing or some objects moving while a table of contents is shown is now possible.

Each timeline visually has three parts - intro, main part and outro. The intro and outro cannot be looped, while the main part can. By default both the intro and outro are 0 seconds long, while the main part is infinitely looped with all objects on the page shown (or hidden if the author has provided so). This will ensure that if an author wants a simple book with no interactions and animations, he doesn't have to bother with timelines. On the other hand, it is robust enough for the experienced. The intro and outro allow fade in/out effects to be created not only for the page, but for grouped elements as well.

There are two areas of timeline channels. The first one contains group channels - one for each group of elements on the page. This channel is specialised for this group and can contain only entries holding this group. This will prevent constructing invalid sets of actions by overlapping the same object on several channels. Still, it will provide flexibility to work with multiple channels and objects easily. The second type of channeles is the so called "event channels" . They do not hold an object or group that is on the page. Instead, they can hold an audio resource or eventually some specific events (for example opening a book in a new window). There can be several such channels, similarly to what Sophie 1 allows.

When a page is entered, its timeline starts playing automatically, showing the intro, main part and outro. The user will be able to skip the intro and outro. If the main part is not looped, the next page will be shown automatically after the timeline of the current stops playing. If it is looped, it will show continuously until the user (or a link action triggered by the timeline) requests to go to another page. In such case, the outro is shown. Timeline playback cannot be controlled by links. The user, however, will be able to pause the timeline and change the position of the head in the bottom flap timelines panel.

As a start, timelines will be able to control the visibility of page elements and the state of the media. Additional actions might be supported at a later stage. The current LinkActions for Hide/Show/Toggle Frame will be used.

This concept of timelines allows for easy implementation of the most popular of their uses - a slide show of photos with audio annotations and a table of contents or dvd-like menu. The design should be still flexible enough to allow plenty of other uses of timelines.

Internal design

Timelines, channels and timeline entries will be pro-objects instead of resources (similarly to links, linк actions, triggers and rules). Each page will keep references to its timeline. The same applies for each group of page elements. For each of these groups, a channel will be added to the timeline.

Each timeline will keep a list of channels and a reference to its parent, i.e. the page it controls. Timelines will have methods for playing, stopping, jumping to a location, adding and removing of channels and entries, etc. Timelines cannot be created or deleted by the user. They are automatically created for each page and group of page elements.

Each channel will contain a reference to the group of elements it holds and a list of timeline entries. Group channels are added automatically and cannot be created/deleted by the user. This does not apply to channels for audio and misc actions.

There will be two types of entries. Each entry will have a start time and a duration. Group channel entries will play the timeline of the corresponding element. If that timeline's length is finite, the duration of the entry cannot exceed it. Entries for the event channels will perform the action specified instead of playing the timeline of the element.

Edit

Discussion log

This is an overview of what the design discusson about timelines on 2009-04-24 was about.

• Issues to consider:
  • Are timelines a resource?
  • Having a timeline for the book vs. having a timeline for each page.
  • Single page/multiple page timelines in Sophie 1.0 - should we keep them?
  • Frame behaviour when dragged on timelines - it seems strange that a frame is visible all the time when not on the timeline and only for a short time when dragged to a channel.
  • Overlapping actions with one object on multiple channels.
  • Triggering a timeline from another timeline, playing several timelines simultaneously.
  • Timelines triggering links, links triggering timelines.
  • User interactions while a timeline is playing.
• Timelines concept
  • Each page in the book has its own timeline, the book itself doesn't have a timeline.
  • Each top-level group in the page can have its own timeline.
  • There are no different kinds of timelines (single page/multiple page).
  • A timeline consists of three parts - intro, main and outro. The main part can be looped.
  • There are two sections in a timeline - channels for groups and channels for media and misc events (e.g. audio background).
    • For each group in the page, a channel is created. It can contain only this group in its timeline entries.
    • The user can drag an audio resource to an area below the group channels. Other events can be drawn here as well.
  • User interactions are with higher priority than timeline-triggered events. If a user triggers a link, it is executed no matter whether it conflicts with the actions in the timeline.
  • Timelines play automatically but the user can pause them or change the position of the head:
    • When a page is shown, its intro is played, followed by the main part.
    • If the main part is not looped, the outro is shown after it and the next page is shown.
    • If the main part is looped, the outro is shown when the user requests a page change.
  • Links cannot trigger timelines. The opposite is possible (and is critical to the correct behaviour of timelines).
  • Timelines will be pro-objects.
• Use cases
  • A slideshow of pictures with audio annotations (the most popular use of timelines in Sophie 1).
  • Fade-in/fade-out effects for a page.
  • DVD menu, table of contents.

Sophie 1 Issues

Here you will find Steve's comments, based on his previous experience with timelines in Sophie 1. The new Sophie 2 timelines design eliminates most of the issues mentioned below.

Overview

Timelines allow Sophie objects to be animated over time. Examples of Sophie objects include Sophie frames, audio files and pages. Example frame content include text, images and video frames.

Timelines can control the visibility of a frame, or if a movie is playing or not. The author can use Timelines to control a movie that is on a template spanning pages of a book, and also to change the current page of the book as the movie plays.

Referencing Objects

Sophie 1 timeline events kept hard references to frames and audio. This meant that if a timeline that spanned 1000 pages were to be loaded, every page and thus every object on those pages were loaded. This could cause severe memory issues. Sophie 2 should use a lazy approach to referencing objects, maybe through the ResourceRef class.

Timeline Scope

There are two types of timelines in Sophie 1, normal and page timelines. The difference is that a page timeline had a page channel. A page channel was a special channel that could only contain page events for changing the current page.

A normal timeline would only reference objects on one page, or should have but Sophie never enforced this. It was very easy to create a broken timeline. In a large project, an Author should be able to attach a timeline to a particular page or template and thus, only see those timelines when filtered. This will help in timeline management on huge books.

UI Issues

Boyan mentioned that we may want to ditch the page timeline concept and allow any object to be placed on any channel. We'll have to decide if an author can understand the ramifications of coalescing timeline events in this manner.

How will be handle a timeline with page events and now allow a user to drag an event for a frame that exits on page 1 into the area of the timeline that corresponds to page 15?

Design issues

• Events should be pluggable
• Events should weak reference the objects they control
• Examples of events include:
  • hide/show a frame
  • Start/stop media
  • Go to an object (page, maybe a frame also?)
  • Animate a frame
    • translate
    • tween?
• Synchronizing - Sophie should make sure media on timelines remain in sync
• When an object that a ResourceRef points to is removed from a book, those references need to be notified (spoke with Gogov about this, we don't have this functionality yet)
• Media Events should have a inPoint and duration. inPoint is where in the media the event starts at. This allows for clipping larger media into smaller events.
• In Sophie 1, events kept which time on the timeline they occurred at. This meant an event could only be used once. I'm not sure if we want to think about doing it differently, where channels keep a list of events and what time the event happens. This might allow for re-use of events, but I am not sure how useful that is.

Use Cases

The most popular use cases involve the creation of a slideshow.

• Author wants to create a slideshow of images with sound.
• Author wants to play a simple audio file.
• Author wants to play a portion of a simple audio file.
• Author wants to play a movie on a single page.
• Author wants to play a portion of a movie.
• Author wants to play a movie that spans several pages.
• Author wants to synchronize pages with portions of a movie or audio.
• Author wants to play an animation.
• Author wants to change the rate of an animation or video.
• Author wants to create a book with sidebar annotations that are slideshows related to content on the page.
• Author wants to create 5 annotations that show in a window when triggered by a text link, and these annotations automatically start playing a slideshow with audio.
• Author wants to create an animation changing pages over time in one channel, starting and stopping movies in another channel at various times, and animating images in other channels at even other times.

Edit

User Interface

Here are Dan's notes on some issues that came up in the Sophie 1 UI (from the end user point of view). Not everything here will happen with the Sophie 2 model, but some of these basic issues will still be around.

Dan.1. How things worked in Sophie 1

In Sophie 1, when a frame was dragged to a timeline it became an event: a rounded rectangle in a channel on a timeline. The edges of the event could be dragged back and forth to change how long the event stayed visible. In this diagram, Frame 1 would be visible from 1:00 to 2:00:

One effect of how this worked was that once a frame had been dragged to a timeline, it was hidden on the page: the frame would only be visible when the playhead of the timeline was. When the playhead in the diagram above reached 1:00, Frame 1 would be shown; when it passed 2:00, Frame 1 would be hidden.

This behavior isn’t very important for audio on a timeline, because usually the visibility of the audio file’s image on the page isn’t very important. It’s worth focusing on in two particular cases of timelines & visibility: movies and interactions with links. (A third case, toggling visiblity via the page structure palette, also existed; however, I’m going to ignore that for now, assuming that the page structure palette will work differently in Sophie 2.)

Dan.1.1. Movies on timelines

When a movie is dragged to a page in Sophie 1, it doesn’t start playing automatically: usually you see the first frame, a black rectangle which looks like other frames. This frame can be dragged around the page; in a finished book, this black rectangle may indicate that there’s a movie that can be played, especially if it has controls underneath it. When a movie is dragged to a timeline, the movie behaves in the same way that a regular frame does: it becomes invisible when the movie is not playing.

This wasn’t always what users wanted; often, they wanted a black frame to appear until the timeline made the movie actually play. In later versions of Sophie 1, control-clicking a movie event on a timeline would toggle its visibility: if an event had been control-clicked, it would stay visible the entire time it was on a timeline, with the movie playing when the playhead was over it. This was not the best interface design; a better way to do this would have been to make this a check-box in the time-based media HUD. But this design element further complicated the problem of timeline visibility: the user couldn’t tell from looking at the timeline, what would be visible when.

Another problem with the way movies were handled on timelines was that the interface confused visibility with duration. The author would often want to clip a movie (i.e., if the have a 10 minute movie, they want to show from 2:00 to 5:00); the timeline seems like the logical place to do this. When you dragged your 10 minute movie to the timeline, dragging the edges of the clip would allow clipping from the back (making it play from 0–8:00 rather than from 0–10:00) but not clipping from the front (making it play from 2:00–10:00 rather than from 0–10:00). The problem was trying to include too much functionality with too few controls. Eventually, the ability to clip movies and audio that were on timelines was added to the time-based media HUD on the frame on the page; this probably wasn’t the most obvious place to put it.

Dan.1.2. Links and show/hide actions

Visibility on timelines was also complicated if there were links with show/toggle/hide actions. These links would do the following:

• show: a frame is set to be visible regardless of previous state.
• hide: a frame is set to be invisible regardless of previous state.
• toggle: a frame is set to be visible if it is invisible or invisible if it is visible.

Show and toggle worked the same way as dragging an event to a timeline: making a frame the target of one of these links would make it invisible by default. This is where the conflict with timelines arises: if a frame that is shown or hidden by a link appears on a timeline, the timeline’s display of the frame’s visibility will not necessarily be correct.

Dan.1.3. Show/hide via page structure palette

Because of these conflicts, a checkbox for frame visibility was added next to each frame’s entry in the page structure palette. When a frame was added to a timeline, it was unchecked because the frame became invisible; when a frame was made the target of a show/toggle link, the visibility checkbox was also unchecked. In addition, the user could manually uncheck frame visibility.

In practice, this didn't end up working very well. Users tended to make frames invisible so that they could work on other frames on the page (the way that layers are hid in Photoshop or Illustrator); they were confused when the visibility changes were persistent. Probably visibility for frames and visibility for authoring should have been separated.

Dan.2. Ideas for how things could work

Channels in Sophie 1 weren't specific: an event could be in any channel that the author wanted, and could be moved from channel to channel. It might make sense in Sophie 2 to have a channel for each event on a timeline. This would allow more specific controls for visibility: a channel for video could have a control specifying whether the frame was always visible or not.

Dan.2.1. Changing the representation of the event on the channel

It also might make sense to get rid of the idea of using a draggable rectangle as the signifier of the event's visibility - maybe it should be limited to representing clipping of audio and video. GarageBand on the Mac creates a sub-channel if you want to adjust the volume of a clip; there's a draggable line that can be moved up and down to adjust the volume:

Something like this could be done to adjust visibility rather than to adjust volume - if it could be implemented like this, we could even imagine using this to create fadeins and fadeouts. When visibility is separated from the movie's representation, the movie could be clipped separately from adjusting its visibility.

Dan.2.2. Noting interactions with links

It might make sense to have the timeline check for interactions with links on the page. If a frame on the page is the target of a link that changes its visibility and that frame is also on the timeline, it might make sense to put a small warning (maybe on the channel), indicating that there might be a possible conflict. We can't fix all of the user's problems - it's the author's responsibility to make a book or page that works for the reader.

Edit

Comments

^{Write your comments here.}