wiki:MEDIA_ENGINE_IMPL_MAIN_R2

Version 5 (modified by nenko, 16 years ago) (diff)

--

Error: Macro BackLinksMenu(None) failed
compressed data is corrupt

Error: Macro TicketQuery(summary=MEDIA_ENGINE_IMPL_MAIN_R2, format=table, col=summary|owner|status|type|component|priority|effort|importance, rows=description|analysis_owners|analysis_reviewers|analysis_score|design_owners|design_reviewers|design_score|implementation_owners|implementation_reviewers|implementation_score|test_owners|test_reviewers|test_score|) failed
current transaction is aborted, commands ignored until end of transaction block

Analysis

Overview

To support video frame in Sophie2 we need this frame to be:

  • rotatable
  • support opacity
  • be part of the scene implementation

So we need media framework that can extract an image from a video. On the other hand the video frame display the image. If the video frame is refreshed 24 times in a second we have a video playing ;). Currently the JMF media library is used for audio and video playback and for audio recording. JMF is outdated and doesn't support much of the new codecs that exist. Also it has mucher richer functionality that we need. As an alternative FFmpeg C library is supported on all platforms targeted by Sophie and is a live project. Also it supports many codecs and it is basicly the funcionality we need (a framework to extract frames from various video formats and codecs). It can encode/decode most of the existing formats/encodings with few exceptions. FFmpeg is a C library so we need to have a native wrapper for it.

Task requirements

Media will be designed to work with the FFmpeg (​http://ffmpeg.org/) and within timelines.

A C++ wrapper for capturing video will be created to work with FFmpeg for Linux, Windows and Mac OS X as well as for their 64bit versions.

A C++ wrapper for playing back audio will be created to work with SDL or PortAudio for all platforms.

A protocol has to be defined for communication between the FFmpeg wrappers and Sophie.

Task result

Demo, usage wiki page for the C wrapper.

Implementation idea

Use FFmpeg.
Change the logic of media in author to manipulate the media timeline.

TIMELINES_REDESIGN_R1

How to demo

  • Insert a video frame.
  • Play it to see if it is ok.
  • Insert audio frame.
  • Play the video and audio frames.
  • Start some audio playback in a system player.
  • Check that sound is mixed correctly.

Design

The communication between the C++ program and the Java program (Sophie2, except the demos) will be sequence of integers. We chose integers over string because there are too many encodings for strings. Every integer sequence will be in the following format:
<size> <id> ...
<size> and <id> are mandatory and their meanings are :

  • size - the length of the integer sequence
  • id - command/sequence id

Example(start command, whose id is 0) : 2 0

The communication will be in the following way:

  • the Java program sends a command through the standard input of the C++ process
  • the C++ program creates a respones on the base of the command
  • the C++ program returns the reposne through its standard output
  • the Java program receives the respone and acts on the base of it

This means that the Java program will be blocked until it recieves mething from the standard input of the process so the Java wrapper will be in separate thread. Also the control of the C++ application will be in the Java part - it will run and stop it.

The flow of communication written above is common for all other wrapers and native applications. This code will reside in classes called Bridges.
There will be some common commands and responses:

  • StartCommand
  • StopCommand
  • OkResponse
  • ErrorResponse

Design of the C++ part:

Design of the Java part:

Implementation

(Describe and link the implementation results here (from the wiki or the repository).)

Testing

(Place the testing results here.)

Comments

(Write comments for this or later revisions here.)