CS 414 – Multimedia Systems Design Lecture 33 – Synchronization (Part 1)AdministrativeOutlineNotion of SynchronizationSpatial RelationTemporal Relation (Our focus!!!)Temporal RelationsSynchronization ToolsSynchronization SpecificationLogical Data Units and their ClassificationSynchronization ClassificationSynchronization ClassificationSlide 13Synchronization Requirements during media presentationsGap Problem in SynchronizationLip SynchronizationLip SynchronizationLip Synchronization RequirementsPointer SynchronizationSlide 20Pointer Synchronization RequirementsOther Sync RequirementsConclusionCS 414 - Spring 2011CS 414 – Multimedia Systems Design Lecture 33 – Synchronization (Part 1) Klara NahrstedtSpring 2011AdministrativeMP3 is ongoing‘mi-clicker’ experiment is ongoingCS 414 - Spring 2011Outline Synchronization Concept Synchronization Classification Logical Data Units Live vs Synthetic SynchronizationSynchronization RequirementsCS 414 - Spring 2011Notion of Synchronization Sync in correspondence to Content relation Spatial relationTemporal relation Content Relation Define dependency of media objects for some dataExample: dependency between spreadsheet and graphics that represent data listed in spreadsheet CS 414 - Spring 2011Spatial RelationLayout relation Defines space used for presentation of media object on output device at certain point of multimedia presentationExample: desktop publishingLayout framesPlaced on output device and content assigned to framePositioning of layout frames: Fixed to position of documentFixed to position on pageRelative to position of other frameExample: in window-based system, layout frames correspond to windows and video can be positioned in windowCS 414 - Spring 2011Temporal Relation (Our focus!!!) Defines temporal dependencies between media objectsExample: lip synchronizationTime-dependent objectMedia stream since there exist temporal relations between consecutive units of the stream Time-independent objectTraditional medium such as text or imagesTemporal synchronization Relation between time-dependent and time-independent objectsExample: audio/video sync with slide showCS 414 - Spring 2011Temporal RelationsSynchronization considered at several levels of Multimedia SystemsLevel 1: OS and lower level communication layers CPU scheduling, semaphores during IPC, traffic shaping network scheduling Objective: avoid jitter at presentation time of one streamLevel 2: Middleware/Session layer (Run-time)Synchronization of multimedia streams (schedulers)Objective: bounded skews between various streamsLevel 3: Application layer (Run-time) Support for synchronization between time-dependent and time-independent media together with handling of user interaction Objective: bounded skews between time-dependent and time-independent mediaCS 414 - Spring 2011Synchronization ToolsCS 414 - Spring 2011Synchronization Specification ImplicitTemporal relation specified implicitly during capturing of media objectsGoal: use this temporal relation to present media in the same way as they were originally capturedExample: Audio and Video recording and playback ExplicitTemporal relation specified explicitly to define dependency in case media objects were created independentlyExample: creation of slide showPresentation designerselects slides, creates audio objects, defines units of audio presentation stream, defines units of audio presentation stream where slides have to be presentedCS 414 - Spring 2011Logical Data Units and their Classification Time-dependent presentation units are called logical data units (LDU)s. LDU classificationOpen ClosedLDUs important In specification of synchronization CS 414 - Spring 2011Synchronization ClassificationIntra-object Synchronization Time relation between various presentation units of one time-dependent media streamInter-object Synchronization Time relation between media objects belonging to two dime dependent media streamsCS 414 - Spring 2011Synchronization Classification Live Synchronization Goal: exactly reproduce at presentation temporal relations as they existed during capturing processRequirement: must capture temporal relation information during media capturingExample: video conference, phone serviceExample: recording and retrieval services – presentations with delayCS 414 - Spring 2011Synchronization Classification Synthetic Synchronization Goal: arrange stored data objects to provide new combined multimedia objects via artificial temporal relationsRequirements: support flexible synchronization relations between media Example: authoring, tutoring systemsTwo phases: Specification phase – define temporal relationsPresentation phase – present data in sync modeExample: 4 audio messages recorded related to parts of engine in animation. Animation sequence shows a slow 360 degree rotation of engineCS 414 - Spring 2011Synchronization Requirements during media presentations For intra-object synchronization Need accuracy concerning jitter and EED delays in presentation of LDUsFor inter-object synchronization Need accuracy in parallel presentation of media objectsImplication of blocking: O.K. for time-independent mediaProblem for time-dependent media – gap problemCS 414 - Spring 2011Gap Problem in Synchronization What does blocking of stream mean for output device? Should we repeat previous music, speech, picture? How long should such gap exist? Solution 1: restricted blocking method Switch output device to last picture as still pictureSwitch output device to alternative presentation if gap between late video and audio exceeds predefined threshold Solution 2: resample stream Speed up or slow down streams Off-line re-sampling – used after capturing of media streams with independent streamsExample: concert which is captured with two independent audio/video devicesOnline re-sampling – used during presentation in case gap between media streams occursCS 414 - Spring 2011Lip Synchronization Temporal relation between audio and videoSynchronization skewTime difference between related audio and video LDUsStreams in sync iff skew = 0 or skew ≤boundNegative skew: video before audio Positive skew: Audio before videoCS
View Full Document
Unlocking...