U of I CS 414 - Multimedia Synchronization

Unformatted text preview:

Multimedia SynchronizationAnnouncementsMM SynchronizationContent and Spatial RelationshipsTemporal RelationshipsLip SynchronizationSlide 7Tele-pointer SynchronizationSynchronization GuidelinesInteraction RelationshipsSynchronization ModelModelsTimeline ModelExampleExample (Cont.)Hierarchical Model (SMIL)Slide 17Timed Petri NetsExampleInterval ModelEvent Model (Nsync)Background and Time ModelExample: Delayed TransitionModel SpecificationReactive InterfaceSlide 26Expression EvaluationPredictive Logic StatesPrediction ExampleSlide 30Evaluation Rules for “AND”Take Home ExerciseProsConsSlide 35Multimedia SynchronizationBrian P. BaileySpring 2006AnnouncementsMM Synchronization•Applications composed of more than one media (at least one continuous)•Express desired relationships–content, spatial, temporal, and interaction–combinations of eachContent and Spatial Relationships•Content–define how views relate to data sources–e.g., a graph linked to a table of data•Spatial–define relative positions of media objects–subdivide the space; express relationships–e.g., pack command in Tcl, layout managers in Java, tables in HTML, etc.Temporal Relationships•Define how media are coordinated in time–audio should not drift from video by > 80ms–voice narration should accompany a slide and end when user navigates elsewhere–display different caption for each video scene; and update it in response to user interaction•Intra-media and inter-media relationships•Time-independent and dependent mediaLip SynchronizationLeft: audio after video; Right: audio before videoLip SynchronizationTolerableNottolerableNotdetectableTolerableNottolerableTele-pointer SynchronizationLeft: pointer before audio; Right : pointer after audioSynchronization Guidelines•Lip synchronization within 80ms–video before audio is more tolerable•Other fine-grained synchronization should typically be within range of 500msInteraction Relationships•Define how interaction affects playback–e.g., if user transitions to next slide in narrated slide show, narration should change as well•Classes of interaction–navigation, participation, and control–asynchronous and synchronousSynchronization Model•Enables expression of media and synchronization relationships•An effective model should support:–spatial and temporal relations (fine & coarse)–rich interaction (beyond VCR control)–efficient runtime (interaction monitoring)–be usable and comprehensibleModels•Timeline•Hierarchical•Petri net•Interval•Event-based•Common threads–provide language to express relationships–runtime system to monitor relationships–policies to enforce relationshipsTimeline Model•Uses a single global timeline•Actions triggered when the time marker reaches a specific point along timelineExample•Define a timed sequence of images, each image has a caption that goes with itI1C1t1I2C2t2I3C3t3Example (Cont.)•Rule language–At (t1), show (I1, C1)–At (t2), show (I2, C2)–At (t3), show (I3, C3)•Visual environmentHierarchical Model (SMIL)•Based on sequential and parallel•Apply operators to only the start/end points of each media objectI1 I2 I3 I1 T1Example•Narrated slide show–image, text, audio on each slide–select link to move to the next slideS1 A1 T1…I1S2 A2 T2 I2Timed Petri Nets•tokens, places, transitions, and arcsExample 11ms 11ms 11ms33ms11ms 11ms33msSpecify audio video synchronizationInterval Model•13 relationships between two intervalsBeforeABMeetsABDuringABOverlapsABStartsABEndsABEqualAB•Associate actions with expressions•Expressions may contain scalars, clocks, variables, relations, and connectives•When the expression becomes TRUE, invoke associated action When “Time > Q.end + 5 && !Response” Answer=WRONGEvent Model (Nsync)Background and Time Model•Each media object attached to a clock•Clock implements logical time–Value = Rate * System + Offset•Express temporal behavior as relationships among clocks•Interactive events tied to variablesOverviewOverviewNoNoYesYesMoreMoreInfo?Info?DetailedDetailedNarrationNarrationMore InfoMore InfoMore InfoMore InfoExample: Delayed TransitionModel SpecificationWhen “Narration >= Overview && !MoreInfo” NextSlideWhen “Narration >= Overview && MoreInfo” PlayDetailsWhen “Narration >= Overview + Details” NextSlideNarrati o n: narration’s logical timelineOvervie w: normal transition pointDetails: additional narrative detailsMoreInf o: records kitchen info statusReactive InterfaceModel Specification When “Video >= 0 && Video < T1” Select Kitchen When “Video >= T1 && Video < T2” Select Deck When “Video >= T2 && Video <= T3” Select YardExpression Evaluation•Propositional logic breaks down–returns logic value only at present time–requires polling to catch future transitions•Predictive logic–returns logic value at present time along with a prediction of any future transition–eliminates need for intermittent polling/timersPredictive Logic States•WBT(t) False now, but Will Become Trueat future time t•WBF(t) True now, but Will Become Falseat future time tPrediction Example When “Video > 10” ActionWhen “Video > 10” Action1010Video TimeVideo TimeRate = 1Rate = 1 (then - now)t = ----------------- rateSystem TimeSystem Timet = (10 - 0) / 1 = WBT(10)00Prediction Example When “Video > 10” ActionWhen “Video > 10” Action001010Video TimeVideo TimeRate = 1Rate = 1 (then - now)? = ----------------- rateSystem TimeSystem Time? = (10 - 3) / 2 = WBT(3.5)?Rate = 2Rate = 233Evaluation Rules for “AND”WBT(x) && WBT(y) = WBT( max(x, y) )WBF(x) && WBF(y) = WBF( min(x, y) )WBF(x) && WBT(y) = WBT(infinity) if (x < y) WBT(y) then WBF(x) otherwiseTake Home ExerciseWBT(x) | | WBT(y) = ?WBF(x) | | WBF(y) = ?WBF(x) | | WBT(y) = ?Pros•Complements current languages–adds ability to express combinations of interactive and temporal behavior–syntax can easily be translated into mark up•Predictive logic useful in run-time engines–eliminates need for polling/timers–enables look-ahead pre-fetchingCons•Difficult to visualize rule propagation–makes system difficult to debug•Rules are not groups into hierarchies–enable divide and conquer strategy•Lack of scope–all rules always active–guard actions with complex


View Full Document

U of I CS 414 - Multimedia Synchronization

Documents in this Course
Lecture 1

Lecture 1

32 pages

LECTURE

LECTURE

30 pages

Load more
Download Multimedia Synchronization
Our administrator received your request to download this document. We will send you the file to your email shortly.
Loading Unlocking...
Login

Join to view Multimedia Synchronization and access 3M+ class-specific study document.

or
We will never post anything without your permission.
Don't have an account?
Sign Up

Join to view Multimedia Synchronization 2 2 and access 3M+ class-specific study document.

or

By creating an account you agree to our Privacy Policy and Terms Of Use

Already a member?