CS 414 Multimedia Systems Design Lecture 28 Synchronization

CS 414 – Multimedia Systems Design Lecture 28 – Synchronization Issues Klara Nahrstedt Spring 2014 CS 414 - Spring 2014

Administrative n MP 2 ongoing CS 414 - Spring 2014

Covered Aspects of Multimedia Image/Video Capture Audio/Video Perception/ Playback Audio/Video Presentation Playback Image/Video Information Representation Transmission Audio Capture Transmission Compression Processing Audio Information Representation Media Server Storage CS 414 - Spring 2014 A/V Playback

Internet Multimedia Protocol Stack APPLICATION Media encaps (H. 264, MPEG-4) Synchronization Service DASH SIP RTSP RSVP RTP HTTP KERNEL TCP DCCP UDP IP Version 4, IP Version 6 AAL 3/4 Layer 5 (Session) RTCP AAL 5 MPLS CS 414 - Spring 2014 Layer 3 (Network) Layer 2 (Link/MAC) Ethernet/Wi. Fi ATM/Fiber Optics Layer 4 (Transport)

Outline n Synchronization Specification ¨ Interval-based Specification ¨ Timeline Specification ¨ Control Flow based Specification ¨ Event-based Specification CS 414 - Spring 2014

Requirements on Synchronization Specs n Object consistency and maintenance of sync specifications ¨ Media n n n objects should be kept as one LDU in spec Temporal relations must be specify-able Easy Description of Sync Relations Definition of Qo. S requirements Integration of time-dependent and independent media Hierarchical levels of synchronization CS 414 - Spring 2014

Sync Spec Models n n n Interval Timeline Flow-control Hierarchical ¨ Reference points ¨ n Event-based n Common threads ¨ provide language to express relationships ¨ runtime system to monitor relationships ¨ policies to enforce relationships CS 414 - Spring 2014

Interval-based Specification (1) Presentation duration of an object is specified as interval n Types of temporal relations: n ¨A before B, A overlaps B, A starts B, A equals B, A meets B, A finishes B, A while B n Enhanced interval-based model includes 29 interval relations, 10 operators handle temporal relations (e. g. , before(δ 1), …) CS 414 - Spring 2014

Interval Model (2) n 13 relationships between two intervals B A Before Starts B A B Meets Equal Ends A A B B Overlaps A While A A B B CS 414 - Spring 2014

Example (3) Audio 1 while(0, 0) Video Audio 1 before(0) Recorded. Interaction before(0) P 1 before(0) P 2 before(0) P 3 before(0) Interaction P 3 before(0) Animation while(2, 5) Audio 2 Interaction before(0) P 4 CS 414 - Spring 2014

Interval-based Specification (4) n Advantages: ¨ Easy to handle open LDUs (i. e. , user interactions) ¨ Possible to specify additional nondeterministic temporal relations by defining intervals for durations and delays ¨ Flexible model that allows specification of presentations with many run-time presentation variations CS 414 - Spring 2014

Interval-based Specification (5) n Disadvantages: ¨ Does not include skew spec ¨ Does not allow specification of temporal relations directly between sub-units of objects ¨ Flexible spec leads to inconsistencies n Example: A NOT in parallel with B A while(2, 3) I I before(0) B CS 414 - Spring 2014

Timeline Axis-based Specification Presentation events like start and end of presentation are mapped to axes that are shared by presentation objects n All single medium objects are attached to time axis that represents abstraction of real-time n This sync specification is very good for closed LDUs n CS 414 - Spring 2014

Timeline Model (2) n Uses a single global timeline n Actions triggered when the time marker reaches a specific point along timeline CS 414 - Spring 2014

Example (3) n Define a timed sequence of images, each image has a caption that goes with it I 1 I 2 I 3 C 1 C 2 C 3 t 1 t 2 t 3 CS 414 - Spring 2014

Example (4) n Rule language n Visual environment ¨ At (t 1), show (I 1, C 1) ¨ At (t 2), show (I 2, C 2) ¨ At (t 3), show (I 3, C 3) CS 414 - Spring 2014

Control Flow-based Spec - Hierarchical Model (1) n n Possibility to specify concurrent presentation threads at predefined points of presentation Basic hierarchical spec types: Serial synchronization ¨ Parallel synchronization of actions ¨ n Actions: atomic or compound Atomic action handles presentation of single media object, user input, delay ¨ Compound actions are combinations of sync operators and atomic actions ¨ Delay is atomic action – allows modeling of delays in serial presentations ¨ CS 414 - Spring 2014

Hierarchical Model (SMIL) n Based on sequential and parallel I 1 n I 2 I 3 I 1 T 1 Apply operators to only the start/end points of each media object CS 414 - Spring 2014

Example (3) n Narrated slide show ¨ image, text, audio on each slide ¨ select link to move to the next slide … S 1 A 1 T 1 I 1 S 2 A 2 T 2 I 2 CS 414 - Spring 2014

Example (4) (and Comparison with Intervalbased Spec) Audio 1 while(0, 0) Video Audio 1 before(0) Recorded. Interaction before(0) P 1 before(0) P 2 before(0) P 3 before(0) Interaction P 3 before(0) Animation while(2, 5) Audio 2 Interaction before(0) P 4 CS 414 - Spring 2014

Control Flow-based Spec – Hierarchy (5) n Advantages ¨ Easy to understand ¨ Natural support for hierarchies ¨ Integration of interactive object easy n Disadvantage ¨ Need additional descriptions of skews and Qo. S ¨ No duration description CS 414 - Spring 2014

Control Flow-based Spec – Reference Points (1) Time-dependent single medium objects are regarded as sequences of closed LDUs n Start/stop times of object presentation are reference points n Connected reference point is synchronization point n Temporal relations specified between objects without explicit reference to time n CS 414 - Spring 2014

Example (2) CS 414 - Spring 2014

Control Flow-based Spec – Reference Points (3) n Advantages: ¨ Sync at any time during presentation of objects ¨ Easily integrated object presentation with unpredictable duration ¨ Intuitive type of synchronization spec n Disadvantages: ¨ Not easy way to detect inconsistencies ¨ Cannot specify delays in presentation CS 414 - Spring 2014

Event-based Specification Presentation actions initiated by synchronization events n Example: n ¨ Start presentation ¨ Stop presentation ¨ Prepare presentation n Events initiating presentation ¨ External or internal CS 414 - Spring 2014

Event-based Spec n Advantage: ¨ Easily extended to new sync types ¨ Easy integration of interactive objects n Disadvantage: ¨ Difficult to handle in case of realistic scenarios ¨ Too complex specification ¨ Need separate description of skew/Qo. S ¨ Difficult use of hierarchies CS 414 - Spring 2014

Event Model (Nsync) Associate actions with expressions n Expressions may contain scalars, clocks, variables, relations, and connectives n When the expression becomes TRUE, invoke associated action n When “Time > Q. end + 5 && !Response” Answer=WRONG Source: B. Bailey et al. “Nsync- A Toolkit for Building Interactive Multimedia Presentations”, ACM Multimedia 1998

Example: Delayed Transition Overview More Info? More Info Yes Detailed Narration CS 414 - Spring 2014 No More Info

Model Specification When “Narration >= Overview && !More. Info” Next. Slide When “Narration >= Overview && More. Info” Play. Details When “Narration >= Overview + Details” Next. Slide Narration: narration’s logical timeline Overview: normal transition point Details: additional narrative details More. Info: records kitchen info status CS 414 - Spring 2014

Reactive Interface CS 414 - Spring 2014

Model Specification When “Video >= 0 && Video < T 1” Select Kitchen When “Video >= T 1 && Video < T 2” Select Deck When “Video >= T 2 && Video <= T 3” Select Yard CS 414 - Spring 2014

Synchronization/Editing Tool in CS/UIUC Vegas Video http: //www. sonycreativesoftware. com/vegaspro CS 414 - Spring 2014

Conclusion n Synchronization Specifications ¨ Important for different authoring tools for complex presentation ¨ Be careful as you go from one spec to another ¨ Carefully consider which spec closest allows you to specify sync requirements in your application CS 414 - Spring 2014