CS 598 KN Advanced Multimedia Systems Synchronization Issues

CS 598 KN – Advanced Multimedia Systems Synchronization Issues Klara Nahrstedt Fall 2018 CS 598 KN - Fall 2018

Internet Multimedia Protocol Stack APPLICATION Media encaps (H. 264, MPEG-4) Synchronization Service DASH SIP RTSP RSVP RTP HTTP KERNEL TCP DCCP UDP IP Version 4, IP Version 6 AAL 3/4 Layer 5 (Session) RTCP AAL 5 MPLS CS 598 KN - Fall 2018 Layer 3 (Network) Layer 2 (Link/MAC) Ethernet/Wi. Fi ATM/Fiber Optics Layer 4 (Transport)

Outline Synchronization Concept n Synchronization Classification n Logical Data Units n Live vs Synthetic Synchronization n Synchronization Requirements n Time Concepts n Synchronization Modeling n CS 598 KN - Fall 2018

Notion of Synchronization n Sync in correspondence to ¨ Content relation ¨ Spatial relation ¨ Temporal relation n Content Relation ¨ Define dependency of media objects for some data ¨ Example: dependency between spreadsheet and graphics that represent data listed in spreadsheet CS 598 KN - Fall 2018

Spatial Relation n Layout relation ¨ Defines space used for presentation of media object on output device at certain point of multimedia presentation ¨ Example: desktop publishing n Layout frames ¨ Placed on output device and content assigned to frame ¨ Positioning of layout frames: n n n Fixed to position of document Fixed to position on page Relative to position of other frame ¨ Example: in window-based system, layout frames correspond to windows and video can be positioned in window CS 598 KN - Fall 2018

Temporal Relation (Our focus!!!) n n n Defines temporal dependencies between media objects Example: lip synchronization Time-dependent object ¨ n Time-independent object ¨ n Media stream since there exist temporal relations between consecutive units of the stream Traditional medium such as text or images Temporal synchronization Relation between time-dependent and time-independent objects ¨ Example: audio/video sync with slide show ¨ CS 598 KN - Fall 2018

Temporal Relations n n Synchronization considered at several levels of Multimedia Systems Level 1: OS and lower level communication layers CPU scheduling, semaphores during IPC, network scheduling ¨ Objective: avoid jitter - introduction of low level jitter control ¨ n Level 2: Session layer (Run-time) Synchronization of streams (rate controllers, schedulers) ¨ Objective: bounded jitters within streams and skews between various streams ¨ n Level 3: Application layer (Run-time) Support for synchronization between time-dependent and timeindependent media together with handling of user interaction ¨ Objective: bounded skews between time-dependent and timeindependent media ¨ CS 598 KN - Fall 2018

Synchronization Specification n Implicit Temporal relation specified implicitly during capturing of media objects ¨ Goal: use this temporal relation to present media in the same way as they were originally captured ¨ Example: Audio and Video recording and playback ¨ n Explicit Temporal relation specified explicitly to define dependency in case media objects were created independently ¨ Example: creation of slide show n Presentation designer ¨ ¨ ¨ selects slides, creates audio objects, defines units of audio presentation stream where slides have to be presented CS 598 KN - Fall 2018

Synchronization Tools CS 598 KN - Fall 2018

Logical Data Units and their Classification n n Time-dependent presentation units are called logical data units (LDU)s. LDU classification ¨ Open ¨ Closed n LDUs important ¨ In specification of synchronization CS 598 KN - Fall 2018

Synchronization Classification n Intra-object Synchronization ¨ n Time relation between various presentation units of one timedependent media stream Inter-object Synchronization ¨ Time relation between media objects belonging to two dime dependent media streams CS 598 KN - Fall 2018

Synchronization Classification n Live Synchronization ¨ Goal: exactly reproduce at presentation temporal relations as they existed during capturing process ¨ Requirement: must capture temporal relation information during media capturing ¨ Example: video conference, phone service ¨ Example: recording and retrieval services – presentations with delay CS 598 KN - Fall 2018

Synchronization Classification n Synthetic Synchronization Goal: arrange stored data objects to provide new combined multimedia objects via artificial temporal relations ¨ Requirements: support flexible synchronization relations between media ¨ Example: authoring, tutoring systems ¨ n Two phases: ¨ Specification n phase – define temporal relations ¨ Presentation phase – present data in sync mode Example: ¨ 4 audio messages recorded related to parts of engine in animation. Animation sequence shows a slow 360 degree rotation of engine CS 598 KN - Fall 2018

Synchronization Requirements during media presentations n For intra-object synchronization ¨ Need accuracy concerning jitter and EED delays in presentation of LDUs n For inter-object synchronization ¨ Need accuracy in parallel presentation of media objects n Implication of blocking: ¨ O. K. for time-independent media ¨ Problem for time-dependent media – gap problem CS 598 KN - Fall 2018

Gap Problem in Synchronization n What does blocking of stream mean for output device? Should we repeat previous music, speech, picture? ¨ How long should such gap exist? ¨ n Solution 1: restricted blocking method Switch output device to last picture as still picture ¨ Switch output device to alternative presentation if gap between late video and audio exceeds predefined threshold ¨ n Solution 2: resample stream Speed up or slow down streams ¨ Off-line re-sampling – used after capturing of media streams with independent streams ¨ n ¨ Example: concert which is captured with two independent audio/video devices Online re-sampling – used during presentation in case gap between media streams occurs CS 598 KN - Fall 2018

Lip Synchronization n n Temporal relation between audio and video Synchronization skew ¨ Time difference between related audio and video LDUs Streams in sync iff skew = 0 or skew ≤bound Negative skew: video before audio Positive skew: Audio before video CS 598 KN - Fall 2018

Lip Synchronization Perception of Synchronization Errors Skew Level found to be annoying CS 598 KN - Fall 2018

Lip Synchronization Requirements n In sync: ¨ n -80 ms ≤ skew ≤ 80 ms Out of sync: ¨ Skew < -160 ms ¨ Skew > 160 ms n Transient: ¨ -160 ms ≤ skew < -80 ms ¨ 80 ms < skew ≤ 160 ms CS 598 KN - Fall 2018

Pointer Synchronization Pointer Sync based on technical drawing CS 598 KN - Fall 2018 Pointer Sync based on map

Pointer Synchronization Negative skew: pointer before audio Positive skew: pointer after audio CS 598 KN - Fall 2018

Pointer Synchronization Requirements n In sync: ¨ -500 ms n ≤ skew ≤ 750 ms Out of sync: ¨ Skew < -1000 ms ¨ Skew > 1250 ms n Transient sync situation: ¨ -1000 ms ≤ skew < -500 ms ¨ 750 ms < skew ≤ 1250 ms CS 598 KN - Fall 2018

Other Sync Requirements n Jitter delay of digital audio ¨ n Max. allowable jitter: n 5 -10 ns (perception experiments) n 2 ms (other experiments) Combination of audio and animation Not stringent as lip sync ¨ Max allowable skew: +/- 80 ms ¨ n Stereo audio Tightly coupled ¨ Max allowable skew: 20 ms ¨ Due to listening errors, suggestion even +/- 11 ms ¨ n Loosely coupled audio channels (speaker and background music) ¨ Max allowable skew: 500 ms CS 598 KN - Fall 2018

Synchronization in Distributed Environments n n Information of synchronization must be transmitted with audio and video streams, so that receiver(s) can synchronize streams Sync information can be delivered before start of presentation (used by synthetic synchronization) Advantage: simplementation ¨ Disadvantage: presentation delay ¨ n Sync information can be delivered using separate sync channel - out-band (used by live synchronization) Advantage: no additional presentation delay ¨ Disadvantage: additional channel needed ¨ CS 598 KN - Fall 2018

Sync in Distributed Environments n Sync information can be delivered using multiplexed data streams - in-band sync ¨ Advantage: related sync information is delivered together with media units ¨ Disadvantage: difficult to use for multiple sources CS 598 KN - Fall 2018

Location of Sync Operation Sync media objects by combining objects into new media object n Sync operation placed at sink n ¨ Demand on bandwidth is larger because additional sync operations must be transported n Sync operation placed at source ¨ Demand on bandwidth smaller because streams are multiplexed according to sync requirements CS 598 KN - Fall 2018

SYNCHRONIZATION TIME CONCEPT CS 598 KN - Fall 2018

Clock Synchronization n Sync accuracy depends on clocks at source and sink nodes ¨ Ta = Tav – Nla – Oa ¨ Tv = Tav – Nlv – Ov n End-to-end delay ¨ Nla = EEDa = Tav-Ta-Oa ¨ Nlv=EEDv = Tav-Tv-Ov ¨ EEDa = (Ta 1 -Ta 2)/2 n NTP (Network Time Protocol ) CS 598 KN - Fall 2018

AV Skew Calculation Select synchronization point in AV streams in terms of frame number for each stream n Execute sync operation at sync points 1. Measure n ¨ Arrival time of sync audio packet Tsync. A ¨ Arrival time of sync video packet Tsync. V 2. Calculate: ¨ AVSkew = Tsync. A – Tsync. V 3. Alternative measure and calculate ¨ Tsync. A = Tsend. A + EEDA; Tsync. V = Tsend. V + EEDV ¨ EED = RTT/2 = (Trecv – Tsend)/2 ; Trecv@node 1; CS 598 KN - Fall 2018 Tsend@node 1 of the same packet

Network Time Protocol n Protocol to sync clocks of computer systems over packet-switched, variable – latency data networks ¨ Uses UDP port 123 ¨ Designed to resist effects of variable latency (jitter buffer) ¨ Designed in 1985 by Dave Mills at U. Delaware ¨ Can achieve accuracy of 200 µsec ¨ Based on Marzullo Algorithm CS 598 KN - Fall 2018

Marzullo’s algorithm (1984) (intersection algorithm) n n Agreement protocol for estimating accurate time from a number of noisy time sources If we have estimates 10 2, 12 1, 11 1, then interval intersection is 11. 5 0. 5 If some intervals don’t intersect, consider intersection of majority of intervals Source: wikipedia CS 598 KN - Fall 2018

Clock strata n n NTP uses hierarchical system of “clock strata” Stratum levels define distance from reference clock and exist to prevent cycles in hierarchy ¨ Stratum 0 n ¨ Stratum 1 n n ¨ devices are atomic clocks, GPS clocks, radio clocks computers attached to stratum 0 devices Act as servers for timing requests from Stratum 2 servers via NTP Stratum 2 n (similar to Stratum 1, but they also have peering relation to other stratum 2 servers CS 598 KN - Fall 2018

Precision Time Protocol (PTP) n n High Precision time protocol (original IEEE 1588 -2002) for synchronization used in measurement and control systems residing on a local area network Accuracy in the sub-microseconds PTPv 2 – IEEE 1588 -2008 improves accuracy, precision and robustness PTP serves a niche not well served by NTP and GPS ¨ it achieves accuracy beyond those attainable using NTP; ¨ It is cheaper than GPS ¨ It is accessible to receivers for which GPS signals are inaccessible. CS 598 KN - Fall 2018

PTP Details n IEEE 1588 standards describe a hierarchical masterslave architecture for clock distribution Ordinary clock – source for root timing reference, called ‘grandmaster’ ¨ Boundary clock – intermediate synchronization bridge from one network segment to another – used by synchronization master to sync clocks within a network segment ¨ n n PTP messages use IP-multicast over UDP (IPv 4 or IPv 6) Selection algorithm of best clock source uses properties such as quality, accuracy, variance, … CS 598 KN - Fall 2018

PTP Synchronization n After selection of master source of time, clocks determine the offset between themselves and their master ¨ Let t represent physical time, given slave device, the offset o(t) at time t is: o(t) = s(t) – m(t), n n m(t) is measured time at master at time t, and s(t) is measured time at local clock at time t. ¨ Master periodically broadcasts the current time to other clocks (up to 10 measurements per second in 1588 -2008) CS 598 KN - Fall 2018

PTP Synchronization T 1’ – T 1 = o + d and T 2’ – T 2 = -o + d, then we find o = (T 1’ - T 1 – T 2’ + T 2)/2 The clock now knows offset o during the transactions and can correct itself by this agreement with the master clock. CS 598 KN - Fall 2018

Other Sync Issues n n Sync must be considered during object acquisition Sync must be considered during retrieval ¨ Sync n Sync must be considered during transport ¨ If n possible use isochronous protocols Sync must be considered at sink ¨ Sync n access to frames of stored video delivery to output devices Sync must consider support of functions such as pause, forward, rewind with different speeds, direct access, stop or repeat CS 598 KN - Fall 2018

SYNCHRONIZATION MODELING CS 598 KN - Fall 2018

Reference Models n We need reference models to ¨ Understand various requirements for multimedia sync ¨ Identify and structure run-time mechanisms to support execution of sync ¨ Identify interface between run-time mechanisms ¨ Compare system solutions for multimedia sync CS 598 KN - Fall 2018

Synchronization Reference Model n Sync model we will be evaluating in detail is according to Meyer, Effelsberg, Steinmetz: ¨ Sync multimedia objects are classified according to Media level n Stream level n Object level n Specification level n CS 598 KN - Fall 2018

Media Level (1) Each application operates single continuous media streams composed of sequence of LDUs n Each media stream responsible for intrastream synchronization n Assumption at this level: device independence n Supported operations at this level: n ¨ read(devicehandle, LDU) ¨ write(devicehandle, LDU) CS 598 KN - Fall 2018

Media Level (2) - Example window = open(“videodevice”); movie = open(“file”); while (not EOF (movie) ) { read(movie, &LDU); if (LDU. time == 20) printf(“Subtitle 1”); else if (LDU. time == 26) printf(“Subtitle 2”); write(window, LDU); } close(window); close(movie); CS 598 KN - Fall 2018

Stream Level (1) Operates on continuous media streams and groups of streams n Models inter-stream synchronization for need of parallel presentation n Offers abstractions: n ¨ notion of streams, ¨ timing parameters concerning Qo. S for intra-stream and inter-stream synchronization CS 598 KN - Fall 2018

Stream Level (2) n Supports operations: ¨ Start(stream), stop(stream), create-group(list-of- streams); ¨ Start(group), stop(group); ¨ Setcuepoint(stream/group, at, event); n Classifies implementation according to ¨ Support for distribution (end-to-end, local) ¨ Support of type of guarantees (best effort, deterministic) ¨ Support of types of supported streams (analog, digital) CS 598 KN - Fall 2018

Object Level (1) n n Operates on all types of media and hides differences between discrete and continuous media Offers abstractions: ¨ Complete n n sync presentation Computes and executes complete presentation schedules that include presentation of noncontinuous media objects and calls to stream level Does not handle intra-stream and inter-stream synchronization ¨ (relies on media and stream levels) CS 598 KN - Fall 2018

Object Level (2) - Example n MHEG – Multimedia Hypermedia Experts Group of ISO Defines representation and encoding of multimedia and hypermedia objects (object-based declarative programming language) ¨ Provides abstractions suited to real-time presentations ¨ n ¨ Provides abstracts for real-time exchange n ¨ implemented with minimal buffering Evaluates status of objects and performs actions (e. g. , prepare, run, stop, destroy) n n n implemented via multimedia synchronization functionalities For time-dependent streams – access to stream level For time-independent streams – direct access the object to present it Classification of this level according to (a) distribution capabilities, (b) type of presentation schedule, (c) schedule calculation CS 598 KN - Fall 2018

Specification Level (1) n n Open layer included in tools which allow to create sync specifications Examples: Synchronization editors, document editors, authoring systems, conversion tools ¨ Examples of such tools: multimedia document formatter that produces MHEG specifications ¨ n Classification: Interval-based spec ¨ Time-axes based spec ¨ Control flow-based spec ¨ Event-based spec ¨ CS 598 KN - Fall 2018

Specification Level (2) - SMIL n n SMIL – World Wide Web Consortium’s Synchronized Multimedia Integration Language SMIL 2. 0 and SMIL 3. 0 (developed by CWI, PI: D. Bulterman) Features: Animations, Time Manipulations, Timing and Synchronization, Content Control, Meta-information, Media objects, Linking, Layout, Transitions http: //www. w 3. org/TR/SMIL 3/ CS 598 KN - Fall 2018

SMIL “Standards, SMIL 2. 0, Examples and Comparisons”, D. Bulterman, Siemens Corporate research, Editor: P. Liu, 2002 CS 598 KN - Fall 2018

Summary CS 598 KN - Fall 2018

MORE DETAILS ON SYNC SPECIFICATION (IF TIME PERMITS) CS 598 KN - Fall 2018

Requirements on Synchronization Specs n Object consistency and maintenance of sync specifications ¨ Media n n n objects should be kept as one LDU in spec Temporal relations must be specify-able Easy Description of Sync Relations Definition of Qo. S requirements Integration of time-dependent and independent media Hierarchical levels of synchronization CS 598 KN - Fall 2018

Sync Spec Models n n n Interval Timeline Flow-control Hierarchical ¨ Reference points ¨ n Event-based n Common threads ¨ provide language to express relationships ¨ runtime system to monitor relationships ¨ policies to enforce relationships CS 598 KN - Fall 2018

Interval-based Specification (1) Presentation duration of an object is specified as interval n Types of temporal relations: n ¨A before B, A overlaps B, A starts B, A equals B, A meets B, A finishes B, A while B n Enhanced interval-based model includes 29 interval relations, 10 operators handle temporal relations (e. g. , before(δ 1), …) CS 598 KN - Fall 2018

Interval Model (2) n 13 relationships between two intervals B A Before Starts B A B Meets Equal Ends A A B B Overlaps A While A A B B CS 598 KN - Fall 2018

Example (3) Audio 1 while(0, 0) Video Audio 1 before(0) Recorded. Interaction before(0) P 1 before(0) P 2 before(0) P 3 before(0) Interaction P 3 before(0) Animation while(2, 5) Audio 2 Interaction before(0) P 4 CS 598 KN - Fall 2018

Interval-based Specification (4) n Advantages: ¨ Easy to handle open LDUs (i. e. , user interactions) ¨ Possible to specify additional nondeterministic temporal relations by defining intervals for durations and delays ¨ Flexible model that allows specification of presentations with many run-time presentation variations CS 598 KN - Fall 2018

Interval-based Specification (5) n Disadvantages: ¨ Does not include skew spec ¨ Does not allow specification of temporal relations directly between sub-units of objects ¨ Flexible spec leads to inconsistencies n Example: A NOT in parallel with B A while(2, 3) I I before(0) B CS 598 KN - Fall 2018

Timeline Axis-based Specification Presentation events like start and end of presentation are mapped to axes that are shared by presentation objects n All single medium objects are attached to time axis that represents abstraction of real-time n This sync specification is very good for closed LDUs n CS 598 KN - Fall 2018

Timeline Model (2) n Uses a single global timeline n Actions triggered when the time marker reaches a specific point along timeline CS 598 KN - Fall 2018

Example (3) n Define a timed sequence of images, each image has a caption that goes with it I 1 I 2 I 3 C 1 C 2 C 3 t 1 t 2 t 3 CS 598 KN - Fall 2018

Example (4) n Rule language n Visual environment ¨ At (t 1), show (I 1, C 1) ¨ At (t 2), show (I 2, C 2) ¨ At (t 3), show (I 3, C 3) CS 598 KN - Fall 2018

Control Flow-based Spec - Hierarchical Model (1) n n Possibility to specify concurrent presentation threads at predefined points of presentation Basic hierarchical spec types: Serial synchronization ¨ Parallel synchronization of actions ¨ n Actions: atomic or compound Atomic action handles presentation of single media object, user input, delay ¨ Compound actions are combinations of sync operators and atomic actions ¨ Delay is atomic action – allows modeling of delays in serial presentations ¨ CS 598 KN - Fall 2018

Hierarchical Model (SMIL) n Based on sequential and parallel I 1 n I 2 I 3 I 1 T 1 Apply operators to only the start/end points of each media object CS 598 KN - Fall 2018

Example (3) n Narrated slide show ¨ image, text, audio on each slide ¨ select link to move to the next slide … S 1 A 1 T 1 I 1 S 2 A 2 T 2 I 2 CS 598 KN - Fall 2018

Example (4) (and Comparison with Intervalbased Spec) Audio 1 while(0, 0) Video Audio 1 before(0) Recorded. Interaction before(0) P 1 before(0) P 2 before(0) P 3 before(0) Interaction P 3 before(0) Animation while(2, 5) Audio 2 Interaction before(0) P 4 CS 598 KN - Fall 2018

Control Flow-based Spec – Hierarchy (5) n Advantages ¨ Easy to understand ¨ Natural support for hierarchies ¨ Integration of interactive object easy n Disadvantage ¨ Need additional descriptions of skews and Qo. S ¨ No duration description CS 598 KN - Fall 2018

Control Flow-based Spec – Reference Points (1) Time-dependent single medium objects are regarded as sequences of closed LDUs n Start/stop times of object presentation are reference points n Connected reference point is synchronization point n Temporal relations specified between objects without explicit reference to time n CS 598 KN - Fall 2018

Example (2) CS 598 KN - Fall 2018

Control Flow-based Spec – Reference Points (3) n Advantages: ¨ Sync at any time during presentation of objects ¨ Easily integrated object presentation with unpredictable duration ¨ Intuitive type of synchronization spec n Disadvantages: ¨ Not easy way to detect inconsistencies ¨ Cannot specify delays in presentation CS 598 KN - Fall 2018

Event-based Specification Presentation actions initiated by synchronization events n Example: n ¨ Start presentation ¨ Stop presentation ¨ Prepare presentation n Events initiating presentation ¨ External or internal CS 598 KN - Fall 2018

Event-based Spec n Advantage: ¨ Easily extended to new sync types ¨ Easy integration of interactive objects n Disadvantage: ¨ Difficult to handle in case of realistic scenarios ¨ Too complex specification ¨ Need separate description of skew/Qo. S ¨ Difficult use of hierarchies CS 598 KN - Fall 2018

Event Model (Nsync) Associate actions with expressions n Expressions may contain scalars, clocks, variables, relations, and connectives n When the expression becomes TRUE, invoke associated action n When “Time > Q. end + 5 && !Response” Answer=WRONG Source: B. Bailey et al. “Nsync- A Toolkit for Building Interactive Multimedia CS 598 KN - Fall 2018 Presentations”, ACM Multimedia 1998

Example: Delayed Transition Overview More Info? More Info Yes Detailed Narration CS 598 KN - Fall 2018 No More Info

Model Specification When “Narration >= Overview && !More. Info” Next. Slide When “Narration >= Overview && More. Info” Play. Details When “Narration >= Overview + Details” Next. Slide Narration: narration’s logical timeline Overview: normal transition point Details: additional narrative details More. Info: records kitchen info status CS 598 KN - Fall 2018

Reactive Interface CS 598 KN - Fall 2018

Model Specification When “Video >= 0 && Video < T 1” Select Kitchen When “Video >= T 1 && Video < T 2” Select Deck When “Video >= T 2 && Video <= T 3” Select Yard CS 598 KN - Fall 2018

Synchronization/Editing Tool Vegas Video http: //www. sonycreativesoftware. com/vegaspro CS 598 KN - Fall 2018

Conclusion Carefully analyze what kind of synchronization is needed in your multimedia system and application n Determine at which level you need synchronization n Determine what the synchronization requirements should be based on prior experiments n Understand timing model and sync model n CS 598 KN - Fall 2018