More Pixels and Samples High Resolution Media Streaming
- Slides: 34
More Pixels and Samples: High Resolution Media Streaming Roger Zimmermann Data Management Research Laboratory University of Southern California Los Angeles, CA 90089 http: //dmrl. usc. edu
Outline • Motivation • Background – Remote Media Immersion – Distributed Immersive Performance • High-performance Data Recording Architecture • Demonstration • Conclusions APAN, January 2004 Integrated Media Systems Center, USC
Motivation • The charter of the Integrated Media Systems Center (IMSC) is “Immersipresence” – Immerse real (e. g. people) and virtual elements into a common space • Becomes much more interesting in a distributed environment – Many sub-problems: tracking, gesture recognition, data management, … – Video and audio are an important component APAN, January 2004 Integrated Media Systems Center, USC
What is the problem? • Live streaming is either – Low to medium quality, or – Very expensive, i. e. , there are only a few people to call … • Other obstacles – Complicated (not like the telephone) – Often requires room engineering – Network bandwidth is not available • Some of the technical constraints can and will be solved APAN, January 2004 Integrated Media Systems Center, USC
Ex. : Network Infrastructure • UTOPIA (Utah Telecommunications Open Infrastructure Agency): public works project to provide fiber to the home (FTTH). • Super. Net, Alberta, Canada. Public project to provide a high speed Internet infrastructure. • NSF sponsored workshop, Oct. 23 -24, 2003, Chicago, Illinois. The importance of “broaderband” networks is recognized. APAN, January 2004 Integrated Media Systems Center, USC
Research Timeline 2002 Jun 2 -3 Unveiling of RMI Demonstration Oct 29 Internet 2 Meeting: RMI Demonstration Dec 28 DIP Experiment 1: Distributed Duet 2003 Jan 18 Recording from Stream Jan 19 DIP Experiment 2: Remote Master Class Jun 2 -3 DIP Experiment 3: Duet with Audience 2004 Jan 29 APAN Meeting: HYDRA Experiment APAN, January 2004 Integrated Media Systems Center, USC
What is the RMI? “The goal of the Remote Media Immersion system is to build a testbed for the creation of immersive applications. ” Immersive application aspects: 1. Multi-model environment (aural, visual, haptic, …) 2. Shared space with virtual and real elements 3. High fidelity 4. Geographically distributed 5. Interactive APAN, January 2004 Integrated Media Systems Center, USC
RMI Challenges n Immersive, high-quality video acquisition and rendering n High Definition video 1080 i and 720 p (40 Mb/s) n Immersive, high-quality audio acquisition and rendering n 10. 2 channels of uncompressed audio (12 Mb/s) n Storage and transmission of media streams across networks n Synchronization between streams (A/V, A/A, V/V)! APAN, January 2004 Integrated Media Systems Center, USC
RMI Architecture APAN, January 2004 Integrated Media Systems Center, USC
RMI Experimental Setup • Synchronized immersive audio and HDTV streamed playback from Yima server over Internet 2 – 16 channels of immersive audio, uncompressed at 16 Mb/s – 1920 x 1080 i HDTV content, MPEG-2 compressed at 40 Mb/s • Control of end-to-end process: capturing, network interface, transmission, rendering ISI East IMSC APAN, January 2004 Integrated Media Systems Center, USC
Internet 2 Fall ‘ 02 Member Meeting Video: HDTV 1280 x 720 p Audio: 10. 2 channel, immersive sound system New World Symphony, Miami, FL APAN, January 2004 Integrated Media Systems Center, USC
Distributed Immersive Performance • Outgrowth of Remote Media Immersion (RMI) – Create seamless immersive environment for distributed musicians, conductor (active) and audience (passive) – Compelling relevance for any human interaction scenario: education, journalism, communications • Scenario: – Orchestra not available in town – Famous soloist cannot fit travel into schedule – Multiple soloists in different places APAN, January 2004 Integrated Media Systems Center, USC
APAN, January 2004 Integrated Media Systems Center, USC
APAN, January 2004 Integrated Media Systems Center, USC
APAN, January 2004 Integrated Media Systems Center, USC
APAN, January 2004 Integrated Media Systems Center, USC
60 ms 20 ms 30 ms 40 ms 10 ms 30 ms Challenge: network latency APAN, January 2004 Integrated Media Systems Center, USC
• Key observations: – Network latency maps to audio delay on stage – Video delay is zero • Challenge: – Synchronization – Transmitting low latency video of conductor to players and audience – Maintaining constant delay between players Player 1 15 m: 45 ms Conductor Player 2 APAN, January 2004 10 m: 30 ms Integrated Media Systems Center, USC
Barriers and Requirements 1. Real-time continuous media (CM) stream transmission (network protocol) with low latency 2. Precise timing: GPS clock, synchronization 3. Data loss management: error concealment, FEC, retransmission, multi-path streaming 4. Many-to-many transmission capability 5. Low latency, high-quality real-time video and audio acquisition and rendering 6. Real-time CM stream recording 7. User experiments, requirements, specifications, performance evaluation APAN, January 2004 Integrated Media Systems Center, USC
Distributed Immersive Performance v. 1. 0 -The Duet • Experiments and Objectives • • – Experimental testbed and demonstration system – Demonstrate and document a distributed musical performance with two musicians (a duet) – Two-way interactive video and 10. 2 channel immersive audio capability – Explore other applications involving passive and active participants, such as two-site interactive meetings – Evaluate technical barriers and psychophysical effects of latency and fidelity on music and other forms of human interaction between two interconnected sites Dennis Thurmond - USC Thornton School of Music Elaine Chew - USC Industrial and Systems Engineering APAN, January 2004 Integrated Media Systems Center, USC
Distributed Immersive Performance v. 1. 0 -The Duet Linux PC DV Fire. Wire Camera 100 Base. T campus net 100 Base. T IMSC net 350 meters Ramo Hall of Music (RHM 106) • • Powell Hall (PHE 106) Video: NTSC resolution, 31 Mb/s DV, software decode, one-way latency: 110 ms due to DV camera compression + < 5 ms network Audio: uncompressed, 16 or more channels at 1 Mb/s each, one-way latency: < 10 ms due to audio processing + < 5 ms network APAN, January 2004 Integrated Media Systems Center, USC
Distributed Immersive Performance v. 1. 0 -The Duet APAN, January 2004 Integrated Media Systems Center, USC
HYDRA Streaming Architecture • Most previous work in streaming media has focused on the retrieval and playback functionality. • More and more devices directly output digital media streams: – E. g. , camcorders (Fire. Wire, USB, SDI), microphones (Bluetooth), mobile handsets (3 G) • Need for a backend data stream recording / playback system (“Super Ti. Vo”) Þ HYDRA (High-performance Data Recording Architecture) [ICEIS 2003] APAN, January 2004 Integrated Media Systems Center, USC
Challenges • Variable bit rate media streams • Multi-zoned disks • Different read and write transfer rates APAN, January 2004 Integrated Media Systems Center, USC
Live Streaming • Latency is a crucial limiting factor: – Only ~ 20 -40 ms is unnoticeable (for universal interactive applications) • Tradeoff: Latency versus bandwidth – Compression reduces bandwidth – But: high compression increases latency (e. g. , interframe MPEG compression) • Approach: – Perform experiments within this design space e. g. DV: NTSC resolution, 31 Mb/s, SW/HW codecs e. g. uncompressed audio and video APAN, January 2004 Integrated Media Systems Center, USC
Architecture HYDRA HD Live Streaming JVC HD 10 U Fire. Wire MPEG TS Extractor HD-SDI RTP/ UDP/IP VGA Display MPEG-2 Decoder • Acquisition and rendering PC are both Linux based (RH 9 includes kernel support for Fire. Wire). • MPEG transport stream extraction. • Data transport via UDP packets with single retransmissions APAN, January 2004 Integrated Media Systems Center, USC
Rendering • Solution 1: Software based rendering • Use X 11 hw acceleration: Xv. MC (libmpeg 2) – Motion compensation and i. DCT with GPU • Our hw: NVIDIA FX 5200 ($100) • Performance: ~ 90 fps @ 1280 x 720 with 3 GHz P 4 APAN, January 2004 Integrated Media Systems Center, USC
Rendering • Issues with software rendering – Precise timing: 29. 97 fps – Decoding time for I, P, and B frames varies – Buffering of decoded frames necessary to achieve precise timing – Transport stream splitter and audio decoding – Video card refresh rate (timing) is independent of MPEG timing, but • Non-standard display modes are possible: 720 p on Linux (16 x 9) – Decoding latency APAN, January 2004 Integrated Media Systems Center, USC
Rendering • Solution 2: Hardware based rendering • E. g. : Cine. Cast HD board from Vela Research – Digital HD-SDI and analog RGB/YPr. Pb outputs • Great and stable picture (but $$$) • Genlock input for synchronization APAN, January 2004 Integrated Media Systems Center, USC
Rendering • Issues with hardware rendering – Linux drivers hard to come by – Cine. Cast HD board uses SCSI interface • Wrote our own SCSI extensions to the Linux SCSI Generic driver (/dev/sg 0) – Decoding latency: requires 8 x 64 k. B to start decoding – Consumer HD card: Telemann Hi. Pix ($400) But: No Linux drivers (no Windows filters? ) – New Vela card: Cine. Cast HD LE APAN, January 2004 Integrated Media Systems Center, USC
Live HD Video Streaming (1280 x 720 p) APAN, January 2004 Integrated Media Systems Center, USC
Distributed Immersive Performance v. 2. 0 Extended Architecture • Conflicting requirements: Low latency and low bandwidth (i. e. , use of compression) • Solution - two-tier architecture: • Between performers – Low latency stereo audio streaming – Low latency video streaming • Between performers and audience – High definition video streaming – Multichannel audio streaming (10. 2 channel) • Recording of all streams sychronously for archival purposes and later playback. APAN, January 2004 Integrated Media Systems Center, USC
Multichannel audio Stereo audio Low latency, low resolution video High latency, high resolution video Performer 1 Performer 2 Playback and Recording Audience APAN, January 2004 Integrated Media Systems Center, USC
Thank You! Questions? • More info at: – Data Management Research Lab • http: //dmrl. usc. edu – Integrated Media Systems Center • http: //imsc. usc. edu • Acknowledgments: – Kun Fu, Beomjoo Seo, Shihua Liu, Dwipal A. Desai, Didi Shu-Yuen Yao, Mehrdad Jahangiri, Farnoush Banaei-Kashani, Rishi Sinha, Hong Zhu, Nitin Nahata, Sahitya Gupta, Vasan N. Sundar, APAN, January 2004 Integrated Media Systems Center, USC
- More more more i want more more more more we praise you
- More more more i want more more more more we praise you
- Scratch
- Scratch
- The scratch stage is 480 pixels wide and 360 pixels high
- The scratch stage is 480 pixels wide and 360 pixels high
- The scratch stage is 480 pixels wide and 360 pixels high
- 800 pixels wide and 200 pixels tall pictures
- Adjacency in image processing examples
- High resolution low resolution
- High resolution media
- Ll-hls
- Streaming media delivery
- What are plds in education
- Ball setting and streaming
- Sentiment analysis using pyspark
- Human history becomes more and more a race
- If an image contains k disjoint
- High-resolution terahertz
- High resolution smith chart
- High resolution site characterization
- Leader memeber exchange theory
- Mass spec of chlorine
- High resolution images
- Rotational state
- High resolution images
- Mirror
- High resolution images
- The prime of miss jean brodie streaming
- Dash dynamic adaptive streaming over http
- Alexis pribula
- Hadoop streaming python
- United streaming videos
- Streaming telemetry data
- Streaming simd extensions