Dali Virtual Machine Wei Tsang Ooi With Brian
Dali Virtual Machine Wei Tsang Ooi (With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)
Motivations z. We know how to encode, transmit and store multimedia data today z. People start looking into ways to process multimedia data
Video Editing z. Concat, cross-fade, overlay, cut and paste
Gateway z. Convert from one format to another z. Change compression rate input video (format A) output video transcoder format B
DBMS “ Find all movies in the database that contain this image “
Set-top Boxes z. Subtitling and close captioning z. Mixing of incoming movies
Lecture Browser z. Match video to slide z. Switch between video streams
Current Solutions z. Black box C code (mpeg_play) y. Hard to reuse/adapt/break apart y. Unpredictable performance z. Standard library (PPM, IJG) y. Least common denominator (RGB for all) x. Video frames --> RGB --> gray
History z. Rivl - high level scripting language for multimedia processing z. Still suffers from “black box” problem
Experience z. Real cost of processing y. Inherently complex operations x. Projective transform y. Layered operations x. Video decoding y. Simple operations but lots of data x. Copy
Dali Architecture User Dali Code Dali Compiler Expert User DVM Code Dali Virtual Machine MMX C Tri. Media
Today’s Talk DVM Code Dali Virtual Machine
DVM Code : Design Goal z. Small number of composable abstractions z. Simple, predictable opcodes z. High performance z. Target for compiler z. Easy to extend
Abstractions z. Byte image z. Bitstream z. MPEG z. JPEG z. Bit masks z. PCM
Abstractions z. Byte image z. Bitstream z. MPEG z. JPEG z. Bit masks z. PCM
Byte Image z. Two dimensional array of byte z. Can be either physical or virtual physical image virtual image
Byte Image Can Represent z. Gray scale image z. RGB image z. YUV image z. Alpha channel
Example Code : Fractal
image newh smaller neww smaller = byte_new (neww, newh); byte_shrink_2 x 2(image, smaller);
dx image target 2 dy target = byte_clip(image, 0, 0, dx, 2*dy); byte_set(target, 0); target = byte_clip(image, 3*dx, 0, dx, 2*dy); byte_set(target, 0);
target image dx smaller newh neww target = byte_clip(image, dx, 0, neww, newh); byte_copy(smaller, target);
dx image 2 dy newh smaller neww target = byte_clip(image, 2*dx, 2*dy, neww, newh); byte_copy(smaller, target); target = byte_clip(image, 0, 2*dy, neww, newh); byte_copy(smaller, target);
DVM Code Dx = 0. 25 * byte_width(image); dy = 0. 25 * byte_height(image); neww = 0. 5 * byte_width(image); newh = 0. 5 * byte_height(image); smaller = byte_new (neww, newh); for (i = 0; i < n; i++) { byte_shrink_2 x 2(image, smaller); Target = byte_clip(image, 0, 0, dx, 2*dy); byte_set(target, 0); target = byte_clip(image, 3*dx, 0, dx, 2*dy); byte_set(target, 0); target = byte_clip(image, dx, 0, neww, newh); byte_copy(smaller, target); target = byte_clip(image, 2*dx, 2*dy, neww, newh); byte_copy(smaller, target); target = byte_clip(image, 0, 2*dy, neww, newh); byte_copy(smaller, target); }
General Dali Strategies z. Specific instruction ybyte_shrink_2 x 2, byte_shrink_4 x 4, etc z. Explicit memory allocation ybyte_new z. Reduce data ybyte_clip z. Composable abstraction ybyte image
Performance run fractal with n = 4 on 800 x 600 gray scale image about 21 frames/sec for 320 x 240 video
Abstractions z. Byte image z. Bitstream z. MPEG z. JPEG z. Bit masks z. PCM
MPEG z. Getting important and pervasive z. Complex format z. Complex code (mpeg_play) z. Most decoders provides Get. Next. Frame() interface z. Random access, direct transfer difficult
Abstraction for MPEG Video seq hdr gop gop hdr gop . . . seq end
Abstraction for MPEG Video seq hdr pic hdr gop hdr pic gop . . . seq end
DVM Code Examples z. Skip to the n-th frame for (i = 0; i < n; i++) { mpeg_pic_hdr_find(bs); mpeg_pic_hdr_skip(bs); }
DVM Code Examples z. Skip to the n-th frame for (i = 0; i < n; i++) { mpeg_pic_hdr_find(bs); mpeg_pic_hdr_skip(bs); }
DVM Code Examples z. Skip to the n-th frame for (i = 0; i < n; i++) { mpeg_pic_hdr_find(bs); mpeg_pic_hdr_skip(bs); }
DVM Code Examples z. Skip to the n-th frame for (i = 0; i < n; i++) { mpeg_pic_hdr_find(bs); mpeg_pic_hdr_skip(bs); }
Concat 2 Video Sequences inbs 1 hdr gop inbs 2 hdr gop outbs hdr gop
DVM Code z. Concat two sequences While not end_of(inbs 1) { gop_hdr_dump(inbs 1, outbs); gop_dump(inbs 1, outbs); } While not end_of(inbs 2) { gop_hdr_dump(inbs 2, outbs); gop_dump(inbs 2, outbs); }
Performance Analysis z. Decode MPEG sequences to PPM z. No output z. Run on 3 different sequences
Performance
Performance
MPEG Strategies z. Expose structure y. Gop header, gop instead of just frames z. Break up layer y. Parse, decode, color space conversions
Show Time ! z. Decode a MPEG sequence z. For each frame scale it to “stamp” size z. Create a contact sheet
Abstractions z. Byte image z. Bitstream z. MPEG z. JPEG z. Bit masks z. PCM
What Is Bitstream ? z. Skip to the n-th frame for (i = 0; i < n; i++) { mpeg_pic_hdr_find(bs); mpeg_pic_hdr_skip(bs); }
Bitstream chunk of memory 1011001 parser
Mode of Operations z. File i/o static memory parser fread/fwrite
Mode of Operations z. Network static memory parser socket
Mode of Operations z. Memory map entire file parser
mpeg system stream inbs audio video
mpeg system stream inbs audio video filter videobs audio video
mpeg system stream inbs audio video filter videobs parser audio video
Bitstream Strategies z. Predictable performance y. Explicit I/O
Recap: Current Solutions z. Black box C code (mpeg_play) y. Hard to reuse/adapt/break apart y. Unpredictable performance
Recap: Cost of Processing z. Inherently complex operations z. Layered operations z. Simple operations but lots of data
How We Solve the Problems z. Difficult to reuse/adapt/break apart y. Minimum set of abstractions y. Abstractions are simple y. Simple reusable opcodes z. Unpredictable performance y. Explicit I/O y. Explicit memory allocations
How We Solve the Problems z. Inherently complex operations y. Special case y. Clipping and masking z. Layered operations y. Expose structure z. Simple operations but lots of data y. Special case y. Clipping and masking
Work in Progress z. Extending with new formats : JPEG, WAV, AVI etc z. Tri. Media chip implementation z. Application : lecture browser
Future Work z. Multithreading z. MMX implementation z. Compiler
- Slides: 56