Multimedia Continuous Media Audio Animation Video 1 T

Multimedia Continuous Media: Audio, Animation, Video 1 T. Sharon-A. Frank

Sound • Conversion of energy into vibrations in the air. • Most sound sources vibrate in complex ways leading to sounds with components at several different frequencies. • Frequency spectrum – relative amplitudes of the frequency components. • Range of human hearing: roughly 20 Hz– 20 KHz, falling off with age. 2

Use of Audio 1. Convey Information 2. Set a mood 3. Capture attention 4. Explain a process 5. Provide personal contact 6. Tell a story 3 T. Sharon-A. Frank

Nature of Sound • Sound is produced by the vibration of matter. • Pressure variations are created in air surrounding matter. • Pattern of oscillation is called a waveform. 4 T. Sharon-A. Frank

Waveforms • Sounds change over time – e. g. musical note has attack and decay, speech changes constantly. • Frequency spectrum alters as sound changes. • Waveform is a plot of amplitude against time: – Provides a graphical view of characteristics of a changing sound. – Can identify syllables of speech, rhythm of music, quiet and loud passages, etc. 5

Basic Sound Concepts • Waveform: – Period – Frequency – Amplitude • Computer Representation of Sound: – Sampling – Quantization 6 T. Sharon-A. Frank

Period • Repeating portion of the sound wave, although sound waves are never perfectly smooth or uniformly periodic. 7 T. Sharon-A. Frank

Frequency • Reciprocal of the period, that is, number of periods in a second; measured in hertz (Hz) or cycles per second. • Human hearing frequency range is from 20 Hz to 20 KHz: – Infrasound: 0 - 20 Hz – Ultrasound: 20 KHz - 1 GHz – Hypersound: 1 GHz - 10 THz 8 T. Sharon-A. Frank

Amplitude • Measure of displacement of the air pressure wave from its mean. • Subjectively heard as loudness. amplitude 9 T. Sharon-A. Frank

Sound Wave Example 10 T. Sharon-A. Frank

Idea of Audio Digitization • Smooth continuous curve of waveform is not directly represented in computer. • Amplitude of waveform is measured and recorded at regular time intervals process is called encoding. • Each interval is called a sample. 11 T. Sharon-A. Frank • Need Analog-to-Digital Converter (ADC).

Audio Digitization • Converting signal from analogue to digital form – Analogue signal can vary continuously; digital is restricted to discrete values. • Two-stage process: – Sampling – measure the value at discrete intervals. – Quantization – restrict the value to a fixed set of quantization levels. 12

Digital Sampling Sample Height Samples Time Samples 13 T. Sharon-A. Frank

Digital Quantization Sample Height Example: 3 bit quantization Samples 14 T. Sharon-A. Frank

Audio Encoding (a) A sine wave. (b) Sampling the sine wave. (c) Quantizing the samples to 4 bits. 15 T. Sharon-A. Frank

Process of Sampling Audio 16 T. Sharon-A. Frank

Sampling Rate • Number of samples per second measured in hertz (Hz). • More samples = higher quality = bigger files. • Example: – CD quality = 44, 100 Hz (44. 1 KHz) – Others: 8, 000 Hz, 11, 025 Hz, 22, 050 Hz 17 T. Sharon-A. Frank

Sampling Size • • • Number of bits used to represent each sample. Called also Bit Depth and Sound Resolution. . (8 -bit - telephone quality (typical of Web. 16 -bit - CD quality Examples: 16 bit, 44. 1 k. Hz 18 8 bit , 44. 1 KHz T. Sharon-A. Frank Web quality (11. 025 KHz, 8 bit)

Channels • Audio files can support from 1 to 6 channels of audio information. • Most common: – mono (1 channel) – stereo (2 channels) • 4 -6 channels are usually used for surround sound. 19 T. Sharon-A. Frank

Record Settings Menu for Audio 20 T. Sharon-A. Frank

Sample Audio File Formats • • 21 aif/aiff wav au mov MPEG/MP 3 (MPEG 1 Level 3) RA/RAM MIDI (Musical Instrument Digital Interface) T. Sharon-A. Frank

Audio File Formats (platform specific) • aif (aiff) – Audio Interchange File Format – original audio format for Mac – supported by Windows and other platforms • wav – Waveform Audio File Format – original Windows audio file format – supported on Mac • au – u-law (myoo-law) – Unix standard audio format 22 T. Sharon-A. Frank

Audio File Formats (container formats) • mov (qt) – Quick. Time, not just Apple/MAC • mpg (mpe, mpeg) – Moving Pictures Expert Group – High quality, excellent compression file format – Files tend to be small and sound good • ra (ram) – Real. Audio 23 T. Sharon-A. Frank

Real Audio Format • First example of streaming audio on the Web. • Quality like that of an AM radio station. • Best used for narration and not for music or other sounds. • Must have Real. Player on your system to listen to files. 24 T. Sharon-A. Frank

MIDI • MIDI = Musical Instruments Digital Interface • Instructions about how to produce music, which can be interpreted by suitable hardware and/or software. • Standard protocol for communicating between electronic instruments (synthesizers, samplers, drum machines. ( • Allows instruments to be controlled by hardware or software sequencers. 25

MIDI and Computers • MIDI interface allows a computer to send MIDI data to instruments. • Store MIDI sequences in files, exchange them between computers, incorporate them into multimedia. • Computer can synthesize sounds on a sound card, or play back samples from disk in response to MIDI instructions 26 – Computer becomes primitive musical instrument (quality of sound inferior to dedicated instruments).

Uncompressed Audio Digital Telephony • Sampled at 8 KHz (8 K cycles per second) • Quantized to 8 bits per sample (1 Byte) => 64, 000 bits/s • Bandwidth = 64, 000/1, 024 = Total: 62. 5 Kb/s 27 T. Sharon-A. Frank

Uncompressed Stereo Audio CD Quality • • 28 Sampled at 44. 1 KHz (44. 1 K cycles per second) Quantized to 16 bits per sample (2 bytes) Two channels for stereo Bandwidth = 2 * 44, 100 * 16 = 1, 411, 200 b/s 1, 411, 200/1024= 1, 378 Kb/s Total: ~1. 4 Mb/s T. Sharon-A. Frank

Animation • The creation of moving pictures one frame at a time: – Literally 'to bring to life‘. – e. g. make a sequence of drawings on paper, in which a character's position changes slightly in each drawing. – Photograph drawings in sequence, using movie camera that advances one frame at a time. – Play back resulting film – character will move. 29

Use of Animation 1. Capture the viewer’s attention, entertain, or promote. 2. Explain a system or process. 3. Set a mood or create an on-screen environment. 30 T. Sharon-A. Frank

Animation Principles • Persistence of vision: – object seen by human eye remains mapped on retina for a brief time after viewing. – display series of images rapidly and they blend together to create illusion of movement. – requires at least 16 frames per second (fps) to look seamless. • Television: displayed at 30 frames per second. • Movies: displayed at 48 frames per second. 31 T. Sharon-A. Frank

Cell Animation (Frame Animation) • Start with first and last image in motion (called keyframes). • Draw images between keyframes (process called tweening/morphing) - small changes between images. • Images are then layered onto background scene. • Images displayed rapidly – 15/24/30 frames per second. 32 T. Sharon-A. Frank

Captured Animation 33 • Logic and procedural concepts are same as in cell animation (keyframe, tweening, layering techniques. ( • But use computer and video camera. • Frame grabbing – record each frame to disk. • Edit non-linearly like for video. • Can also use scanner or digital still camera, or create each frame in a graphics program (e. g. , Painter. (

Cell-based Techniques • Morphing: – uses frames to create illusion of one object changing into another – more keyframes = smoother morph. • Need at least 15 frames per second. • Professional quality uses 24 -30 frames per second. • Cell-based animation is done more easily on computer than by hand. • Easier to create and process than movies. 34 T. Sharon-A. Frank

Path-based Animation (Vector Animation) • Create animated objects by following object’s transition over a line or vector. • Artist creates one drawing and a path. • Computer program manipulates object by drawing frames as object travels over the path. 35 T. Sharon-A. Frank

Sample Animation File Formats • • 36 Animated GIF mov (Quick. Time) SWF (Flash) PNG (Portable Network Graphics) T. Sharon-A. Frank

Animated GIFs • Sequence of images can be stored in a single GIF file, and displayed one after another by a Web browser or other software: – No browser plug-in required – Can specify looping, delay between frames – 256 colour palette – No sound. 37

SWF • Popular Web animation format. • Usually generated by Macromedia Flash. • Vector animation format – Motion represented as numerical operations on vector data. • Can also include bitmapped images (e. g. , as backgrounds). 38

Video • Video is sequence of images displayed at constant rate. • Rate is in frames per second (fps). • Needs around 10 -15 fps for smooth motion. • For example, video is displayed at 24 -30 fps. 39 T. Sharon-A. Frank

Use of Video 1. Tell a story 2. Entertain an audience 3. Explain a complex process 4. Personalize a Multimedia experience 40 T. Sharon-A. Frank

Movie - Characteristics • Built from Frames - Each frame is an Image – Image characteristics (i. e. , resolution, colors) Clip Scene Shot Frame 41 T. Sharon-A. Frank

A movie may consist of several files 42 T. Sharon-A. Frank

Sample Video File Formats • • 43 MOV AVI RA/RAM ASF WMV MPEG 1/2/4 FLV T. Sharon-A. Frank

MPEG • • MPEG = Moving Picture Experts Group. Family of compression algorithms. High compression rate. Algorithm stores only changes from one frame to another rather than each entire frame. • Video info then encoded using DCT. • Lossy compression but loss of data is generally imperceptible to human eye. 44 T. Sharon-A. Frank

MPEG Video Format • ISO/IEC MPEG (Moving Picture Experts Group) Series of standards including: 1. MPEG-1: used for video CD 352 x 240 @30 frames/second. 2. MPEG-2: used for compressing DVD and high-definition television (HDTV). 3. MPEG-4: used to transmit audio, video, and graphics at low bitrate – can be delivered over very slow connections (56 Kbps). 45 T. Sharon-A. Frank

Video Acquisition • TV set and PC monitor look similar. • TV is analog – Gets info from continuously varying broadcasting waves. • PC is digital – Data displayed is digital. • Digitization performed by video capture card. • Typical card is combination of hardware and software. • Capture card typically creates uncompressed files that then must be passed into a codec chip or software. 46 T. Sharon-A. Frank

Issues Affecting Video File Size • Movie Length - simple principle: shorter video clips result in smaller files. • Frame Size - full-screen video is 640 x 480. Typical frame size on WWW = 160 x 120 although can handle 320 x 240. • Frame Rate: – TV quality = 30 fps – Web = 15 (or even 10) fps. 47 T. Sharon-A. Frank

Video Presentation • TV – PAL – NTSC – SECAM – HDTV • Computer Display 48 T. Sharon-A. Frank

TV Standards • NTSC (National Television Standard Committee) – Used by TV and video industries in US and Japan Taiwan, parts of South America, … • PAL (Phase Alternating Line) – Used in Western Europe, Australia & New Zealand, China , … • SECAM (Sequential Color and Memory) 49 – France and former Soviet Union – Standard only used for transmission – Very different technology than PAL.

Video (PAL, HDTV) Bandwidth • • 50 PAL Resolution: 640 x 480 3 bytes per pixel HDTV bandwidth increased by factor Frame rate: 25 fps of 5. 33: ~936 Mb/s Bandwidth: 640 * 480 * 25 * 24 = 184, 320, 000 b/s Total: ~176 Mb/s T. Sharon-A. Frank

More PAL and NTSC Examples • PAL uncompressed – 768 x 576 pixels per frame • x 3 bytes per pixel • x 25 frames per second ≈ 31 MB per second ≈ 1. 85 GB per minute 51 • NTSC uncompressed – 640 x 480 pixels per frame • x 3 bytes per pixel • x 30 frames per second ≈ 26 MB per second ≈ 1. 6 GB per minute