Components of Multimedia Capture Multimedia Authoring Multimedia Production

§ § § Components of Multimedia Capture Multimedia Authoring Multimedia Production Multimedia Presentation Multimedia compression

§ Multimedia items generally fall into one of five main categories and use varied techniques for digital formatting. Text § Text in multimedia systems can express specific information, or it can act as reinforcement for information contained in other media items. Eg. Web pages may include image elements and a short amount of text as an alternative, in case the digital image item is not available. § Source: keyboard, speech input, optical character recognition, data stored on disk. § Stored as 1 byte per char § Format: Raw text or formatted text e. g HTML, Rich Text Format (RTF),

Graphics § Is a digital representation of information such as a drawing, a chart, or a photograph. § First media used to enhance the originally text-based Internet. § Format: Common graphical formats on the Web are JPEG and GIF. § Other graphical formats such as BMP and TIFF have larger file sizes, and may require special viewer software to display on the Web. § To reduce download times for graphics, some Web sites use thumbnails, which is a smaller version of a larger graphical image that a user may click to display the full sized image.

Graphics… § Input: Graphics are usually generated by a graphics editor program (e. g. Illustrator) or automatically by a program (e. g. Postscript). § Graphics are usually editable or revisable (unlike Images). § Graphics files usually store the primitive assembly and do not take up a very high storage overhead.

Images § Still pictures which (uncompressed) are represented as a bitmap (a grid of pixels). § Input: digitally scanned photographs/pictures or direct from a digital camera. May also be generated by programs similar" to graphics or animation programs. § Stored at 1 bit per pixel (Black and White), 8 Bits per pixel (Grey Scale, Colour Map) or 24 Bits per pixel (True Colour) § Size: a 512 x 512 Grey scale image takes up 1/4 MB, a 512 x 512 24 bit image takes 3/4 MB with no compression. § This overhead soon increases with image size. Compression is commonly applied.

Audio § Can be music, speech, or any other sound. § Common audio formats include WAV, MID, and MP 3. § Audio signals are continuous analog signals. § Some Web sites use streaming audio, which allows a user to listen to the sound as it downloads to the computer. Two accepted standards for streaming audio on the Web are Windows Media Player and Real. Audio. § Input: microphones and then digitised and stored § CD Quality Audio requires 16 -bit sampling at 44. 1 KHz. Even higher audiophile rates (e. g. 24 -bit, 96 KHz)

§ Q. 1: What is the bit-rate (bps, bits per second) of the digitized audio using PCM coding? E. g. : CD. § Sampling frequency is F=44. 1 KHz § (Sampling period T=1/F=0. 0227 ms) § Quantization with B=16 bits (N=216=65, 536). § Bit rate = BXF = 705. 6 Kbps = 88. 2 KBytes/s

Video § Consists of full-motion images that are played back at various speed. Most video is also accompanied with audio. § Input: Analog Video is usually captured by a video camera and then digitised. § MPEG is a popular video compression standard defined by the Moving Picture Experts Group (MPEG). Other formats include Flash, AVI, WMV and Quick. Time § Raw video can be regarded as being a series of single images. § There are typically 25, 30 or 50 frames per second. § Streaming video allows a user to view longer or live video images as they download to the computer from the Web.

What is animation? § Is the appearance of motion that is created by displaying a series of still images in rapid sequence. § Animated GIF is a popular type of animation format, which combines several images into a single GIF file.

§ The capture process converts various electronic media forms into digital representations of the original media and stores them as computer files. § Examples of these media might include a television camera's image of a flower, a scanner's image of a document, a microphone's audio signal of human speech, and a compact disc's audio output of a symphony. § The ACPA (Audio Capture and Playback Adapter) , VCA (Video Capture Adapter) and the AVC (Audio Visual Connection) software perform this conversion, and accept files created by certain external programs as well.

§ Additional equipment, such as a video camera, video monitor, video cassette recorder, compact disc player, digital audio tape recorder, microphone, audio amplifier, and speakers complete the multimedia capture system. § Knowledge of the audio and video equipment, lighting techniques, and methods of recording a noise-free audio signal are essential for the successful capture of a multimedia production. Multimedia Data basics § Multimedia systems/applications have to deal with the: Generation of data, Manipulation of data, Storage of data, Presentation of data, and Communication of information/data

Multimedia Data basics. . § Recall from our definition of multimedia; All data must be in the form of digital information and data may be in a variety of formats: text, graphics, images, audio and video. § A majority of this data is large and the different media may need synchronisation: § Static or Discrete Media : Some media is time independent: § Normal data, text, single images, graphics are examples. § Continuous Media : Time dependent Media: § Video, animation and audio are examples. § The world we sense is full of analog signals:

Multimedia Data basics. . § Analog: continuous signals must be converted or digitised for computer processing. § Digital: discrete digital signals that computer can readily deal with. § Analog-to-Digital Converter (ADC): Take analog signals from analog sensor (e. g. microphone) and digitally sample data

Multimedia Data basics. . § Digital-to-Analog Converter (DAC): Playback - a converse operation to Analog-to-Digital § Takes digital signal, possible after modification by computer (e. g. volume change, equalisation) § Outputs an analog signal that may be played by analog output device (e. g. loudspeaker, RGB monitor/display)

Analog-to-Digital-to-Analog Pipeline § Begins at the conversion from the analog input and ends at the conversion from the output of the processing system to the analog output as shown:

Analog-to-Digital-to-Analog Pipeline. . § Anti-aliasing filters (major part of Analog Conditioning) are needed at the input to remove frequencies above the sampling limit that would result in aliasing. § The anti-aliasing filter at the output removes the aliases that result from the sampling (sample theorem) § After the anti-aliasing filter, the analog/digital converter (ADC) quantises the continuous input into discrete levels. § After digital processing, the output of the system is given to a digital/analog converter (DAC) which converts the discrete levels into continuous voltages or currents. This output must also be filtered with a low pass filter to remove the aliases from the sampling.

§ The authoring process establishes a set of procedures called a story that controls the multimedia presentation. § The story defines the sequence and methods in which the various media formats are displayed or heard

§ There basically six stages: Planning Meeting § It creates a shared vision for everyone working on the project. During this, the project manager communicates the major goals and lays out the milestones. Script Writing § After the initial meeting, the people in charge of the background story write a script, creative brief or outline. Story Boarding § A multimedia project usually includes multiple pieces: audio, video, imagery, text for voiceovers and on-screen titles. Story boarding ties everything together

Designing § Designers take over the visual aspects of the project to determine how it looks and feels. § Using the notes from the storyboard, they create graphics, design the navigation and give direction to photographers and videographers regarding the correct shots. Editing § Is one of the most involved and complex stages of the multimedia development process. § This stage usually involves internal review iterations and may also include rounds of client review and editing.

Production § This is when all the parts of a multimedia project come together. § The production staff gathers all of the edited assets in one place and puts them together in a logical sequence, using the story board as a guide. § During this stage, test members of the audience use the multimedia piece while team members observe. After user testing, there are usually further adjustments to the project.

§ Graphics Styles: Human visual dynamics impact color schemes and lettering. § Color principles and guidelines: Some color schemes and art styles are best combined with a certain theme or style (hint: not to use too many colors) Some color combinations are more pleasing than others § Fonts: For effective visual communication in a presentation, it is best to use large fonts (i. e. , 18~36 points), and < 6~8 lines per screen (fewer than on this screen!). § A color contrast program: If the text color is (R, G, B), a legible background color is (R, G, B) (1 - R, 1 - G, 1 - B)

§ The need for compression: Raw video, image, and audio files can be very large. § Relying on higher bandwidths is not a good option. Traffic will always increase to fill the current bandwidth limit whatever this is. § Compression HAS TO BE part of the representation of audio, image, and video formats. Basics of Information Theory § Suppose we have an information source (random variable) S which emits symbols {s 1, s 2, . . . , sn} with probabilities p 1, p 2, . . . , pn. According to Shannon, the entropy of S is defined as:

Basics of Information Theory. .

Entropy ex:

Entropy ex. .

Entropy ex. .

Entropy ex. .

Compression basically employs redundancy in the data: § Temporal in 1 D data, 1 D signals, audio, between video frames etc. § Spatial correlation between neighbouring pixels or data items. § Spectral e. g. correlation between colour or luminescence components. This uses the frequency domain to exploit relationships between frequency of change in data. § Psycho-visual exploit perceptual properties of the human visual system.

Compression can be categorised in two broad ways: § Lossless Compression: after decompression gives an exact copy of the original data. § Example: Entropy encoding schemes (Shannon. Fano, Huffman coding), arithmetic coding, LZW algorithm (used in GIF image le format). § Lossy Compression: after decompression gives ideally a “close” approximation of the original data, ideally perceptually lossless. § Example: Transform coding - FFT/DCT based quantisation used in JPEG/MPEG differential encoding, vector quantisation.

§ Lossy methods are typically applied to high resolution audio, image compression. § Have to be employed in video compression (apart from special cases). Basic reason: § Compression ratio of lossless methods (e. g. Huffman coding, arithmetic coding, LZW) is not high enough for audio/video. § By cleverly making a small sacrifice in terms of fidelity of data, we can often achieve very high compression ratios. § Cleverly = sacrifice information that is psychophysically unimportant.

§ Repetitive Sequence Suppression. § Run-Length Encoding (RLE). § Pattern Substitution. § Entropy Encoding: § Shannon-Fano Algorithm. § Huffman Coding. § Arithmetic Coding. § Lempel-Ziv-Welch (LZW) Algorithm.