Video Information Retrieval Mark Ruzomberka IST 497 110702

Video Information Retrieval Mark Ruzomberka IST 497 11/07/02

Joke

Outline z What is Video Information Retrieval (VIR) ? z Reasons VIR is necessary z Theoretical z Where we are today z Examples z Problems z Future Work z Conclusion

What is Video Information Retrieval (VIR) ? z Recognition technologies y Image y Voice y Text transcripts z Document retrieval technologies y Topic segmentation y Topic matching y Text summarization z Presentation Technologies y Combine Recognition and retrieval technologies z Result is an integrated application

VIR-Need, or Why do I care? z Consider the task of trying to find a five minute video clip of interest in a library of 1000 hour long tapes. z. Consider the “go to the part where” problem

What do people want from IR D-Lib Magazine’s asks: “What do People want from Information Retrieval? ” # 8 Multimedia

Specificly, Reasons for Video IR z Reading is slow compared to your potential for understanding information z Humans think in pictures not words z Reading is particularly slow on a computer screen z Example: Daydreaming while some one is talking z Reading a page in a book and not remembering what it was about

VIR makes for quicker human understanding. z z z z Palm/Grafitti Hand Writing Typing Speaking Reading Listening Thinking 25 35 -40 50 -70 135 -175 200 400 - 500+ • Video IR allows for faster access to information

Theoretical: z Think of the “Jetsons mail system” z You “talk” to the computer, z Computer intelligently “talks” back to you

Where we are today z. Two of Video Information Retrieval System are currently available: z. Type One- keyword/text based z. Type Two- Content based

Type One- keyword/text based • DVR- basic expansion of image IR, • not as interesting

Type Two- Content based Video Mail Informedia MSR Video Skimmer

Example: Video Mail z University of Cambridge z 1994 -1996 z AT&T z 1999 z 2000 -project ended

Video Mail: Medusa network z z Medusa multimedia environment at Olivetti Research Ltd. In Cambridge It takes a modular approach unlike that of a pc or workstation Unified by a common interface to ATM network Devices plug directly into network and include: y y y Cameras Audio devices Networked frame buffers Processor farms Disk drives

Video Mail: Medusa Network z “The network is the computer” metaphor is used z Solves storage and network speed problems z Complicates expense problem

How it works-Overview

The Integrated Application z“narrow” by sender, date, time

Video Mail: Video Browser z. Content is now being viewed z. Keywords are flagged

Video Mail: Video Browser z. In the latest version “thumb-nailed” pictures of key frames replace color coded line of the search keyword

Informedia The Informedia Digital Video Library Project automatically combines speech, image and natural language understanding to create a full-content searchable digital video library.

Informedia

Informedia: human factor issues z Interaction z Motivation z Effective usage modes z Commercial compression z VHS quality playback. z Terabyte (1, 000 gigabytes) of storage z 1000 hours of video.

Problems 1. Human understanding 2. Spoken document retrieval 3. Poor video browsers 4. Expensive 5. Slow access to data 6. Large amounts of data

Microsoft Research (MSR) Video Skimmer

Microsoft Research (MSR) Video Skimmer z Enhanced Browser Controls: y. Time Compression y. Pause Removal y. Textual Indices: x. TOC, Notes y. Visual Indices x. Shot Boundary Frames x. Timeline Markers y. Jump Control (Back/Next)

Problem: Poor Content Based Video Browsers z Current VCR model allows for poor navigation z “go the part where they say” problem

Problem: Expensive z Hard drive space expensive Per Year Drive Size Drive Cost MB/ Cost y. Video adds to problem z High bandwidth needs are also expensive 1956 5 megabytes 50, 000. 00 1980 26 megabytes 5, 000. 00 193. 00 1985 10 megabytes 710. 00 71. 00 1989 40 megabytes 1, 199. 00 36. 00 1995 1. 2 gigabytes 680. 00 68. 60 2000 30. 0 gigabytes 249. 99 0. 96 • http: //www. littletechshoppe. com/ns 1625/winchest. html

Problem: Slow Access to Data z. Broadband still not available everywhere z. Availability doesn’t mean acceptance z. Especially after dot com crash 2000

Problem: Large Amounts of Data z. Current Systems use MPEG 2 z. Newer compression technologies y. MPEG 4 -DIVX -DVD Quality z. Video consumes orders of magnitude more storage than text z. MPEG 7 is on horizon

Future Work ? z. Sky the limit ? z. Sci-Fi the limit ? z. Hard Drive Space, Bandwidth are current limitations.

Conclusion z Not yet ready for prime time z Storage and Network Costs decreasing z Success is in day to day usage z Slowly Becoming Mainstream E. x. Tivo z Problems of “real world tests” y. Idiot proof y. ATM and Medusa aren’t mainstream

Papers z z z z Video Mail Retrieval Using Voice: Report on Keyword. . - Jones, Foote, Jones. . (1994) What do people want from Information Retrieval? . Croft, Bruce W. D-Lib Magazine. (1995) Video Skimming for Quick Browsing based on Audio and Image. . - Smith, Kanade (1995) The VISION digital video library (context) - Gauch, Li et al. – (1997) Informedia: News-on-Demand Multimedia Information. . - Hauptmann, Witbrock (1997) M. G. Christel and D. J. Martin, "Information Visualization within a Digital Video Library", J. Intelligent Info. Systems 11(3), (1998), pp. 235 -257 Browsing Digital Video. Li, Gupta, Sanocki et. Al.

Questions?

Joke? z"There are 10 types of people in the world. . . zthose who understand binary and those who don't. "