Introduction l Multimedia initial focus multimedia communications multimedia

Introduction l Multimedia initial focus Ø multimedia communications, multimedia authoring and presentation Ø digital multimedia data: text, image, video & audio l Goal: efficient and effective storage and retrieval of multimedia information l DBMSs cannot handle multimedia data effectively Ø differences between the characteristics of alphanumeric data and multimedia data l Required: new multimedia indexing & retrieval techniques Multimedia Database Management Systems (MMDBMSs) l overview of multimedia indexing and retrieval systems (MIRSs). l capabilities and common applications of MIRSs Ming-Yen Lin, IECS, FCU 1

Outline - Introduction l Definitions l Need for MIRS 2 Ming-Yen Lin, IECS, FCU

Definitions l Media Types & Multimedia l Databases & DBMSs l Text Document Information Retrieval l Multimedia Indexing & Retrieval l Feature Extraction, Content Representation & Indexing 3 Ming-Yen Lin, IECS, FCU

Definitions - Media l Media: types of information or types of information representation Øalphanumeric data, images, audio, and video l classify media: based on physical formats and media relationships with time Øbased on whether or not there are time dimensions Øtwo classes of media n static ndynamic (or time continuous) 4 Ming-Yen Lin, IECS, FCU

Classes of Media l Static media Ø do not have a time dimension Ø their contents and meanings do not depend on the presentation time Ø alphanumeric data, graphics, and still images l Dynamic media Ø have time dimensions Ø their meanings and correctness depend on the rate at which they are presented Ø animation, audio, and video n intrinsic unit intervals or rates (eg. 25 frames per second ) Ø continuous media, isochronous (等時 ) media 5 Ming-Yen Lin, IECS, FCU

Def. - Multimedia l a collection of media types used together l at least one media type is not alphanumeric data l Multimedia data: computer-readable representation of multiple media types l Multimedia information: information conveyed by multiple media types l used interchangeably ØMultimedia, media item and object: any autonomous entity in an MIRS that can be queried, retrieved, and presented 6 Ming-Yen Lin, IECS, FCU

Def. - Databases, DBMSs, Text Document Information Retrieval l Databases: a collection or repository of data or media items l DBMSs: entire system l Text Document Information Retrieval ØAutomated information retrieval (IR): manage the huge amount of scientific literature (1940 s) nmain function: store and manage a large number of text documents, relevant to user queries can be retrieved quickly ØIR is also called text-based retrieval 7 Ming-Yen Lin, IECS, FCU

Def. - Multimedia Indexing & Retrieval l Multimedia Indexing ØContent-based retrieval : retrieval based on actual media features such as color and shape, instead of text annotation of the media item nbased on similarity instead of an exact match l MIRS: system providing multimedia information retrieval using a combination of DBMS, IR, and content-based retrieval techniques l multimedia DBMS (MMDBMS). : A fully fledged MIRS Ming-Yen Lin, IECS, FCU 8

Def. - Feature Extraction, Content Representation & Indexing l Feature Extraction/content representation ØWhat are the main features or contents in a MM item Øautomatic or semiautomatic process Øalso called indexing l Index: data structure or the organization of extracted features for efficient search and retrieval. 9 Ming-Yen Lin, IECS, FCU

Need for MIRS l more and more multimedia data is being captured and stored Øefficient and effective indexing and retrieval system is required l multimedia data has special characteristics and requirements Ødifferent from alphanumeric data ØDBMS issues l IR techniques are not adequate to handle multimedia data effectively l Integration! 10 Ming-Yen Lin, IECS, FCU

Proliferation of Multimedia Data l an explosion of multimedia information l Not only is an increasing amount of data being stored, but also the types of data its characteristics are different from alphanumeric data l Multimedia data is very data intensive Ø 10 -minute video: 1. 5 GB of storage without compression l Audio and video have a temporal dimension and they must be played out at a fixed rate to achieve the desired effect l Digital audio, image, and video are represented in a series individual sample values and lack obvious semantic structure for computers to automatically recognize the contents l multimedia applications require simultaneous presentation of multiple media types in a spatially and temporally coordinated way l meaning of multimedia data is sometimes fuzzy and subjective l Multimedia data is information rich 11 Ming-Yen Lin, IECS, FCU

DBMSs and Their Role in Handling Multimedia Data l relational database management systems (RDBMSs): organized in tables or relations l structured query language (SQL) is used to create such tables and to insert and retrieve information Ø create table STUDENT ( stu# integer, name char(20), address char(100) ); Ø insert into STUDENT values (10, "Lew, Tom", "2 Main St. , Churchill, Australia"); Ø select name from STUDENT where stu#=32 12 Ming-Yen Lin, IECS, FCU

BLOB l BLOB: support large variable fields in a RDBMS, a concept called binary large objects (BLOBs) Øa large bit string of variable length Øcreate table STUDENT ( stu# integer, name char(20), address char(100) picture BLOB); l a RDBMS does not know the contents or semantics of a BLOB 13 Ming-Yen Lin, IECS, FCU

OODBMSs and Multimedia Data l object-oriented database management systems (OODBMSs) Ø combine database capabilities (such as store and search) and objectoriented features (encapsulation, inheritance, and object identity) Ø combine object-oriented features with a relational database Ø create type IMAGE ( private size integer, resolution integer, content float[], public . . . ); Ø create table STUDENT ( stu# integer, name char(20), address char(100) picture IMAGE); Ming-Yen Lin, IECS, FCU 14

Capabilities to handle content-based multimedia retrieval l Tools, to automatically, or semiautomatically extract contents and features contained in multimedia data l Multidimensional indexing structures, to handle multimedia feature vectors l Similarity metrics, for multimedia retrieval instead of exact match l Storage subsystems, redesigned to cope with the requirements of large size and high bandwidth and meet real time requirements l The user interface, designed to allow flexible queries in different media types and provide multimedia presentations Ming-Yen Lin, IECS, FCU 15

IR Systems and Their Roles in Multimedia Retrieval l IR techniques are important Øa large number of text documents in many organizations Øtext can be used to annotate other media l limitations Øannotation is commonly a manual process and time consuming Øannotation is incomplete and subjective Øcannot handle queries in forms other than text Øfeatures such as image texture and object shapes are difficult to describe using text Ming-Yen Lin, IECS, FCU 16

Integrated Approach to Multimedia Information Indexing and Retrieval l Parts of multimedia data are structured Øcreation date and author of a multimedia document, … Øcan be handled with DBMS techniques l IR techniques for text annotation l an integrated approach combining DBMSs, IR, and specific techniques for handling multimedia data 17 Ming-Yen Lin, IECS, FCU

An Overview of the MIRS Information items in the database are preprocessed to extract features semantic contents and are indexed based on these features and semantics many issues to be addressed! Ming-Yen Lin, IECS, FCU 18

Expected Capabilities l Metadata-Based Queries: formal attributes of database items (name/attributes) used in queries l Annotation-Based Queries: text description of the contents of database items (keywords, free-text) l Queries Based on Data Patterns or Features: Data patterns - statistic information about multimedia data (loudness, color distribution, and texture description) l Query by Example: multimedia objects such as images, a sketches, and sound pieces l Application-Specific Queries: such as the size of of an object and the aging process of a person Ming-Yen Lin, IECS, FCU 19

Common Applications of MIRS l Medicine l Security Øpolice l Education ØScanned pictures l The press Øinterview l Entertainment ØVideo clips l Trademark registration Ming-Yen Lin, IECS, FCU 20