Multimedia Databases Storage and Retrieval of Media Data

Multimedia Databases Storage and Retrieval of Media Data Jukka Teuhola Dept. of Information Technology, University of Turku Fall 2012 MMDB-1 J. Teuhola 2012 1

General course info n http: //staff. cs. utu. fi/kurssit/multimedia_databases/autumn_2012/ n Lectures 28 h, Tue 14 -16, Fri 10 -12, in classroom B 2033 n Homework 6 times, every solution gives a bonus of 1 point to the final score in the examination. n Examination: ¨ 5 tasks, max 8 points, so the total score is 0 -40 points ¨ Minimum accepted = 20 points, giving grade 1 ¨ Linear interpolation: 20 -40 1 -5, formula: ¨ Preliminary knowledge: Databases; data structures and algorithms MMDB-1 J. Teuhola 2012 2

Course material n Powerpoint slides: <course homepage>/slides Optional reading: n H. M. Blanken, A. P. de Vries, H. E. Blok, L. Feng (Eds. ): Multimedia Retrieval, Springer 2007. n L. Dunckley: Multimedia Databases – An Object-Relational Approach, Addison-Wesley, 2003. n P. Rigaux, M. Scholl, A. Voisard: Spatial Databases, with Application to GIS, Morgan-Kaufmann, 2002. n D. C. Gibbon, Z. Liu: "Introduction to Video Search Engines", Springer 2008. n Miscellaneous articles. MMDB-1 J. Teuhola 2012 3

Course Contents (tentative) 1. 2. 3. 4. 5. 6. 7. 8. 9. Introduction Management of Large Objects Text and Document Databases Multidimensional Data Structures Spatial Databases Image Databases Video Databases Audio Databases Integration and Standardization MMDB-1 J. Teuhola 2012 4

Main themes of the course Structural and algorithmic database issues: n n Storage principles Data representation Queries, searching, content-based retrieval Indexing NOT: n n n Usage of software products MM authoring and content production MM presentation MMDB-1 J. Teuhola 2012 5

1. Introduction: ‘Multimedia revolution’ n n n n What is multimedia? A dataset or document containing at least two different media types. Multimedia and imaging are continuously growing trends. Enhanced quality and quantity of information, compared to plain text Brings dramatic improvements to human-computer interaction Rich and expressive way of representing, browsing and interacting with information “Second information revolution” Revolutionizes business, science, engineering, manufacturing, art, entertainment… Crucial issues: ¨ Size can be huge ¨ Speed required to satisfy audio/video transmission rates ¨ Semantics: both type- and instance-level metadata MMDB-1 J. Teuhola 2012 6

Levels of sophistication in multimedia databases (1) MM files and archives: n Simple browsing and retrieval n No queries n Supporting software: e. g. web/ media server, browser, player (3) MM archive as part of a wider application n Browsing/search by keyword, plus related actions, e. g. ¨ Web shop: Amazon (2) Annotated & indexed archives: n Search by keyword; see e. g. ¨ Image archive: Gimp-Savvy ¨ Audio archive: Spotify ¨ Spatial db: Google maps ¨ Video archive: You. Tube (4) ’True’ MM databases: n General queries by media content, e. g. ¨ Painting search: Hermitage ¨ Melody search: Musipedia MMDB-1 J. Teuhola 2012 7

Multimedia data types (a) n n Text Integrated to most multimedia applications; complements (as metadata) non-textual forms of data. Nowadays text is usually structured/formatted by markup (e. g. XML) Visual variability through fonts and layout The most space-effective data type to store (b) Audio n Increasingly popular data type n Different formats (WAV, CD, MP 3, AU, AIFF, QT, RA, WMA, Vorbis) n Digitized audio rather space-consuming (tens of Kbytes per second) n Compression is needed (e. g. MP 3 compression ratio 12: 1) n More compact: synthetic music in MIDI format (Musical Instrument Digital Interface); MPEG-4 SA (Structured Audio) MMDB-1 J. Teuhola 2012 8

Multimedia data types (cont. ) (c) Still raster images n Black-and-white / grey-scale / color n One high-resolution image may take several megabytes n Large number of image formats (GIF, TIFF, JPEG, JP 2, PNG, …) n Lossy compression ratio (e. g. for JPEG) normally about 1: 10 (d) Vector graphics n 2 D or 3 D drawings, models, maps n Rather space-effective; consists of larger objects than pixels n Parameters of (meta) objects: scaling, orientation, rotation, etc. n Applications: CAD (computer-aided design), GIS (geographic information systems), animations, computer games (e) Integrated documents (text & images) n Can be generated by today’s text processing programs MMDB-1 J. Teuhola 2012 9

Multimedia data types (cont. ) (f) Digital video n Sequence of frames (= still images) n Highly data-intensive n Integrated audio (interleaved in playback time-sequencing) n Higher compression ratio than with still images (subsequent frames resemble each other). n Compression, transmission and decompression speed must be 20 -30 frames per second. n Animations less space-intensive (synthetic images, standard shapes) n MPEG-4: Object-based representation, many special techniques n Streaming formats: ASF (Micro. Soft), Quick. Time (Apple), RM (Real. Media), FLV (Flash video), Web. M (Google) (g) General integrated multimedia/hypermedia presentations n MS Powerpoint, Adobe Flash, SMIL MMDB-1 J. Teuhola 2012 10

Sample application areas of MMDBs (a) Educational multimedia services: n Distance learning n Teaching material n Educational audio/video document archives n Preview possibility (b) Video-on-demand: n Selection of movie, possibly using queries n Preview possibility; wind/rewind n Requires high bandwidth n Method of payment must be simple MMDB-1 J. Teuhola 2012 11

Sample application areas (cont. ) (c) Audio-on-demand: n Less bandwidth-consuming than video n Recorded programs, music, and live net radio stations (d) Electronic commerce: n Online info about products: pictures, explanations, availability, etc. n Possibility to make queries n Online ordering systems with credit card / net bank payment. n Examples: bookstore, travel agency (e) Intelligent systems (‘expert systems’): n Machine repair: Automatic assistants of different repair jobs. Manuals may be hard to read; demonstrative videos tell it better n Medical care: Standard surgery operations n Crime investigations: combination of surveillance & other info MMDB-1 J. Teuhola 2012 12

Sample application areas of MMDBs (cont. ) (f) Digital libraries n Organized collections of digital information n Both documents and their metadata in digital form n Versatile metadata- and content-based retrieval opportunities n Usually accessible through the web n The web itself & search engines may be considered some kind of (poorly organized) digital library (g) Medical information systems n Patient data, including X-rays, EKG curves, MRI images, . . . n Strict confidentiality n Used for diagnosis, monitoring and research n Automated tools: image/signal processing, pattern recognition, . . . MMDB-1 J. Teuhola 2012 13

General observation n All multimedia applications share some common aspects and functions. The goal of this course is to find the domain-independent set of “core algorithms” which can be used in many applications by varying a few parameters. A generalized multimedia DBMS (MMDBMS) would be useful; probably as an extension to a standard DBMS. MMDB-1 J. Teuhola 2012 14

Technology enabling multimedia n n n n Hardware components: High-speed processors (CPU, GPU), highperformance multimedia workstations, scanners, digital cameras, video cameras, high-resolution monitors, touch-screen monitors, high-precision printers and plotters. High-bandwidth networks (WAN, LAN, mobile), fiber optics, network standards High-capacity storage devices: hard disks, optical disks and jukeboxes, solid-state & non-volatile memories. Image/video processing software: Compression (JPEG, MPEG), analysis, filtering, segmenting, feature extraction. CAD and animation software: 2 D and 3 D graphics, applications in science, engineering, medicine, computer games, etc. Pattern recognition (characters, shapes, etc. ): E. g. neural networks Advanced software systems: OO languages, OO databases, operating systems, multithreading, etc. MMDB-1 J. Teuhola 2012 15

‘Definition’ of MMDB (1) Supports the main types of MM data (2) Can handle a very large number of MM objects (3) Supports high-performance, high-capacity storage management: Hierarchical storage (on-line, near-line, off-line) (4) Offers DB capabilities: Persistence, transactions, concurrency control, recovery from failures, querying with high-level declarative constructs, versioning, integrity constraints, security. (5) Information-retrieval capabilities: Exact-match retrieval, probabilistic (best-match) retrieval, content-based retrieval, ranking of results MMDB-1 J. Teuhola 2012 16

Features related to MM retrieval Functional considerations: n n n Interactive querying Relevance feedback Query refinement Automatic feature extraction and indexing Content- and context-based indexing of different media Single- and multidimensional indexing Efficiency considerations: n n Clustering of media data Storage organization for large media objects Optimization of multimedia queries Replication, parallelism, distribution, scalability MMDB-1 J. Teuhola 2012 17

Architectural considerations Traditional approach: n n Relational or extended relational DBMS, with support for large objects (BLOBs and CLOBs) Information retrieval module (content-based access of objects) Emerging approach: n n n ’No. SQL’ databases (’Not only SQL’) Improved retrieval speed for very large quantities of data. Restricted update types (mainly append), restricted transaction support (relaxed consistency requirements) Ideal: n n n Extensible database system with OO capabilities Support for queries and transactions involving MM objects Support for complex objects with MM subobjects MMDB-1 J. Teuhola 2012 18