page 9 Conventional Database Multimedia Database simple textual
page 9 멀티미디어 데이타베이스 특성 Conventional Database Multimedia Database Ê simple textual retrieval Ê content-based retrieval ç exact match Ë special interface not needed for querying ç similarity-based match Ë special interface needed for visual querying Ì frequent update and overwrite Ì mostly archival or versioning
page 12 객체지향 데이타 모델 q Basic Concepts m objects and OIDs m methods and encapsulation m classes and inheritance m composition and user-defined types m polymorphism q Extension m lists, sets, multisets m relationships m versions
page 14 멀티미디어 질의어 q 확장된 관계형 데이타베이스 질의어 m 비정형 데이타는 BLOB(Binary Large Object)에 저장 m BLOB에 대한 검색은 불가 m 표준 SQL 사용 q 예제 m Find the name and photo image of an employee whose empno is 125 $char name[20]; $struct blob photo; $SELECT Name, Photo INTO $name, $photo FROM Employee WHERE Emp. No = 125;
page 15 멀티미디어 질의어 q 객체지향 데이타베이스 질의어 m Abstract Data Type 지원 m Generalization과 Aggregation 지원 m Extended OOSQL 사용 q 예제 m Display the photo image of the vehicle owned by the employee named “Kim” SELECT FROM WHERE e. owned_vehicle. image() Employee e e. name = “Kim”
page 16 MHEG (Multimedia and Hypermedia information coding Expert Group) q MHEG 이란? m 표준 멀티미디어 객체 모델 m ISO/IEC Standard for multimedia and hypermedia composition and presentation q MHEG Object Hierarchy MH-OBJECT ACTION LINK SCRIPT MODEL CONTAINER COMPONENT CONTENT COMPOSITE DESCRIPTOR
page 17 MHEG 저작도구 q KMEDIA (KAIST 개발)
page 18 MHEG 객체 데이타베이스 시스템 MHEG-5 Authoring Tool (KMEDIA) Netscape MHEG-5 Runtime Engine (Netscape Plug-In Module) OODBMS Client 통신망 WWW Server MHEG-5 Object Server OODBMS Server (오디세우스) OODBMS Client MHEG-5 Object Database
page 19 OCPN(Object Composition Petri Net) q OCPN 이란? m Petri Net을 확장한 멀티미디어 객체 모델 m augmented with time duration and resource utilization q Spatio-temporal binary relations for Petri Net model m a Before b a m a Meets b a m a overlaps b a b b b a m a during b m a starts b m a equals b m a finishes b b a b a b
page 20 OCPN 모델링 예 parameter duration size deadline rate content text video image video audio t 1 t 2 value t 3 time
page 21 멀티미디어 문서 (Hy. Time, HTML) q 멀티미디어 문서 기술 언어로 표현된 멀티미디어 객체 q 구성 m Media (Content) n 텍스트, 그래픽스, 이미지 n 오디오, 비디오 m Structure (Integration) n 논리적 구조 (Logical Structure) n 하이퍼링크 구조 (Hyper. Linked Structure) n 시공간적 구조 (Spatial and Temporal Structure) m Style (Presentation) n 모양 (Appearance) n 행위 (Behavior/Script)
page 23 멀티미디어 문서 데이타베이스 q 스키마 m 문서의 내용을 표현하는 클래스 (1) m 문서의 구조를 표현하는 클래스 (2) m 문서의 스타일을 표현하는 스키마 (3) Element (3) Style. Element (1) Content Center Font . . . Text Graph Image Audio Video (2) Logical. Element Abstract Body Chapter Link. Element Reference . . . Paragraph Section Subsection . . . Italic Bold
page 24 멀티미디어 문서 예 <article ID=cs 96 -15> <front> <title> An Object-Oriented Hypermedia System</title> <author> B. K Park, W. K Loh, and K. Y. Whang</author> <date>Dec. 7, 1996</date></front> <abstract> In this paper, . . . </abstract> <body> <chapter title=“Introduction”> As the amount of information is growing fast. . . </chapter> <chapter title=“MHEG Standard” ID=mheg>The <italic>MHEG</italic> standard provides the. . . </chapter> <chapter title=“Design and Implementation”>In this section, we describe the. . . </chapter> <section title=“Overview”> <bold>Kmedia-Unix</bold> has been implemented using C++ language for the following reasons. . . <image caption=“The system architecture” file=sysarch. gif>. . . </section> <section title=“Runtime Layer”> <subsection title=“User Interface”>The user interface consists of three kinds of windows. . . </subsection> <subsection title=“Sessions”>A session manages all the document windows. . . </subsection> <subsection title=“Multimedia Mail System”>Multimedia mail system. . . using the <link end=mime. rfc>MIME</link> protocol. As the <link end=#mheg>MHEG</link> standard. . . </subsection></section> <section title=“Implementation>></bold>Kmedia-Unix</bold> has been implemented. . . </section></chapter> <chapter title=“Conclusion”>Hypermedia systems have been actively studied. . . </chapter></body> <reference>. . . </reference></article>
page 25 멀티미디어 문서 데이타베이스 스키마 그래프 Element Content Text Image name start end Video contains() Document id title authors date abstract chapters reference Logical. Element root() ancestor() descendent() leaf() flink() blink() class/subclass link attribute/domain link Style. Element Link. Element anchor target Font Chapter title paragraphs sections Section title paragraphs subsections Subsection title paragraphs subsections Paragraph content style link Italic Bold
page 26 멀티미디어 문서 데이타베이스 예 q 논리적 구조 표현 text-1 link-1 paragraph-1 italic-1 image-1 paragraph-2 article-1 id title authors date abstract chapters reference link-2 bold-1 paragraph-4 paragraph-6 paragraph-5 paragraph-3 chapter-1 bold-2 chapter-4 chapter-3 section-1 section-3 section-2 subsection-1 subsection-2 subsection-3 paragraph-8 paragraph-7
page 27 멀티미디어 문서 질의 예 q Q 1: Select the title and the first author of articles having a section with a title containing the words “SGML” and “OODBMS” [Christophides 94] SELECT tuple (t: a. title, f: first(a. authors)) FROM a in Articles, s in a. sections WHERE s. title contains (“SGML” and “OODBMS”) q Q 2: Select the titles of the articles that are referenced by the second section of the article with id ‘cs 96 -15’ [Lee 96] SELECT k. title FROM (h, k) in Articles WHERE h. id = ‘cs 96 -15’ AND k in h. sections[2]. flink[1]
page 28 Limitations of Relational Database q No type support for multimedia q BLOBs cannot be interpreted q Events/contents m complex m dynamic m grouping based on space, time, and shared semantics m arbitrary attributes and relations q Multimedia document: Structure recursive in space, time, and logic q Dynamic schema to define new types Object-oriented paradigm is well-suited to meet these requirements
page 30 대용량 멀티미디어 객체 관리 q 대용량 객체 관리를 지원하는 저장 시스템 m Wi. SS (Wisconsin Storage System), Univ. of Wisconsin m EXODUS, Wisconsin Univ. m Starburst, IBM m EOS (Experimental Object Store), Boston Univ. m COSMOS, KAIST
page 31 대용량 멀티미디어 객체 관리 시스템 q Wi. SS m Data segments are fixed-sized(size=1 page) m Data segments are pointed to by a directory (size=1 page) m Maximum object size equals to 1. 6 MB(=4 KB * 400) segments 5 700 600 800 900 600 directory segments
page 32 대용량 멀티미디어 객체 관리 시스템 q EXODUS m Data segments are fixed-sized(parameter) m Data segments are indexed by a B-tree-like structure 900 350 600 1830 900 Full page (100 bytes) 400 Partially full page 650 Empty page 930
page 33 대용량 멀티미디어 객체 관리 시스템 q Starburst m Data segments are organized into a binary buddy system m Data segments are pointed to by a long field descriptor m Reads and Appends are efficiently supported (video and audio) m Insertions and Deletion in the middle are gracefully supported (length-changing update) segments 5 first 100 1 page=100 bytes last 330 Long field descriptor
page 34 대용량 멀티미디어 객체 관리 시스템 q EOS m The two ideas proposed in EXODUS and Starbust are combined m Data segments are variable-sized m Data segments are indexed by a B-tree-like structure 1230 950 1230 1830 470 600
page 37 연속 데이타 스트림을 지원하기 위한 저장 시스 템 기능 q 실시간 디스크 스케쥴링 m SCAN-EDF (SCAN 방식과 EDF(Earliest Deadline First) 방식의 결합) q 버퍼 관리 m preloading and replacement 예: L/MRP (Least/Most Relevant for Presentation) q Data Placement m continuous placement, random placement, constrained placement m disk array (data striping, data interleaving) q 저장장치 관리 m 저장장치 계층구조 상의 적절한 위치에 데이타 배치 m 하나의 저장장치 크기를 초과하는 대용량 데이타의 관리 m 저장 장치 들의 이질성 처리 (device transparency) q 승인 제어 (Admission Control)
page 38 L/MRP 버퍼 관리 방법 m Single interactive continuous data stream 가정 m Relevance value of each unit n access probability m Replacement strategy n Least relevant units are replaced m Preloading strategy n Most relevant units are preloaded relevance Curr. presentation point Presentation mode: Forward skip 2 1. 0 0. 5 0 505 600 605 700 705 unit
page 39 내용기반 멀티미디어 정보검색 개요 user query multimedia objects (images, video, audio, …) feature extraction n-dimensional feature vectors indexing Database (MLGF, R*-tree) Storage sample objects, n-dim feature vectors filtering candidates super-set refinement solution set (feature vectors) multimedia objects Retrieval incremental search
page 40 멀티미디어 인덱싱 q Feature Extraction (automatic or manual) m Text Features n keywords, . . . m Image Features n color, texture, shape, sketch, volume, spatial constraints, . . . m Video Features n representative frame, panning, zooming, overlap, . . . m Audio Features n word spotting, speaker identification, silence and non-speach sounds, … q Multi-Dimensional Index Structure m Multi-Dimensional Point Access Methods n 예: MLGF (Multi-Level Grid File) m Multi-Dimensional Spatial Access Methods n 예: R-tree
page 41 멀티미디어 정보검색 q 검색 방법 m Feature-Based Retrieval based on Similarity q 검색 유형 m k-Nearest Neighbor Search m Range Search m All Pair Search m Sub-Pattern Match f 2 f 1
page 44 정보검색을 위한 인덱스 q 역색인 파일 (Inverted file) m IR 시스템에서 사용하는 인덱스 파일 m 주어진 키워드가 발생한 텍스트의 doc. ID와 텍스트내 발생 위치를 알려줌 q 구성 m 역색인 리스트(Inverted List)의 집합으로 구성 m 역색인 리스트는 키워드와 포스팅 리스트로 구성 Postings list Keyword doc. ID Posting 1 frequency Posting 2 Posting 3 . . . Posting n Inverted List Info. Position 1 Position 2 . . . Position m Posting info. Sentence no. word no. Position info.
page 45 정보검색을 위한 인덱스 q 시그너춰 파일 (Signature file) m word signature n a bit pattern of size F, with m bits set to 1 l m bit positions to be set to 1 are decided by hash functions m block signature n a bit pattern formed by OR’ing word signatures m document signature n a bit pattern formed by concatenating word signatures 예) Word multimedia database block signature Signature 001 000 110 000 010 101 001 010 111 011
page 46 정보검색을 위한 인덱스 q 시그너춰 파일 저장 (Bit-Sliced Signature Files) m F 개의 bit-file로 나누어 저장 N blocks F files pointer file 0 1 1 0 0 1 1 1 0 0 0 1 0 0 N 0 1 0 0 0 1 1 1 0 0 1 0 1 1 0 0 1 1 q 시그너춰 파일 검색 m 질의로 주어진 키워드의 word signature 계산 m m 개의 bit-vector를 읽어서 AND’ing . . .
page 47 텍스트 정보검색 q Weight (키워드의 중요도) m Term Frequency (tf) n number of occurrences of particular keyword in the document collection m Document Frequency (df) n number of occurrences in the collection where particular keyword occurs n Inverse Document Frequency (idf) : document discriminator m weight n weight = tf * idf q Extended Boolean Retrieval Model m Fuzzy set extension n w 1 AND w 2 = max(w 1, w 2) n w 1 OR w 2 = min(w 1, w 2) n w 1 NOT w 2 = if w 1 > w 2 then w 1 - w 2 else 0 m ranking
page 50 텍스트 질의 사례 q 멀티미디어 문서에 대한 질의어 (Hy. OQL) m 프랑스 Versailles Saint-Quentin 대학에서 제안 q 예제 m 문서 구조에 관한 질의 예 n Find the 3 rd section in the paper identified by “P 1” SELECT p. sections[2] FROM p in Papers WHERE p. id = “P 1” m 문서 내용에 관한 질의 예 n Find all the papers containing the string “hypertext” SELECT p FROM p in Papers WHERE p. includes(“hypertext”)
page 51 텍스트 질의 사례 q 멀티미디어 문서에 대한 질의어 (Hy. OQL) 예제 m 하이퍼링크에 관한 질의 예 n Find all the outgoing nodes of the paper “P 1” SELECT p. forward_nodes() FROM p in Papers WHERE p. id = “P 1” n Find all the papers containing the string “hypertext” and connected by a link typed “yes” SELECT p FROM c in Comments, p in c. forward_nodes(“yes”) WHERE p. includes(“hypertext”)
page 56 공간 질의 예 q 서로 교차하는 강과 도로를 보여라. SELECT river, road FROM river, road WHERE river OVERLAP road;
page 59 R-tree의 구조 y B’ D E A’ A C’ F H C G B x A’ B’ C’ A B C D E F to leaf nodes G H
page 62 Z-order q 특징 m 다른 차원 사이의 bits를 Interleaving하여 영역 표시 m 재귀적으로 공간 분할 m proximity 정보를 유지하면서 다차원 공간을 일차원 공간으로 매핑 0101 11 5 0100 10 01 00 4 0001 1 0000 0111 7 0110 6 0011 3 0010 1101 13 1100 12 1001 9 1000 1111 15 1110 14 1011 11 1010 0 2 8 10 00 01 10 11
page 63 Peano-Hilbert order q 특징 m Hilvert curve를 이용하여 재귀적으로 공간 분할 m 공간 순서화 방법 중 최상의 클러스터링 유지 0101 11 5 0100 10 01 00 4 0011 3 0000 0110 6 0111 7 0010 2 0001 1001 9 1000 8 1101 13 1110 10 1011 11 1100 12 1111 0 1 14 15 00 01 10 11
page 64 이미지 내용 기반 검색 q Image Representation m Local Features n n n Pexels Edges Shape Texture Colors m Global Features n n Color Histograms FFT Hough Transform Eigenvalues Different features are usefule for different types of queries.
page 65 이미지 내용 기반 검색 q Query Processing Image Object Feature Extr. Preprocessing and Database Generation Image Database Features Feature Extr. Query Processing no Vote Success ? yes Image Input Image Object
page 66 이미지 질의 사례 q QBIC (Query By Image Content) m IBM Almaden 연구소에서 개발 m 이미지 feature에 의거한 내용기반 검색 m 검색에 사용되는 feature 들 n n color: averages and histograms texture patterns: coarseness, contrast, directionality shape: area, circularity, orientation, moments sketches (image abstractions) m 질의 n 예를 통하여 feature의 속성을 지정한다 m 사용자 인터페이스 n 객체 윤곽 표시 (outlining) n 대화식 객체 식별
page 67 비디오 내용 기반 검색 q Content of Video data m Semantic Content m Audio Visual Content n n n n Audio Color: color intensity, color histogram Texture: texture pattern Object motion: rotation, translation, … Object relationship: before, after, above, below, … Camera operation: pan, fade in/out, zoom, … Object: shape, area, … m Textual Content n caption of news video clip n title n actors and actresses list
page 68 Video Indexing q Annotation-Based Indexing m Manual annotation m A set of keywords q Feature-Based Indexing m Indexing based on n Objects and Faces (fine-grained / frame-level indexing) n Segments and Scene Changes (medium-grained indexing) n Complex Spatio-Temporal Events (coarse- grained indexing) q Domain-Specific Indexing m Logical Video Structure Model n Anchorperson shot model n CNN Headline News Unit Model
page 69 Example: Facial Database q Face Recognition in Image/Video Database q Three Steps m Face Detection n Locating a face index an image m Feature Extraction n Detection of various parts of a face m Face Recognition n Identifying the person
page 70 비디오 질의 예 q 비디오 질의 예 m query involving spatial events n “Show that segment of the sonogram where the main artery is 90% blocked” m query involving spatio-temporal events n “Get the document with video clip where assembly of Daewoo Nubira’s gearbox is described” m query involving searching of audio data associated with video data n “Get all the documents containing video clips where President Kim, during one of his Cabinet meetings, has mentioned trades with US”
page 71 멀티미디어 질의어 q 멀티미디어 질의의 특성 m Modes of interaction n direct manipulation of graphical objects on the screen n the use of natural language n the use of formal language (SQL) m 질의를 정확하게 표현하기 어려움 n 유사한 내용을 기반으로 점진적 검색 필요 (Query Refinement) n 시각적 질의 도구 필요 (Browsing or Hypermedia Navigation) n 불완전하고 불명확한 질의 처리 지원 (Fuzzy Retrieval) m 통합 질의 처리 n 데이타베이스 질의 + 내용기반 검색 질의
page 72 멀티미디어 질의어 q 멀티미디어 질의 수행 단계 m Object Structure Browsing Using Structure Browser m Query Formulation Using Query Editor m Query Execution m Query Result Browsing Using Authoring Tool m Navigation or Sub-Querying
page 79 멀티미디어 데이타베이스 개발 사례 q MM-VODAK (AMOS) Architecture Application MM Playout Manager SM Pres. Script Interpreter Continuous Obj. Manager MM Client. . . . MM Client MM Playout Manager SM Pres. Script Interpreter Continuous Obj. Manager multimedia capable LAN/MAN traditional LAN/MAN DBMS Interface API MM DBMS Server Query Processor Script Generator Retrieval Engine Transaction Manager Object Manager Ext. Media Server database Continous Obj. Mgr. database
page 91 결론 q 멀티미디어 정보 시스템 m an intergrated system of hypermedia, information retrieval, and database system m providing an information retrieval environment of navigation via hyperlinks and search via queries m integrating hypermedia and information retrieval system with database for efficient storage and retrieval m research platform for efficient storage and retrieval of structured multimedia documents such as HTML and Hy. Time used in WWW and digital libraries q 멀티미디어 데이타베이스 m an essential component of multimedia information system
- Slides: 92