Multimedia Data Stream Management System By David Kleinman
Multimedia Data Stream Management System By David Kleinman
Outline l l Definition Motivating Examples Nine Requirements Current Systems l l l Comparison Brief Overview of current Stream Systems Preview of My Project
What is it? l l l Stream of multimedia data from a source (video camera) Query stored in a system (This query may itself change Process high volumes of data in real-time
Motivating Examples l Security Surveillance l l l Crowd Security Air Security Burglary Baby Sitting Traffic Reports Science l l Animal behavior Ocean
Reqirement #1 - Process Quickly l l Low latency Messages Processed “In-Stream” l l No Storage to perform operation Active System l Avoid Polling
Requirement #2 – Query using Sigma. QL for Streams (Stream. Sigma. QL) l l l Querying Mechanism Based on SQL Express Continuous Streams of Data l Window Construct l l Time Frames Breakpoints Merge Operator
Requirement # 3 –Handle Imperfections l l Data might be late delayed, missing, or out-of sequence Time out individual calculations or computations Challenges with Dealing with out-of-order data Mechanism for additional time
Requirement #4 – Generate Predictable Outcomes l l l Generate deterministic and repeatable results Time-ordered deterministic processing throughout entire pipeline Important for fault tolerance and recovery
Requirement #5 – Integrate Stored and Streaming Data l l Comparing present with past Capability to efficiently store, access, and modify state information
Requirement #6 – Guarantee Data Safety l l Must use a high-availability solution Secondary System l l Synchronizes with primary frequently Takes over in case of failure
Requirement #7 – Partition and Scale Automatically l l Take advantage of distributed computing Support multi-threading l l l Takes advantage of multi-processor Avoids blocking Load Balance across machines Automatic process Transparent
Requirement #8 – Process and Respond Instantaneously l l l Needs to respond in real – time Highly optimized, minimal overhead execution path All system components have high performance
Requirement #9 - Adaptability l l l Change queries without restarting Accept all different types of multimedia streams Allow for custom configuration Work with different systems API
DBMS l Widely used l l l Use SQL – but not equipped for Streams Passive Do not keep data moving Difficult to handle out of order data Difficulty with predictable out comes Incur latency with seamless integration
Rule Engine l l Example – Prolog Active Handle imperfections Troubles with seamless integration
Stream Processing Engine l l l Handle all the requirements Not specifically designed to handle multimedia constraints Not Specifically designed to handle streams of multimedia
Chart DBMS Rule Engine SPE MSPE Keep data moving No Yes Yes Sigma. QL No No No Yes Handle Imperfections Difficult Possible Yes Predictable outcome Difficult Possible Yes High availability Possible Yes Stored and Streamed data No No Yes Distribution and scalability Possible POssible Possible Yes Real time Possible Yes Adaptability Possible Difficult Possible Yes
Aurora l DSMS developed at MIT and Brown
Aurora Query Network Qo. S. . . Qo. S . . .
Stream Management System l Developed at Stanford
Simple Query Plan Q 1 Q 2 State 3 ⋈ State 4 Scheduler State 1 Stream 1 ⋈ State 2 Stream 3
Niagara. CQ l l Developed at Wisconsin First DSMS Uses a grouping strategy Not as complete as other two
System Architecture
Telegraph. CQ l l l Developed at Berkeley Stem – storage point Eddy – route tuples Good at handling multiple queries Adaptive
Adaptivity (Telegraph) Output Queues STe. Ms for join R EDDY grouped filter (R. A) Rx. Sx. T grouped filter (S. B) S T Input Streams R l l l S T Runtime Adaptivity Multi-query Optimization Framework – implements arbitrary schemes
My Project l l l Design a multimedia streaming database Outline the specifications The Scheduling algorithm The query structure The operators Etc.
- Slides: 26