Data Stream Management Systems CS 240 B Notes

Data Stream Management Systems CS 240 B Notes by Carlo Zaniolo 1

Data Streams z Continuous, unbounded, rapid, time-varying streams of data elements z Occur in a variety of modern applications y Network monitoring and traffic engineering y Sensor networks, RFID tags y Telecom call records y Financial applications y Web logs and click-streams y Manufacturing processes z DSMS = Data Stream Management System 2

Many Research Projects … z Amazon/Cougar (Cornell) – sensors z Aurora (Brown/MIT) – sensor monitoring, dataflow z Hancock (AT&T) – Telecom streams z Niagara (OGI/Wisconsin) – Internet DBs & XML z Open. CQ (Georgia) – triggers, view maintenance z Stream (Stanford) – general-purpose DSMS z Tapestry (Xerox) – pubish/subscribe filtering z Telegraph (Berkeley) – adaptive engine for sensors z Tribeca (Bellcore) – network monitoring z Stream Mill (UCLA) - power & extensibility z Gigascope: AT&T Labs – Network Monitoring 3

The (Simplified) Big Picture Clients Register Query Streamed Result DSMS Server Input streams Archive Scratch Stored Relations 4

Databases vs Data Streams Database Systems z Model: persistent data z Table: set|bag of tuples z Updates: All z Query: transient z Query Answer: exact z Query Eval. multi-pass z Operator: blocking OK z Query Plan: fixed Data Stream Systems z Model: transient data z Infinite sequence of tuples z Updates: append only z Query: persistent z Query Answer: Often approx z Query Eval. one-pass z Operators: unblocking only z Query Plan: adaptive 5

Research Challenges z Data Models y Relational Streams first, XML streams important too y Tuple-Time Stamping y Order is important y Windows or other synopses z Query Languages: SQL or XQUERY + extensions y Blocking operators and Expressive Power z Query Plans: y Optimized scheduling for response time or memory z Quality of Services (Qo. S) & Approximation y Load shedding, sampling z Support for Advanced Applications y Data Stream Mining 6

Data Models z Relational Data Streams y. Each data stream consists of relational tuples y The stream can be modelled as an append-only relation y. But repetitions are allowed and order is very important! y Order based on timestamps—or arrival order z Streaming XML Data. y. A stream of structured SAX elements 7

Timestamps z Data streams are (basically) ordered according to their timestamps z The meaning of windows, unions an joins is based on timestamps z External y Injected by data source y Model real-world event represented by tuple y Tuples may be out-of-order, but if near-ordered can reorder with small buffers z Internal y Introduced as special field by the DSMS y Approx. based on the time they arrived z Missing (called latent in Stream Mill) y The system assigns no timestamp to arriving tuples, y But tuples are still processed as ordered sequences y By operators whose semantics expects timestamps… y Thus operators might instantiated timestamps as/when needed 8

Data Stream Query Languages Continuous queries and Blocking Operators 9

Query Operators: Sample Stream Traffic (source. IP, source. Port, dest. IP, dest. Port, %source IP address %port number on source % destination IP address % port number on destination length , %length in bytes time % time stamp ); 10

Blocking Query Operators z No output until the entire input has been seen—i. e. , the last tuple in the input, … often detected after we hit the EOF. z Streams – input never ends: thus blocking operators cannot be used as such z Traditional SQL aggregates are blocking z Many SQL operators have DBMS implementations that are blocking but are not intrinsically blocking y group by, sort join can be implemented in blcoking and nonblocking ways y Other operators are intrinsically blocking y Can we formally characterize which is which? x. We will see that nonblocking operators are the monotonic ones 11

Problematic Operators for Data Streams z Blocking query operators—i. e. , those that must see everything in the input before they can return anything in the output z Non. Blocking query operators are those that can return results now, without seeing the rest of the stream z Selection and projection are nonblocking z Set Difference, and Traditional aggregates are blocking z Continuous aggregates are not. 12

Aggregate Invocation: two Forms z Traditional select G, F 1 from S where P group by G having F 2 op J G: grouping attributes, F 1, F 2: aggregate expressions z With windows (SQL: 2003 OLAP Functions) traffic (source. IP, source. Port, dest. IP , dest. Port, length, Time) select source. IP, Time, avg(lenght) over(order by Time, partition by source. IP 50 rows preceding) Cumulative (running) window: . . . over(order by Time, partition by source. IP unlimited preceding) 13

Aggregate Function Properties 1. distributive: sum, count, min, max 2. algebraic: AVG 3. holistic: count-distinct, median 4. On-line aggregates such as exponentially decaying AVG 5. User-Defined Aggregates (UDAs) y Sliding window invocation 1— 2. Efficient computation for memory and CPU y Sliding window invocation on 3 ? y Continuous window on these ? Yes, also for 5. y UDAs can be similar to any of those 14

Avoiding Blocking Behavior z Windows: aggregates on a limited size window are approximate and nonblocking z DSMS do windows of all kinds: y Sliding windows (same as OLAP functions) y Tumbles: restart every new window (traditional definition) y Panes: the window is broken up into panes z Punctuation [Tucker, Maier, Sheard, Fegaras] y Assertion about future stream contents y Unblocks operators, reduces state z Construct used for avoiding blocking are also useful for avoiding infinite memory 15

Joins z General case problematic on streams: May need to join arbitrarily far-apart stream tuples z Equijoin on timestamps is easy to compute—but not very useful z Majority of work focuses on joins between one stream and a window specified on the other Select A. source. IP, B. source. IP from Traffic 1 as A [window TA], Traffic 2 as B where A. dest. IP = B. dest. IP z The symmetric case also common … Traffic 2 as B [window TB] … z Multi-joins less common but possible. 16

Join of Stream S with a Table T (where T is a DB relation or a Window on a Stream) When a new tuple z with timestamp ts(z) arrives in S, join it with all the tuples in T. - ts(z) is the timestamp of tuples so produced If T is a window on a stream S’ z T must contain all the tuples up to ts(z) included: cumulative window on S’ z But we do not have infinite memory: so we must approximate T with a synopsis. E. g. , 30 minutes preceding 17

Multi-way Sliding Window Joins z Evaluation of n-way sliding window joins queries y n streams with associated sliding windows y continuously evaluate the joins of all n windows z Two natural joins strategies y eager: join is evaluated each time a new tuple arrives in any of the input streams y lazy: join is evaluated with some pre-specified frequency, e. g. , every t time units z Computation incremental, as in differential fixpoint of recursive rules. 18

Query Optimization and Scheduling z Sceduling to minimize response time or minimize memory—no real change in CPU time z Optimization based on sharing, query plans, operators, buffers, … 19

A Query Plan Q 1 Q 2 ⋈ Scheduler Given – query plan and selectivity estimates Schedule – tuples through operator chains ⋈ Stream 1 Stream 2 Stream 3 20

Schedulers and Qo. S Metrics z Round Robin (RR) is perhaps the most basic y operators in a circular queue are given a fixed time slice. y. Starvation is avoided, but little adaptivity z FIFO: takes the first tuple in input and moves it through the chain y. Minimal latency, poor memory z Greedy Alogrithms: y. Buffers with most tuples first y. Tuples that waited longest first y. Operators that release more memory first 21

Memory Optimization on a Chain [Babcock, Babu, Datar, Motwani] Output selectivity = 0. 0 σ3 selectivity = 0. 6 σ2 selectivity = 0. 2 Net Selectivity σ1 best slope σ2 starvation point σ3 σ1 Time Input 22

Main ideas z Operators are thought of as filters which y Operate on a set of tuples y Produce s tuples in return z s selectivity of an operator z If s = 0. 2 we can interpret the value in two ways y Out of every 10 tuples, the operator outputs 2 tuples y If the input requires 1 unit of memory, the output will require 0. 2 units of memory 23

The lower envelope z Imagine there is a line from this point to every operator point (ti, si) to its right z The operator that corresponds to the line with the steepest slope is called the “steepest descent operator point” 24

The Lower Envelope z By starting at the first point (t 0, s 0) and repeatedly calculating the steepest descent operator point we find the lower envelope P’ for a progress chart P z Notice that the slopes of the segments are nonincreasing z The operators in each segment form a chain. z FIFO within chain z Greedy across chains 25

Scheduling z Chain minimizes memory be required in special overload situations y But increases response time (latency) y Typically though we want to optimize for response time z Different scheduling protocols optimize different objectives: latency, inaccuracy, memory use, computation, starvation, … y Computation complexity is independent from scheduler y Different policies give significantly different results only for bursty loads z Research Issues: y Complex query plans (beyond simple paths) y Minimization of response time y Adaptive strategies: how do we switch between the two to adapt to load changes? 26

Optimization by Sharing z In traditional multi-query optimization: ysharing (of expressions, results etc) among queries can lead to improved performance z Examples: Similar issues arise when processing queries on streams: ysharing of query operators and expressions y sharing of sliding windows 27

Multi-query Processing on Streams z Opportunities for optimization when windows are shared---e. g: select sum (A. length) from Traffic 1 A [window 1 hour], Traffic 2 B [window 1 hour] where A. dest. IP = B. dest. IP select count (distinct A. source. IP) from Traffic 1 A [window 1 min], Traffic 2 B [window 1 min] where A. dest. IP = B. dest. IP z Strategies for scheduling the evaluation of shared joins: y Largest window only y Smallest window first y Process at any instant the tuple that is likely to benefit the largest number of joins (maximize throughput) 28

Shared Predicates for R. A > R. A = 6 R. A = 8 R. A ≠ 9 7 1 R. A > 7 R. A > 11 R. A < 3 R. A < 5 [Niagara, Telegraph] 11 A>1 < A>7 3 A<3 = ≠ A>11 Tuple A=8 A<5 6 8 9 29

Qo. S and Load Schedding z. When input stream rate exceeds system capacity a stream manager can shed load (tuples) z Load shedding affects queries and their answers z Introducing load shedding in a data stream manager is a challenging problem z Random and semantic load shedding 30

DSMS Quality of Service (QOS) Approximation and Load Shedding 31

QOS via Synopses and Approximation z Synopsis: bounded-memory history-approximation y Succinct summary of old stream tuples y Like indexes/materialized-views, but base data is unavailable y Examples x. Sliding Windows x. Samples x. Sketching techniques x. Histograms x. Wavelet representation z Approximate Algorithms: e. g. , median, quantiles, … z Fast and light Data Mining algorithms 32

Qo. S and Load Schedding z. When input stream rate exceeds system capacity a stream manager can shed load (tuples) z Load shedding affects queries and their answers: drop the tasks and the tuples that will cause least loss z Introducing load shedding in a data stream manager is a challenging problem z Random load shedding or semantic load shedding 33

XML Data Streams 34

XML Data Streams: Applications • An XML data stream is a sequence of tokens • Data and application integration • Distributed monitoring of computing systems • Message-based web services • Purchase orders, retail transactions • Personalized content delivery 35

XML Streams: Data Model XML data: tree structure Data stream: ~ SAX events <Purchase_Doc> <PR_Number val = “ 50”/> <Supp_Name>ABC</Supp_Na me> <Address> <City>Florham Park</City> <State>New Jersey</State> </Address> <Line_Items> <Item> <Part_Number val= “ 1050”/> <Quantity val=“ 20”/> </Item> [element Purchase_Doc any. Type] [element PR_Number any. Type] [attribute val any. Simple. Type] [chardata 50] [end-attribute] [end-element] [element Supp_Name any. Type] [text ABC] [end-element] … 36

XML Query Languages z XML query languages y Xquery, XSLT, Xpath y Declarative matching of structured data and text y Easy restructuring to meet needs of data consumers 37

XML Streams: research Issue z Efficient Processing of single/multiple queries (e. g. , Xfilters/Yfilters) z Blocking operators/constructs in XQuery— e. g. , XQuery new function definition mechanisms are blocking z Integration of relational and XML DSMS— just like relational and XML DBMS are now being intergrated. 38

Prototype Systems z Aurora (Brandeis, Brown, MIT) [CCC+02] z Gigascope (AT&T) [CJSS 03] z Hancock (AT&T) [CFP+00] z STREAM (Stanford) [MWA+03] z Telegraph (Berkeley) [CCD+03] z … Stream Mill [UCLA] 39

Aurora (Brandeis, Brown, MIT) z Geared towards monitoring applications (streams, triggers, imprecise data, real time requirements) z Specified set of operators, connected in a data flow graph z Optimization of the data flow graph z Three query modes (continuous, ad-hoc, view) z Aurora accepts Qo. S specifications and attempts to optimize Qo. S for the outputs produced z Real time scheduling, introspection and load shedding 40

AT&T: Hancock and Gigascope z Hancock: A C-based domain specific language which facilitates signature extraction from transactional data streams. z Signature: charetizes behavior of customer or services y Support for efficient and tunable representation of signature collections y Support for custom scalable persistent data structures y Elaborate statistics collection from streams z Gigascope: SQL based DSMS for monitoring of network data 41

STREAM [Stanford Uiversity] z General purpose stream data manager z CQL (continuous query language) for declarative query specification z Consider query plan generation z Resource management: y Operator scheduling y Static and dynamic approximations 42

Telegraph [UCB] z Continuous query processing system z Support for stream oriented operators z Support for adaptivity in query processing z Various aspects of optimized multi-query stream processing 43

Commercial Systems z Sybase: publish-subscribe using MQ (Memory Queues) y MQs: are in-memory tables processed using active rules and stored procedures z Similar solutions in Oracle and Teradata. But IBM's MQSeries, Microsoft's MSMQ are web-service oriented: Java Message Service (JMS), Web. Sphere, CORBA. z Two DSMS startups: y CORAL 8: http: //coral 8. com/ y. Streambase: http: //www. streambase. com/ 44

More Tutorial Talks Brian Babcock, Shivnath Babu, Mayur Datar, Rajeev Motwani, Jennifer Widomhttp: //theory. stanford. edu/~rajeev/pods-full-talk. ppt Nick Koudas and Divesh Srivastava. Data stream query processing. Tutorial presented at International Conference on Very Large Databases (VLDB), 1149, 2003. [ PDF | talk slides (PDF) Nick Koudas et al. Matching XML Documents Approximately (with S. Yahia and D. Srivastava) Tutorial delivered at ICDE 2003 Nick Koudas et al. Stream Data Management: Research Directions and Opportunities. Invited Talk at IDEAS 2002. Nick Koudas et al. Mining Data Streams (with S. Guha) Invited Tutorial delivered at PAKDD 2003 45

Implementation Approaches for Continuous Queries on Streaming XML z Automata-based techniques: y XFilter [AF 00]: finite state machine per path expression y XTrie [CFGR 02]: shares common sub-paths of PC paths y YFilter [DF 03]: single NFA for all path expressions y [GMOS 03]: single DFA, limitations on flexibility y XPush [GS 03]: pushdown automaton for tree patterns z Index-based techniques: y Match. Maker [LP 02]: shared tree patterns y Index. Filter [BGKS 03]: shared path expressions, 46 comparison

XML Stream Processing: Key Ideas z Obtain bindings of for clause path expression variables y Ordered sequence, no duplicates z Filter bindings using where clause path expression predicates y Existential check suffices z Compute bindings of return clause path expressions y Ordered (possibly null) sequence z Goal: Efficient matching/binding of XML path expressions y Very large number of path expressions 47