Continuous Query Language From CQL to CAPE Algebra

Continuous Query Language: From CQL to CAPE Algebra Plans Lee Chu Che Wai Kwan MQP 2004/2005

Continuous Query Processing • Emerging Applications: – Traffic management – Network monitoring • Require: – Online processing of data streams • But: – Traditional databases handle persistent data 2

Databases Systems VS Data Stream System • Database System • Data Stream System – One time query – Continuous queries – Random access – Sequential access 3

CAPE: Constraint-exploiting Adaptive Processing Engine • An on-going project at WPI 4

CAPE’s limitation • Desire: – High-level query language, such as SQL • Instead: – Enter queries as low-level execution plan • Problems: – Tedious to enter – Error prone 5

Algebra Plan VS SQL <queryplan> <operator root = “true” id = “ 1” class. Name = “ …”> <class. Variables> <variable name=“group_pos” value=“ 0”/> <variable name=“function” value=“null”/> <variable name=“function_pos” value=“ 0”/> <variable name=“function” value=“count”/> <variable name=“function_pos” value=“ 0”/> <variable name=“propagate” value=“false”/> <variable name=“debug” value=“true”/> </class. Variables> <properties> </properties> <parents> </parents> <children> <child id = “ 2”/> </children> <streams> </operator> <operator root…. . >. . . </operator>. . . </queryplan> ID = 1 Group By ID = 2 select S. A from R, S, Q where R. A = S. A 6

Objective • Define and implement a high-level query language for CAPE 7

Methodology • Study existing Continuous Processing Language proposals • Identify one, adopt and adapt if appropriate • Implement it for CAPE 8

Requirements on Language • SQL-alike • Data Streams • Windows on streams 9

Continuous Processing Languages • UDA – UCLA • Telegraph. CQ – Berkeley • STREAM-CQL – Stanford 10

STREAM-CQL • Well defined semantics • Open source available • Query example: query : rstream (select S. A from R, Q, S[range 1 minute] where R. A = S. A); 11

Our Query Plan Generator: Big Picture CQL STREAM Parser CAPE Engine CAPE XML Plan Writer STREAM Plan Generator CAPE Plan Rewriter 12

Step 1 : STREAM Parser Generates a parse tree CQL STREAM Parser STREAM Plan Generator Yacc and Lex 13

Step 2: STREAM Plan Generator t_rstream. Now t_remove. Istream t_stream. Cross t_remove. Project t_make. Cross. Binary t_make. Stream. Cross. Binary t_push. Select Parse Tree STREAM Plan Generator Modified Plan CAPE Plan Rewriter 14

STREAM Plan Generator : Default Query Plan RStream ID = 7 Project [1, 0] ID = 6 query : rstream (select S. A from R, S [range 1 minute], Q, where R. A = S. A); Select[0, 0]==[1, 0] ID = 5 Cross (1, 3, 4) ID = 0 Stream Source[0] ID = 1 Range Window[60] ID = 3 Stream Source[2] ID = 4 15 Stream Source[1] ID = 2

STREAM Plan Generator: Cleaned Query Plan RStream ID = 7 Project [1, 0] ID = 6 query : rstream (select S. A from R, S [range 1 minute], Q, where R. A = S. A); Cross (10, 4) ID = 9 Select[0, 0]==[1, 1] ID = 10 Stream Source[2] ID = 4 Cross (1, 3) ID = 8 Stream Source[0] ID = 1 Range Window[60] ID = 3 Stream Source[1] ID = 2 16

Step 3: CAPE Plan Rewriter Theta. Join rule Window. Push. Up rule Cleaned Tree CAPE Plan Rewriter Optimized Tree 17

Theta. Join Rule RStream ID = 7 Project [1, 0] ID = 6 Cross (11, (10, 4) ID = 9 Select[0, 0]==[1, 1] Theta. Join[0, 0]==[1, 1] ID = 10 ID = 11 Cross (1, 3) ID = 8 Stream Source[0] ID = 1 Stream Source[2] ID = 4 Range Window[60] ID = 3 Stream Source[1] ID = 2 18

Window. Push. Up Rule RStream ID = 7 Range Project Window[60] [1, 0] IDID == 36 Project 0]0]4) Project Cross[1, (11, ID ==6=6 9 IDID Cross(11, 4)4) ID Range Window[60] ID==99 IDRange = 3 Window[60] Range Window [60] Theta. Join[0, 0]==[1, 1] ID = 11 Stream Source[2] ID = 11 Theta. Join[0, 0]==[1, 1] ID = 4 Range. IDWindow[60] Theta. Join[0, 0]==[1, 1] ID==11 11 Range ID = 11 Window[60] Range Window[60] Stream Source[0] ID = 3 ID = 1 Stream Source[0] Stream IDID == 1 Source[0] Stream Source[0] ID =11 Stream Source[1] ID = 1 ID = 2 19

Step 4: CAPE XML Plan Writer <queryplan> <operator root> <class variables> < /class variables> <properties> </properties> <parents> </parents> <children> </children> <stream> </operator> </queryplan> Optimized Tree CAPE XML Plan Writer CAPE Engine XML Plan 20

Evaluation Methodology • Query test bed: – Test individual operators – Test complex query plans • Evaluation – Manual inspection of generated XML plan – Test XML file on CAPE 21

Evaluation of Individual Operators • • Regular Project Function Project Select Stream Source Range Window Partition Distinct 22

• CQL: Rstream (Select A from S where A =5); 23

• CQL: rstream (select A + B from S); 24

Conclusion • Identified query language for CAPE • Designed a loosely coupled translation frameworks from CQL to CAPE: – Rewrite algebra tree – Generate CAPE XML plans • Evaluation of generated query plans 25

Future Works • Implement Relations – Which will maximize CAPE’s capability • Research on the window size – Support different time range variation • Implement a Graphical User Interface – Drag and Drop feature to input CQL 26

Acknowledgements • Prof. Rundensteiner • Yali Zhu • Luping Ding 27

Question or Comments? 28