RAINDROP XML Stream Processing Engine Murali Mani WPI

RAINDROP: XML Stream Processing Engine Murali Mani, WPI @UPenn, DB seminar June 08, 2006 Partially Supported by NSF grant IIS 0414567 5/20/2004 DSRG Labs

Acknowledgements n n NSF for the financial support Joint work with several others n n Prof. Elke A. Rundensteiner Graduate students – Hong Su, Ming Li, Mingzhu Wei, Shoushen Wang, Jinhui Jian Undergraduate students – Drew Ditto, Bogomil Tselkov … June 08, 2006 DSRG, WPI 2

Applications n Need for efficient stream data processing n n n Monitor patient data in real time Sensor networks – fire detection; battle field deployment; traffic congestion Others – news delivery, monitor network traffic, … June 08, 2006 DSRG, WPI 3

XML Stream Processing <open_auctions> <auction> <privacy>No</privacy> <description> Calendar of <emph>French Impressionism</emph>by<emph>Monet </emph> </description> <initial> $20 </initial> </auction> … Pattern retrieval + Filtering + Restructuring Token-by-Token access manner <open_auctions><auction> <privacy> for $a in open_auctions/auction[privacy] let $e : = $a/description/emph where $e = “French Impressionism” return <Interested. Auction> $a, $e </Interested. Auction> June 08, 2006 DSRG, WPI No timeline 4

Option 1: Automata-Based Pattern Retrieval for $a in open_auctions/auction[privacy] let $e : = $a/description/emph where $e = “French Impressionism” return <Auction>$a, $e </Auction> auctions auction 0 1 privacy 3 2 description 5 4 emph Additional Data Structures for • Buffering • Filtering • Restructuring • … When patterns are retrieved depends on the data June 08, 2006 DSRG, WPI 5

Option 2: “DOM” Based Pattern Retrieval Tagger Select $e = “French Impressionism” for $a in open_auctions/auction[privacy] let $e : = $a/description/emph where $e = “French Impressionism” return <Auction> $a, $e </Auction> Navigate $a, /privacy-> $p Rewrite by “pushing down selection” Navigate $a, /privacy->$p Select $e=“French Impressionism” Navigate $a, /description/emph->$e Rewritten Logic Plan Tagger When patterns are retrieved depends on other patterns Navigate-Scan $a, /privacy -> $p Select $e = “French Impressionism” Navigate-Index Choose low-level implementation alternatives $a, /description/emph -> $e June 08, 2006 DSRG, WPI Physical Plan 6

Which paradigm is better? Minimal pushdown plans win over maximal pushdown when selectivity < 50% June 08, 2006 DSRG, WPI 7

Problem n n n How to provide the framework to choose between these paradigms? Model both paradigms uniformly as algebraic operators. Use a cost model to choose optimal plan given data statistics. June 08, 2006 DSRG, WPI 8

Automaton as Token. Nav for $a in open_auctions/auction[privacy] let $e : = $a/description/emph where $e = “French Impressionism” return <Auction> $a, $e </Auction> Extract $a auctions auction 0 1 privacy 3 2 description 5 4 emph Structural. Join $a Select non-empty($b) $e=“French …” Extract $b $e Token. Nav $a, /privacy->$b $a, /desc/emph->$e Token. Nav $s, /auctions/auction->$a June 08, 2006 DSRG, WPI 9

DOM Navigation as Node. Nav for $a in open_auctions/auction[privacy] let $e : = $a/description/emph where $e = “French Impressionism” return <Auction> $a, $e </Auction> Select $e=“French …” Select non-empty($b) Node. Nav $a, /privacy->$b auctions auction 0 1 2 Node. Nav $a, /desc/emph->$e Extract $a Token. Nav $s, /auctions/auction->$a June 08, 2006 DSRG, WPI 10

Exploring the Search Space n n A pattern can be retrieved inside the automaton or outside the automaton However there are dependencies for $a in …/a, $b in $a/…, $c in $b/… Node. Nav for $b => Node. Nav for $c Token. Nav for $b => Token. Nav/Node. Nav for $c June 08, 2006 DSRG, WPI 11

Run-time Optimization n Statistics unknown before data arrives Statistics could change over time We need techniques for efficient statistics monitoring, search space exploration and plan migration (safe points for migration) June 08, 2006 DSRG, WPI 12

Run-time Optimization Stream statistics Query plan executor Query Optimizer New Query plan Initial Query plan New Query plan Plan Migrator June 08, 2006 DSRG, WPI 13

Executing a Raindrop Plan June 08, 2006 DSRG, WPI 14

Key Ideas n Minimum Memory requirements n n Discard data early Output data early June 08, 2006 DSRG, WPI 15

In-Time Structural Join for $a in open_auctions/auction[privacy] let $e : = $a/description/emph where $e = “French Impressionism” return <Auction> $a, $e </Auction> Extract $a auctions auction 0 1 privacy 3 2 description 5 4 emph Structural. Join $a Select non-empty($b) $e=“French …” Extract $b $e Token. Nav $a, /privacy->$b $a, /desc/emph->$e Token. Nav $s, /auctions/auction->$a June 08, 2006 DSRG, WPI 16

Better than In-Time Structural Join for $r in /root return <root> <a>$r/a</a> <b>$r/b</b> </root> Structural. Join $r Extract $b $a a root 0 2 1 b 3 Token. Nav $r, /a->$a Token. Nav $r, /b->$b “a” tokens need not be stored Token. Nav $s, /root->$r June 08, 2006 DSRG, WPI 17

Evaluating Predicates for $r in /root where $r/a = “value” return <root> <b>$r/b</b> </root> Structural. Join $r Extract Select $b $a=“value” a root 0 2 Extract 1 b 3 $a Token. Nav $r, /b->$b $r, /a->$a Once $a=“value” is satisfied, “b” tokens need not be stored June 08, 2006 Token. Nav $s, /root->$r DSRG, WPI 18

Using schema knowledge for $r in /root return <root> <a>$r/a</a> <b>$r/b</b> </root> root -> (a*, b*) Structural. Join $a Extract $b $a a root 0 2 1 b 3 Token. Nav $r, /a->$a Token. Nav $r, /b->$b “a”, “b” tokens need not be stored Token. Nav $s, /root->$r June 08, 2006 DSRG, WPI 19

Using Schema Knowledge for Predicates for $r in /root where $r/a = “value” return <root> <b>$r/b</b> </root> root -> (b*, a*, c) Structural. Join $r Extract Select $b $a=“value” a root 0 2 Extract 1 b 3 $a Token. Nav $r, /b->$b $r, /a->$a Once “c” is seen and $a=“value” is not yet satisfied, “b” tokens can be discarded June 08, 2006 Token. Nav $s, /root->$r DSRG, WPI 20

Conclusions n n n Raindrop integrates automaton and “DOM” navigation into one algebraic framework. Cost-based optimization possible. Execution minimizes memory requirements. June 08, 2006 DSRG, WPI 21

Ongoing Work n n Load shedding in XML stream processing. Utilizing Dynamic schema changes for optimization. June 08, 2006 DSRG, WPI 22

Fragment of XQuery supported n n n FLWR expressions (no conditionals/user defined functions) Path expressions use only forward axes (child, descendant or self, attribute) Predicates supported are of the form: path. Expr rel. Op constant June 08, 2006 DSRG, WPI 23

Issues with correlated queries for $r in /root return <root> for $a in $r/a return <a>$r/b</a> </root> June 08, 2006 DSRG, WPI 24

Visit our XQuery engine over XML stream project (RAINDROP) website http: //davis. wpi. edu/dsrg/raindrop/ June 08, 2006 DSRG, WPI 25