Streaming Sensor Data Fjord Sensor Proxy Architecture for
- Slides: 8
Streaming Sensor Data � Fjord / Sensor Proxy - Architecture for combining streaming data with static data sources - Streaming (Windowed) Operators � Multiquery Eddy - Share memory and processing between stream queries " Tiny. OS and Telegraph �Enhancements to TOS for Sensor Data Processing
Fjords � Query-plan like data structure for combining streaming (push) and traditional (pull) data sources. �Operators assume non-blocking queue interface between each other. �Queues implement push vs. pull �Pull from A to B : Suspend A, schedule B until it produces data. A cannot go forward until B produces data. �Push from B to A : A polls, scheduler thread invokes B until it produces data. A can process other inputs while waiting for B. �Supports parallelism between operators via queues, state machines, and OS (e. g. NIC buffers, DMA) in operator transparent way.
Fjords (Continued) �Key Insight: Stream-based systems need to operate on traditional (pull-based) sources too! �Example: Combine traffic streams with web-based accident reports to correlate accidents with impact on freeway conditions. �Existing streaming solutions cannot do this!
Fjord Operator Example: Zipper Join (similar to band-join, Dewitt et. al. VLDB '91). Operator agnostic with respect to data-flow direction on input and output.
Sensor Proxy �Energy-sensitive database operator �Buffer sensor tuples and route to multiple user queries to hide query load from sensors �Push aggregation operators into sensors to reduce communications load �Dynamically adjust sample rate based on user demand �Push results into Fjords so that other operators don� t block waiting on slow or dead sensors
Multiquery Eddy �Observation: Queries over streams apply only to now. �Old and new queries always looking at same point in data set �Idea: only allocate on copy of each tuple; route that copy to all user queries �Second idea: combine operators over the same data stream to increase efficiency
Multiquery Eddy (Example and Performance) Tuple Throughput vs Number of Queries
Tiny. OS and Telegraph �Goal: Enhance Tiny. OS and Tiny. OS sensors with infrastructure to participate in Telegraph queries �Establish a consistent catalog and sampling interface across all Tiny. OS sensors �Enable selection predicate / aggregation pushdown into sensors �Partially distribute query plans across sensor networks
