OntheFly Sharing for Streamed Aggregation Sailesh Krishnamurthy Chung

On-the-Fly Sharing for Streamed Aggregation Sailesh Krishnamurthy, Chung Wu , and Michael J. Franklin Presented by: Joshua Lee and Mingrui Wei Material is partially referenced from [1]

Agenda n n n n Motivation Contributions Shared Time Slice (STS) Shared Data Fragments (SDF) Shared Data Shards (SDS) Experimental Evaluation Conclusion

Motivation n n Naive approach leads to scalability and performance problems Static optimization costs too much and does not fit in Dynamic Environments

Contributions n n n Shared Time Slices Shared Data Fragments Shared Data Shards On-the-fly MQO Performance study

Shared Time Slice (STS) To share Windows, there are TWO approaches n Paned n Paired

STS, continued n n Sharing sliced window Combining multiple sliced windows

STS, continued n n To share, or not to share On-the-fly sliced window composition

Shared Data Fragments (SDF) n n Shared processing for aggregate queries with the same window, but different predicates For instance, a set of queries Q, all like Query 3, where each Qi has a window with Range=5 min and Slide=5 min, but different WHERE clauses

SDF: Motivation n n T is the set of input tuples in one window pi(T) is the set of tuples in T that satisfies pi Ai(T) is the aggregate result over pi(T) Unshared approach evaluates each pi(T) and then calculates each Ai(T) as the answer to Qi

SDF: Fragments Defined n n SDF approach is to define fragments Fi as disjoint subsets over T The example, at left, shows that F 5 consists of those tuples that satisfy p 1 and p 3

SDF: Conceptual View n n n A partial aggregation, G, is applied to each Fi Each Ai(T) is formed by a final aggregation, H, on a set of G(Fi) aggregations For instance, in the previous example: n A 1(T) = A(p 3(T) = H{G(F 1), G(F 3), G(F 5), G(F 7)}

SDF: Implementation n Each tuple is augmented with a signature indicating which predicates it satisfies, identifying the fragment in which the tuple belongs (it also identifies the queries to which the fragment belongs) The Fragment Manager dynamically aggregates all tuples with identical signatures A final aggregation is performed on the Fragment Aggregates using the signatures to route to each query

SDF: Cost Comparison n n The unshared approach has no partial aggregation step and each tuple is subjected to as many aggregations as the number of queries it satisfies The SDF approach requires a partial aggregation operation for each tuple, as well as a final aggregation per fragment for each query of which the fragment is a part

Shared Data Shards (SDS) n n Both STS and SDF partition the input, form partial aggregates over the partitions, and then final aggregates SDS first slices the input, then fragments the slices Partial aggregates are calculated on the fragmented slices Final aggregates are calculated using sets of the partial aggregates

Experimental Evaluation n n A performance study using trading data from NYSE and NASDAQ was performed They looked at STS vs. unshared with (same predicates, different windows) They looked at SDF vs. unshared with (different predicates, same windows) They looked at SDS vs. unshared with (different predicates, different windows)

Conclusion n Paired beats Paned Shared Data Fragments beats Unshared Shared Data Shards beats Unshared Slice

References 1. Krishnamurthy, S. , Wu, C. , and Franklin, M. 2006. On-the-fly sharing for streamed aggregation. In Proceedings of the 2006 ACM SIGMOD international Conference on Management of Data (Chicago, IL, USA, June 27 - 29, 2006). SIGMOD '06. ACM Press, New York, NY, 623 -634.