Realtime Stream Processing Architecture for Comcast IP Video

Real-time Stream Processing Architecture for Comcast IP Video Strata Conference + Hadoop World 2013 Chris Lintz Gabriel Commeau

Agenda o Comcast VIPER Overview o Architecture Overview o Q&A

Comcast Video IP Engineering and Research (VIPER) Packaging Transcoding Storage Origination Samsung i. OS Xbox Live Android Storm

Why Do We Focus on Real-time? • Proactively diagnose issues • Form real-time intelligence • Help deliver best possible video experience Viewership Prime Time

Video Player Analytics Protocol • Live and On Demand • JSON event objects • Key metrics • • Bitrate Frame rate Fragments Errors We collect and use all data in accordance with best consumer privacy practices and applicable laws

Player Sessions: Key In Understanding Video Experience

High Level Architecture And Data Flow

Flume: Data collection Tier o Collect, aggregate and move large amounts of data o Distributed, scalable, reliable, customizable o Multi-tier architecture

Storm: Stream Processing Tier

Player Sessions in Real-time o Sessions in Flume? • Technical issues: consistent hash and exactly-once semantics • Design goals • Separation of concerns o Session write-through rate?

Flume Edge Tier: Video Player Analytics End Point o Analytics events over HTTPS o HTTP Source o Re-batch with inner sink and source

Flume Mid Tier: Processing and Routing Data o Video Player Event processing • Geo-location, asset metadata, validation, to-storm o Replication channel processor: • HDFS sink • Storm sink

Bridging Flume to Storm: Flume 2 Storm Connector o Service discovery o Distributed, scalable and reliable o Low latency

Simplified Video Player Storm Topology

Requirements for Read/Writes from Storm Bolts o Functionality beyond key/value stores o Real-time and historic window queries o Speed of in-memory writes and durability of disk

Utilizing Mem. SQL for Persistence • Distributed in-memory SQL database • ACID, highly available, fault tolerant • Aggregators route queries to leaves • Leaves are auto-sharded • Solves our intense read/writes

Isolated Analysts and Ingest Aggregators

Achievements In Utilizing Mem. SQL • Complex queries in milliseconds • Fault-tolerant Storm bolt state • Joins now available outside of Storm bolts • Foreign key shards • Complex data streams • • Dynamic alters without locks or down time JSON type

Wrapping Up o Real-time at Comcast scale • • Millions of video players Horizontal scale everywhere Aggregated metrics across US and complex analysis Real-time API o Builds foundation • Advanced real-time analytics • Better platform for innovation – Alerts on complex objects – Supplemental real-time data back to clients – Popularity-based CDN

Thank You christopher_lintz@cable. comcast. com gabriel_commea@cable. comcast. com