The Evolution of Big Data Platform Netflix Eva

The Evolution of Big Data Platform @ Netflix Eva Tse July 22, 2015

Our biggest challenge is scale

Netflix Key Business Metrics 65+ million members 50 countries 1000+ devices supported 10 billion hours / quarter

Global Expansion 200 countries by end of 2016

Big Data Size Total ~20 PB DW on S 3 Read ~10% DW daily Write ~10% of read data daily ~ 500 billion events daily ~ 350 active users

Our traditional BI stack is our competition

How do we meet the functionality bar and yet make it scale? How do we make big data bite-size again?

Our North Star

Event Data Cloud apps Suro/Kaf ka Ursula 15 min AWS S 3 Dimension Data Cassandra SS Tables Daily Aegisthus

Storage AWS S 3 Compute Service Tools

Evolving Big Data Processing Needs Analytics ETL Interactive data exploration Interactive slice & dice RT analytics & iterative/ML algo

Service Tools API Portal Big Data API Big Data Portal Evolving Services/Tools Ecosystem

AWS S 3 as our DW Storage

Evolution of Big Data Processing Systems

Parquet

Service Tools API Portal Big Data API Big Data Portal Evolution of Services/Tools Ecosystem

Metacat

Service d Tools API Portal Big Data API Big Data Portal d

Big Data API

Big Data Portal

Open source is an integral part of our strategy to achieve scale

Big Data Processing Systems Services/Tools Ecosystem

Why use Open Source?

Why contribute back?

Why contribute our own tool?

Is open source right for you?

Measuring big data - understanding data by usage By Charles Smith, Netflix Tomorrow @ 1: 40 -2: 20 pm

Eva Tse etse@netflix. com jobs. netflix. com