Intelligent Services Serving Machine Learning Joseph E Gonzalez

Intelligent Services Serving Machine Learning Joseph E. Gonzalez jegonzal@cs. berkeley. edu; Assistant Professor @ UC Berkeley joseph@dato. com; Co-Founder @ Dato Inc.

Contemporary Learning Systems Big Data Training Big Models

Contemporary Learning Systems Create BIDMach MLlib VW LIBSVM MLC Oryx 2

What happens after we train a model Data Training Conference Papers Dashboards and Reports Model Drive Actions

What happens after we train a model Data Training Conference Papers Dashboards and Reports Model Drive Actions

Suggesting Items at Checkout Low-Latency Fraud Detection Cognitive Assistance Personalized Internet of Things Rapidly Changing

Data Train Model

Data e Ad rv Se ap t Actions Train Model

Machine Learning Intelligent Services 9

Top-K Query New Page Images … User Feedback: Preferred Item Web Serving Tier Request: Items like x Top α Items Feedback Intelligent Service The Life of a Query in an Intelligent Service Lookup Model μ ∫ σFeature Lookup math ∑ β ρ Feature Model Info User Data Lookup Content Request Product Info

Essential Attributes of Intelligent Services Responsi Intelligent applications ve are interactive Adaptiv ML models out-of-date the e is done moment learning Manageabl Many models e people created by multiple

Responsive: Now and Always Compute predictions in < 20 ms for complex Queries Top K Models Features SELECT * FROM users JOIN items, click_logs, pages WHERE … under heavy query load with system failures.

Experiment: End-to-end Latency in Spark MLlib To JSON 4 HTTP Req. Feature Trans. Evaluate Model HTTP Response Encode Prediction

0 50 0 10 ex Al et N . . . rre Fo LR 21. 8 22. 4 ++ m do ss e 137. 7 C an R la -C 10 Tr e R "L "4 4. 3 n io is ec D Is vg t. A ic ed Pr Latency in Milliseconds 450 400 350 300 250 200 150 100 50 0 418. 7 172. 6 50. 5

Adaptive to Change at All Scales Population Granularity of Data Session Shopping for Mom Shopping for Me Months Rate of Change Minutes

Adaptive to Change at All Scales Population Granularity of Data Session Law of Large Numbers Shopping Change Slow for Mom Months Rely on efficient. Shopping offline for Me retraining Change High-throughput Rate of Minutes Systems

Adaptive to Change at All Scales Population Granularity of Data Session Small Data Rapidly Changing Low Latency Online Learning Sensitive to feedback bias Months Rate of Change Shopping for Mom Shopping for Me Minutes

The Feedback Loop I once looked at cameras on Amazon … Similar cameras and Opportunity for accessories Bandit Algorithms Bandits present new challenges: • computation overhead • complicates caching + indexing My Amazon Homepag e

Management: Collaborative Development Teams of data-scientists working on similar tasks Ø“competing” features and models Complex model dependencies: Cat Photo is. Cat Classifier is. Animal Classifier Cute! Cuteness Predictor

UC Berkeley AMPLab Daniel Crankshaw, Xin Wang, Joseph Gonzalez Peter Bailis, Haoyuan, Zhao Zhang, Michael J. Franklin, Ali Ghodsi, and Michael I. Jordan Predictive Services

UC Berkeley AMPLab Daniel Crankshaw, Xin Wang, Joseph Gonzalez Peter Bailis, Haoyuan, Zhao Zhang, Michael J. Franklin, Ali Ghodsi, and Michael I. Jordan Active Research Project Predictive Services

Velox Model Serving System [CIDR’ 15, Learning. Sys’ 15] Focuses on the multi-task learning (MTL) domain Spam Classification Content Rec. Scoring Session 1: Session 2: Localized Anomaly Detection

Velox Model Serving System Personalized Models (Mulit-task Learning) [CIDR’ 15, Learning. Sys’ 15] Input Output “Separate” model for each user/context.

Velox Model Serving System Personalized Models (Mulit-task Learning) [CIDR’ 15, Learning. Sys’ 15] Feature Model Personalization Model Split

Hybrid Offline + Online Learning Update feature functions offline using batch solvers Feature Personalization Model systems. Model • Leverage high-throughput (Apache Spark) • Exploit slow change in population statistics Update the user weights online: Split • Simple to train + more robust model • Address rapidly changing user statistics

Hybrid Online + Offline Learning Results Similar Test Error Hybrid Offline Full User Pref. Change Substantially Faster Training Offline Full Hybrid

Evaluating the Model Cache Feature Evaluation Input Split

Evaluating the Model Feature Caching Across Input. Users Cache Feature Evaluation Approximate Feature Hashing Anytime Feature Evaluation Split

Feature Caching New input: Compute feature: Hash input: Feature Hash Table

LSH Cache Coarsening New input Hash new input: Use Wrong Value! LSH hash fn. Feature Hash Table

LSH Cache Coarsening Locality-Sensitive Hashing: Hash new input: Locality-Sensitive Caching: Use Value Anyways Req. LSH Feature Hash Table

Anytime Predictions Compute features asynchronously: __ __ __ if a particular element does not arrive use estimator instead Always able to render a prediction by the latency deadline

Coarsening + Anytime Predictions More Features Approx. Expectation Best Checkout our poster! No Coarsening Better Overly Coarsened Coarser Hash

Part of Berkeley Data Analytics Stack Management + Serving Training Velox MLbase Spark Blink. DB Graph Streamin Spark ML X g SQL library Spark Predictio Model n Manager Service Mesos Tachyon HDFS, S 3, …

UC Berkeley AMPLab Daniel Crankshaw, Xin Wang, Peter Bailis, Haoyuan, Zhao Zhang, Michael J. Franklin, Ali Ghodsi, and Michael I. Jordan Predictive Services

Dato Predictive Services Production ready model serving and management system ØElastic scaling and load balancing of docker. io containers ØAWS Cloudwatch Metrics and Reporting ØServes Dato Create models, scikit-learn, and custom python ØDistributed shared caching: scale-out to address latency ØREST management API: Demo?

UC Berkeley AMPLab Daniel Crankshaw, Xin Wang, Joseph Gonzalez Peter Bailis, Haoyuan, Zhao Zhang, Michael J. Franklin, Ali Ghodsi, and Michael I. Jordan Responsiv e Key Insights: Predictive Services Adaptive Caching, Bandits, & Management Manageable Online/Offline Learning Latency vs. Accuracy

Future of Learning Systems Dat a e Ad rv Se ap t Actions Train Mode l

Thank You Joseph E. Gonzalez jegonzal@cs. berkeley. edu, Assistant Professor @ UC Berkeley joseph@dato. com, Co-Founder @ Dato