Petuum An IterativeConvergent Distributed Machine Learning Framework SU

Petuum: An Iterative-Convergent Distributed Machine Learning Framework SU YUXIN JAN 20, 2014

Outline Introduction Implementation Questions Demo

Introduction to Petuum

Bulk Synchronous Parallel

Asynchronous Parameters read / update at any time

Stale Synchronous Parallel

Convergence

Programming read(table, row, col) inc(table, row, col, value) iteration()

Implementation

Overview in Logic

Overview in the Real

Main Components

Table

Consistency. Controller: : Do. Get()

Consistency. Controller: : iterate()

Server: : Get. Row()

Least-Recently-Used(LRU) Strategy

Questions

Is Lock-Free Possible ? Data exchange in real-time ? next …

Is Auto-Rescheduling Possible ? sub-centralized server reduce communication cost

Is Auto-Partition Possible ? Run ML algorithms like that in a single thread A Solution for all ML algorithms

In-Memory or In-Storage ? Data capacity is greater than memory size. Memory should be a cache for disk storage. Solution for disk storage: Hadoop Spark ….

New Schema to Reduce the Upper Bound?

STRADS Scheduler Variable Correlations Auto-Parallelization Dynamic Prioritization Monitor the contribution of variables to objective function Load-Balancing in Task

Demo Switch to my laptop …