Optimization on Graphs Speaker Rasmus Kyng ETH Zurich

Isotonic Regression Height of child Predict child’s height from mother’s height Model? Height of

Isotonic Regression Height of child Predict child’s height from mother’s height Model? Increasing function?

Isotonic Regression Height of child Height of mother

Isotonic Regression Cost: Height of child Height of mother

Isotonic Regression igh t o f ch ild Age of child Height of mother

Isotonic Regression Height of mother Age of child Height of mother Taller mother AND

Isotonic Regression Age of child Height of mother Cost:

Isotonic Regression He igh t o f ch ild Age of child Height of

Optimization Primer 2 nd order 1 st order

Convex Functions No negative eigenvalues!

Second Derivatives If every term looks like the sum is an M-matrix 2 -by-2

Second Derivatives If every term looks like 2 -by-2 PSD non-negative eigenvalues Newton Step

Optimization on Graphs Variants of this framework have been used for many optimization problems:

Optimization on Graphs M-matrix & Laplacian matrix solvers used for many other problems in

Solving M-matrices Daitch-Spielman ‘ 08 gave reduction to Laplacian matrices Spielman-Teng ’ 04 gave

Optimization on Graphs: Toward Practical Algorithms

Julia Package: Laplacians. jl Graph Apx Elim CMG LAMG 1000 x 1000 grid 6.

Thanks! github. com/danspielman/Laplacians. jl/ Slides: rasmuskyng. com/#talks References Approximate Gaussian Elimination for Laplacians (R.

Slides: 62

Download presentation

Optimization on Graphs Speaker Rasmus Kyng ETH Zurich Dept. of Computer Science inf. ethz. ch

Optimization on Graphs ( ) Objective

Optimization on Graphs ( ) Objective ( )

Optimization on Graphs: An Example

Isotonic Regression Height of child Predict child’s height from mother’s height Model? Height of mother

Isotonic Regression Height of child Predict child’s height from mother’s height Model? Increasing function? Height of mother

Isotonic Regression Height of child Height of mother

Isotonic Regression Cost: Height of child Height of mother

Isotonic Regression igh t o f ch ild Age of child Height of mother He He igh t o f ch ild Height of mother

Isotonic Regression Height of mother Age of child Height of mother Taller mother AND older child

Isotonic Regression Age of child Height of mother Cost:

Isotonic Regression He igh t o f ch ild Age of child Height of mother Cost:

Optimization on Graphs Cost:

Optimization on Graphs

Optimization on Graphs: Fast Algorithms

Optimization Primer

Optimization Primer 2 nd order 1 st order

Optimization Primer

Optimization Primer A good update step!

Second Order Methods

Graphs and Hessian Linear Equations

Convex Functions

Convex Functions No negative eigenvalues!

Hessians & Graphs

Second Derivatives If every term looks like the sum is an M-matrix 2 -by-2 PSD non-negative eigenvalues

Second Derivatives

Second Derivatives If every term looks like the sum is an M-matrix 2 -by-2 PSD non-negative eigenvalues

Second Derivatives If every term looks like 2 -by-2 PSD non-negative eigenvalues Newton Step can be computed in nearly linear time!

Optimization on Graphs Variants of this framework have been used for many optimization problems: Maximum flow [DS 08, CKMST 11, KMP 12, Mad 13, Mad 16] Minimum cost flow [LS 14] Negative weight shortest paths [CMSV 16] Isotonic regression [KRS 15] Regularized Lipschitz learning on graphs [KRSS 15] P-norm flow [KPSW 19]

Optimization on Graphs M-matrix & Laplacian matrix solvers used for many other problems in TCS Learning on graphs [ZGL 03, ZS 04, ZBLWS 04] Graph partitioning [OSV 12] Sampling random spanning trees [KM 09, MST 15, DKPRS 17, DPR 17, S 18] Graph sparsification [SS 08, LKP 12, KPPS 17]

Solving M-matrices Daitch-Spielman ‘ 08 gave reduction to Laplacian matrices Spielman-Teng ’ 04 gave fast Laplacian solver Since, much work on trying to get a practical solver: [KMP 10, KMP 11, KOSZ 13, LS 13, CKM+14, PS 14, LPS 16] We finally succeeded? Kyng-Sachdeva ’ 16

Optimization on Graphs: Toward Practical Algorithms

Julia Package: Laplacians. jl Graph Apx Elim CMG LAMG 1000 x 1000 grid 6. 3 3. 0 15 Other Apx Elim = Approximate Elimination (K & Spielman) CMG = Combinatorial Multigrid (Koutis) LAMG = Lean Algebraic Multigrid (Livne & Brandt)

Julia Package: Laplacians. jl Graph Apx Elim CMG LAMG 1000 x 1000 grid 6. 3 3. 0 15 100 x 100 grid 8. 7 3. 4 15 Other Apx Elim = Approximate Elimination (K & Spielman) CMG = Combinatorial Multigrid (Koutis) LAMG = Lean Algebraic Multigrid (Livne & Brandt)

Julia Package: Laplacians. jl Graph Apx Elim CMG LAMG 1000 x 1000 grid 6. 3 3. 0 15 100 x 100 grid 8. 7 3. 4 15 Rand 4 -regular 17. 2 12 13. 7 Other 1. 7 (CG) Apx Elim = Approximate Elimination (K & Spielman) CMG = Combinatorial Multigrid (Koutis) LAMG = Lean Algebraic Multigrid (Livne & Brandt) CG = Conjugate Gradient

Julia Package: Laplacians. jl Graph Apx Elim CMG LAMG 1000 x 1000 grid 6. 3 3. 0 15 100 x 100 grid 8. 7 3. 4 15 Rand 4 -regular 17. 2 12 13. 7 1. 7 (CG) Pref attach 9. 5 10. 8 5. 1 3. 2 (ICC) Other Apx Elim = Approximate Elimination (K & Spielman) CMG = Combinatorial Multigrid (Koutis) LAMG = Lean Algebraic Multigrid (Livne & Brandt) CG = Conjugate Gradient, ICC = Incomplete Cholesky

Julia Package: Laplacians. jl Graph Apx Elim CMG LAMG 1000 x 1000 grid 6. 3 3. 0 15 100 x 100 grid 8. 7 3. 4 15 Rand 4 -regular 17. 2 12 13. 7 1. 7 (CG) Pref attach 9. 5 10. 8 5. 1 3. 2 (ICC) Chimera (2500100, 11) 28 350 74 Other Apx Elim = Approximate Elimination (K & Spielman) CMG = Combinatorial Multigrid (Koutis) LAMG = Lean Algebraic Multigrid (Livne & Brandt) CG = Conjugate Gradient, ICC = Incomplete Cholesky

Julia Package: Laplacians. jl Graph Apx Elim CMG LAMG 1000 x 1000 grid 6. 3 3. 0 15 100 x 100 grid 8. 7 3. 4 15 Rand 4 -regular 17. 2 12 13. 7 1. 7 (CG) Pref attach 9. 5 10. 8 5. 1 3. 2 (ICC) Chimera (2500100, 11) 28 350 74 Min Cost Flow 1 5. 9 4. 3 >60 Min Cost Flow 2 6. 8 23 29 Other Apx Elim = Approximate Elimination (K & Spielman) CMG = Combinatorial Multigrid (Koutis) LAMG = Lean Algebraic Multigrid (Livne & Brandt) CG = Conjugate Gradient, ICC = Incomplete Cholesky

Julia Package: Laplacians. jl Graph Apx Elim CMG LAMG 1000 x 1000 grid 6. 3 3. 0 15 100 x 100 grid 8. 7 3. 4 15 Rand 4 -regular 17. 2 12 13. 7 1. 7 (CG) Pref attach 9. 5 10. 8 5. 1 3. 2 (ICC) Chimera (2500100, 11) 28 350 74 Min Cost Flow 1 5. 9 4. 3 >60 Min Cost Flow 2 6. 8 23 29 Other Summary: Approximate Elimination processes between 300 k and 500 k entries per second, for 8 digit accuracy Others vary widely

Thanks! github. com/danspielman/Laplacians. jl/ Slides: rasmuskyng. com/#talks References Approximate Gaussian Elimination for Laplacians (R. Kyng, S. Sachdeva 2016) Faster Approximate Lossy Generalized Flow via Interior Point Algorithms (S. Daitch, D. Spielman 2008) Fast, Provable Algorithms for Isotonic Regression in all Lp norms (R. Kyng, A. B. Rao, S. Sachdeva 2015) Nearly-Linear Time Algorithms for Preconditioning and Solving SDD Linear Systems (D. Spielman, S. -H. Teng 2004)

Related presentations