Fast Spectral Algorithms from SumofSquares Proofs Tensor Decomposition

  • Slides: 84
Download presentation
Fast Spectral Algorithms from Sum-of-Squares Proofs: Tensor Decomposition and Planted Sparse Vectors Sam Hopkins

Fast Spectral Algorithms from Sum-of-Squares Proofs: Tensor Decomposition and Planted Sparse Vectors Sam Hopkins Tselil Schramm Jonathan Shi David Steurer Cornell UC Berkeley Cornell

Competing Themes in Algorithms Polynomial time = Efficient algorithms BUT Stronger convex programs ↓

Competing Themes in Algorithms Polynomial time = Efficient algorithms BUT Stronger convex programs ↓ better (poly-time) algorithms ↓ (which aren’t really efficient) versus

Algorithms, Hierarchies, and Running Time HUGE, accurate SDP Relaxation Hard problem SDP Relaxation add

Algorithms, Hierarchies, and Running Time HUGE, accurate SDP Relaxation Hard problem SDP Relaxation add variables & constraints

Algorithms, Hierarchies, and Running Time Better approximation ratios, noise tolerance, than linear programs, semidefinite

Algorithms, Hierarchies, and Running Time Better approximation ratios, noise tolerance, than linear programs, semidefinite programs. New Algorithms for: • Scheduling [Levey-Rothvoss] • Independent sets in bounded-degree graphs [Bansal, Chlamtac] • Independent sets in hypergraphs [Chlamtac, Chlamtac-Singh] • Planted problems [Barak-Kelner-Steurer, Barak-Moitra, Hopkins • Shi-Steurer, Ge-Ma, Raghavendra-Rao-Schramm, Ma-Shi-Steurer] Unique games [Barak-Raghavendra-Steurer, Barak-Brandao. Harrow-Kelner-Steurer-Zhou]

Algorithms, Hierarchies, and Running Time

Algorithms, Hierarchies, and Running Time

Algorithms, Hierarchies, and Running Time This work: fast spectral algorithms with matching guarantees for

Algorithms, Hierarchies, and Running Time This work: fast spectral algorithms with matching guarantees for planted problems. Use eigenvectors of matrix polynomials

Algorithms, Hierarchies, and Running Time This work: fast spectral algorithms with matching HUGE, accurate

Algorithms, Hierarchies, and Running Time This work: fast spectral algorithms with matching HUGE, accurate guarantees for planted problems. SDP Relaxation

Results (1) Planted Sparse Vector (2) Random Tensor Decomposition (3) Tensor Principal Component Analysis

Results (1) Planted Sparse Vector (2) Random Tensor Decomposition (3) Tensor Principal Component Analysis

Results • So. S (previous champion) has to solve large SDP (much larger than

Results • So. S (previous champion) has to solve large SDP (much larger than input size)

Results (1) Planted Sparse Vector Match So. S guarantees, nearly-linear time (3) Tensor Principal

Results (1) Planted Sparse Vector Match So. S guarantees, nearly-linear time (3) Tensor Principal Component Analysis Match So. S guarantees, linear time

Results (1) Planted Sparse Vector Match So. S guarantees, nearly-linear time (3) Tensor Principal

Results (1) Planted Sparse Vector Match So. S guarantees, nearly-linear time (3) Tensor Principal Component Analysis Match So. S guarantees, linear time

Planted Sparse Vector

Planted Sparse Vector

Planted Sparse Vector

Planted Sparse Vector

The Speedup Recipe So. S algorithm (large SDP) Spectral algorithm with big matrices Fast

The Speedup Recipe So. S algorithm (large SDP) Spectral algorithm with big matrices Fast spectral algorithm

The Speedup Recipe So. S algorithm (large SDP) Spectral algorithm with big matrices Dual

The Speedup Recipe So. S algorithm (large SDP) Spectral algorithm with big matrices Dual certificates from So. S SDP (variant of primal-dual) Fast spectral algorithm

The Speedup Recipe Compression to small matrices So. S algorithm (large SDP) Spectral algorithm

The Speedup Recipe Compression to small matrices So. S algorithm (large SDP) Spectral algorithm with big matrices Dual certificates from So. S SDP (variant of primal-dual) Fast spectral algorithm

The Speedup Recipe Compression to small matrices Different from — matrix multiplicative weights [Arora-Kale]

The Speedup Recipe Compression to small matrices Different from — matrix multiplicative weights [Arora-Kale] — simpler spectral algorithms So. S algorithm (large SDP) Spectral algorithm with big matrices Dual certificates from So. S SDP (variant of primal-dual) Fast spectral algorithm

The Speedup Recipe Not local rounding [Guruswami-Sinop] So. S algorithm (large SDP) Compression to

The Speedup Recipe Not local rounding [Guruswami-Sinop] So. S algorithm (large SDP) Compression to small matrices Different from — matrix multiplicative weights [Arora-Kale] — simpler spectral algorithms Spectral algorithm with big matrices Dual certificates from So. S SDP (variant of primal-dual) Fast spectral algorithm

The Speedup Recipe So. S algorithm (large SDP) Spectral algorithm with big matrices Fast

The Speedup Recipe So. S algorithm (large SDP) Spectral algorithm with big matrices Fast spectral algorithm

The Speedup Recipe So. S algorithm (large SDP) Spectral algorithm with big matrices Fast

The Speedup Recipe So. S algorithm (large SDP) Spectral algorithm with big matrices Fast spectral algorithm

The Speedup Recipe Spectral algorithm with big matrices So. S algorithm (large SDP) Fast

The Speedup Recipe Spectral algorithm with big matrices So. S algorithm (large SDP) Fast spectral algorithm signal noise

The Speedup Recipe Spectral algorithm with big matrices So. S algorithm (large SDP) Fast

The Speedup Recipe Spectral algorithm with big matrices So. S algorithm (large SDP) Fast spectral algorithm signal noise basis Planted sparse vector

The Speedup Recipe So. S algorithm (large SDP) Spectral algorithm with big matrices Fast

The Speedup Recipe So. S algorithm (large SDP) Spectral algorithm with big matrices Fast spectral algorithm

The Speedup Recipe So. S algorithm (large SDP) Spectral algorithm with big matrices Fast

The Speedup Recipe So. S algorithm (large SDP) Spectral algorithm with big matrices Fast spectral algorithm

The Speedup Recipe Spectral algorithm with big matrices So. S algorithm (large SDP) Fast

The Speedup Recipe Spectral algorithm with big matrices So. S algorithm (large SDP) Fast spectral algorithm

The Speedup Recipe Spectral algorithm with big matrices So. S algorithm (large SDP) Fast

The Speedup Recipe Spectral algorithm with big matrices So. S algorithm (large SDP) Fast spectral algorithm redundant information with tensor structure

The Speedup Recipe Spectral algorithm with big matrices So. S algorithm (large SDP) Fast

The Speedup Recipe Spectral algorithm with big matrices So. S algorithm (large SDP) Fast spectral algorithm redundant information with tensor structure

The Speedup Recipe Spectral algorithm with big matrices So. S algorithm (large SDP) Fast

The Speedup Recipe Spectral algorithm with big matrices So. S algorithm (large SDP) Fast spectral algorithm

Partial Trace

Partial Trace

Partial Trace

Partial Trace

Partial Trace

Partial Trace

Partial Trace

Partial Trace

 Partial Trace

Partial Trace

 Partial Trace

Partial Trace

 Partial Trace

Partial Trace

 Partial Trace

Partial Trace

 Partial Trace

Partial Trace

 Partial Trace Conclusion: signal-to-noise ratio is preserved!

Partial Trace Conclusion: signal-to-noise ratio is preserved!

The Speedup Recipe Spectral algorithm with big matrices So. S algorithm (large SDP) Fast

The Speedup Recipe Spectral algorithm with big matrices So. S algorithm (large SDP) Fast spectral algorithm

The Speedup Recipe Spectral algorithm with big matrices So. S algorithm (large SDP) Fast

The Speedup Recipe Spectral algorithm with big matrices So. S algorithm (large SDP) Fast spectral algorithm

The Speedup Recipe Avoid explicitly computing large matrix Spectral algorithm with big matrices So.

The Speedup Recipe Avoid explicitly computing large matrix Spectral algorithm with big matrices So. S algorithm (large SDP) Fast spectral algorithm

Resulting Algorithms are Simple and Spectral

Resulting Algorithms are Simple and Spectral

Resulting Algorithms are Simple and Spectral

Resulting Algorithms are Simple and Spectral

Resulting Algorithms are Simple and Spectral

Resulting Algorithms are Simple and Spectral

Resulting Algorithms are Simple and Spectral basis

Resulting Algorithms are Simple and Spectral basis

Resulting Algorithms are Simple and Spectral basis

Resulting Algorithms are Simple and Spectral basis

Conclusions By exploiting tensor structure in dual certificates, randomness in inputs, impractical So. S

Conclusions By exploiting tensor structure in dual certificates, randomness in inputs, impractical So. S algorithms can become practical spectral algorithms. Thanks For Coming!

The Resulting Algorithms are Simple and Spectral

The Resulting Algorithms are Simple and Spectral

The Resulting Algorithms are Simple and Spectral

The Resulting Algorithms are Simple and Spectral

The Resulting Algorithms are Simple and Spectral

The Resulting Algorithms are Simple and Spectral

Contrast to Previous Speedup Approaches (Matrix) Multiplicative Weights [Arora-Kale]: Cannot go faster than matrix-vector

Contrast to Previous Speedup Approaches (Matrix) Multiplicative Weights [Arora-Kale]: Cannot go faster than matrix-vector multiplication for matrices in the underlying SDP.

The Speedup Recipe (1) Understand Spectrum of So. S Dual Certificate (avoid SDP) (2)

The Speedup Recipe (1) Understand Spectrum of So. S Dual Certificate (avoid SDP) (2) Reduce Dimensions via Tensor Structure in Dual Cert.

The Speedup Recipe • (2) Reduce Dimensions via Tensor Structure in Dual Cert.

The Speedup Recipe • (2) Reduce Dimensions via Tensor Structure in Dual Cert.

The Speedup Recipe •

The Speedup Recipe •

The Speedup Recipe Understand Spectrum of So. S Dual Certificate (1) In 2 dimensions:

The Speedup Recipe Understand Spectrum of So. S Dual Certificate (1) In 2 dimensions: the result: a spectral algorithm using high-dimensional matrices.

The Speedup Recipe Understand Spectrum of So. S Dual Certificate (1) the result: a

The Speedup Recipe Understand Spectrum of So. S Dual Certificate (1) the result: a spectral algorithm using high-dimensional matrices.

The Speedup Recipe Understand Spectrum of So. S Dual Certificate (1) the result: a

The Speedup Recipe Understand Spectrum of So. S Dual Certificate (1) the result: a spectral algorithm using high-dimensional matrices.

The Speedup Recipe Understand Spectrum of So. S Dual Certificate (1) the result: a

The Speedup Recipe Understand Spectrum of So. S Dual Certificate (1) the result: a spectral algorithm using high-dimensional matrices.

Thanks For Coming!

Thanks For Coming!

Can We Use Previous Approaches to Speeding Up Relaxation-Based Algorithms?

Can We Use Previous Approaches to Speeding Up Relaxation-Based Algorithms?

A Benchmark Problem: Planted Sparse Vector • . . . Random linear combinations. .

A Benchmark Problem: Planted Sparse Vector • . . . Random linear combinations. . .

A Benchmark Problem: Planted Sparse Vector

A Benchmark Problem: Planted Sparse Vector

A Benchmark Problem: Planted Sparse Vector

A Benchmark Problem: Planted Sparse Vector

A Benchmark Problem: Planted Sparse Vector Related to compressed sensing, dictionary learning, sparse pca,

A Benchmark Problem: Planted Sparse Vector Related to compressed sensing, dictionary learning, sparse pca, shortest codeword, small-set expansion

A Benchmark Problem: Planted Sparse Vector Simple problem where sum-of-squares (So. S) hierarchy beats

A Benchmark Problem: Planted Sparse Vector Simple problem where sum-of-squares (So. S) hierarchy beats LP, (small) SDPs, local search

Previous Work (recovery version) Authors [Spielman-Wang-Wright, Demanet-Hand] Subspace Dimension Technique Linear Programming

Previous Work (recovery version) Authors [Spielman-Wang-Wright, Demanet-Hand] Subspace Dimension Technique Linear Programming

Previous Work (recovery version) Authors Subspace Dimension Technique [Spielman-Wang-Wright, Demanet-Hand] Linear Programming Folklore Semidefinite

Previous Work (recovery version) Authors Subspace Dimension Technique [Spielman-Wang-Wright, Demanet-Hand] Linear Programming Folklore Semidefinite Programming

Previous Work (recovery version) Authors Subspace Dimension Technique [Spielman-Wang-Wright, Demanet-Hand] Linear Programming Folklore Semidefinite

Previous Work (recovery version) Authors Subspace Dimension Technique [Spielman-Wang-Wright, Demanet-Hand] Linear Programming Folklore Semidefinite Programming [Qu-Sun-Wright] Alternating Minimization

Previous Work (recovery version) Authors Subspace Dimension Technique [Spielman-Wang-Wright, Demanet-Hand] Linear Programming Folklore Semidefinite

Previous Work (recovery version) Authors Subspace Dimension Technique [Spielman-Wang-Wright, Demanet-Hand] Linear Programming Folklore Semidefinite Programming [Qu-Sun-Wright] Alternating Minimization [Barak-Brandao-Harrow-Kelner-Steurer-Zhou, Barak-Kelner-Steurer] So. S Hierarchy

Previous Work (recovery version) Authors [Spielman-Wang-Wright, Demanet-Hand] Subspace Dimension Technique Linear Programming All require

Previous Work (recovery version) Authors [Spielman-Wang-Wright, Demanet-Hand] Subspace Dimension Technique Linear Programming All require polynomial loss in Folklore Semidefinite Programming sparsity or subspace dimension or both, compared with So. S. [Qu-Sun-Wright] Alternating Minimization [Barak-Brandao-Harrow-Kelner-Steurer-Zhou, Barak-Kelner-Steurer] So. S Hierarchy

Sum-of-Squares (and So. Sinspired) Algorithms Running Time Distinguishing Recovery [Barak-Brandao-Harrow-Kelner-Steurer. Zhou] [Barak-Kelner. Steurer]

Sum-of-Squares (and So. Sinspired) Algorithms Running Time Distinguishing Recovery [Barak-Brandao-Harrow-Kelner-Steurer. Zhou] [Barak-Kelner. Steurer]

Sum-of-Squares (and So. Sinspired) Algorithms Running Time Distinguishing Recovery [Barak-Brandao-Harrow-Kelner-Steurer. Zhou] [Barak-Kelner. Steurer]

Sum-of-Squares (and So. Sinspired) Algorithms Running Time Distinguishing Recovery [Barak-Brandao-Harrow-Kelner-Steurer. Zhou] [Barak-Kelner. Steurer]

Sum-of-Squares (and So. Sinspired) Algorithms Running Time Distinguishing Recovery [Barak-Brandao-Harrow-Kelner-Steurer. Zhou] [Barak-Kelner. Steurer] [Barak-Brandao-Harrow-Kelner-Steurer.

Sum-of-Squares (and So. Sinspired) Algorithms Running Time Distinguishing Recovery [Barak-Brandao-Harrow-Kelner-Steurer. Zhou] [Barak-Kelner. Steurer] [Barak-Brandao-Harrow-Kelner-Steurer. Zhou] (implicit)

Sum-of-Squares (and So. Sinspired) Algorithms Running Time Distinguishing Recovery [Barak-Brandao-Harrow-Kelner-Steurer. Zhou] [Barak-Kelner. Steurer] [Barak-Brandao-Harrow-Kelner-Steurer.

Sum-of-Squares (and So. Sinspired) Algorithms Running Time Distinguishing Recovery [Barak-Brandao-Harrow-Kelner-Steurer. Zhou] [Barak-Kelner. Steurer] [Barak-Brandao-Harrow-Kelner-Steurer. Zhou] (implicit) This work

Sum-of-Squares (and So. Sinspired) Algorithms Running Time Distinguishing Recovery [Barak-Brandao-Harrow-Kelner-Steurer. Zhou] [Barak-Kelner. Steurer] [Barak-Brandao-Harrow-Kelner-Steurer.

Sum-of-Squares (and So. Sinspired) Algorithms Running Time Distinguishing Recovery [Barak-Brandao-Harrow-Kelner-Steurer. Zhou] [Barak-Kelner. Steurer] [Barak-Brandao-Harrow-Kelner-Steurer. Zhou] (implicit) This work Running-time barrier from dimension of convex program

Sum-of-Squares (and So. Sinspired) Algorithms Running Time Distinguishing Recovery [Barak-Brandao-Harrow-Kelner-Steurer. Zhou] [Barak-Kelner. Steurer] [Barak-Brandao-Harrow-Kelner-Steurer.

Sum-of-Squares (and So. Sinspired) Algorithms Running Time Distinguishing Recovery [Barak-Brandao-Harrow-Kelner-Steurer. Zhou] [Barak-Kelner. Steurer] [Barak-Brandao-Harrow-Kelner-Steurer. Zhou] (implicit) This work Running-time barrier from dimension of convex program This work

[Barak et al]’s Distinguishing Algorithm Running Time Distinguishing Recovery [Barak-Brandao-Harrow-Kelner-Steurer. Zhou] [Barak-Kelner. Steurer] [Barak-Brandao-Harrow-Kelner-Steurer.

[Barak et al]’s Distinguishing Algorithm Running Time Distinguishing Recovery [Barak-Brandao-Harrow-Kelner-Steurer. Zhou] [Barak-Kelner. Steurer] [Barak-Brandao-Harrow-Kelner-Steurer. Zhou] (implicit) This work

Running Time Distinguishing Recovery [Barak-Brandao-Harrow-Kelner-Steurer. Zhou] [Barak-Kelner. Steurer] [Barak-Brandao-Harrow-Kelner-Steurer. Zhou] (implicit) This work

Running Time Distinguishing Recovery [Barak-Brandao-Harrow-Kelner-Steurer. Zhou] [Barak-Kelner. Steurer] [Barak-Brandao-Harrow-Kelner-Steurer. Zhou] (implicit) This work

Running Time Distinguishing Recovery [Barak-Brandao-Harrow-Kelner-Steurer. Zhou] [Barak-Kelner. Steurer] [Barak-Brandao-Harrow-Kelner-Steurer. Zhou] (implicit) This work

Running Time Distinguishing Recovery [Barak-Brandao-Harrow-Kelner-Steurer. Zhou] [Barak-Kelner. Steurer] [Barak-Brandao-Harrow-Kelner-Steurer. Zhou] (implicit) This work

(Breaking) The Dimension Barrier

(Breaking) The Dimension Barrier

(Breaking) The Dimension Barrier

(Breaking) The Dimension Barrier

 The Recipe (1) So. S algorithms (often) come with dual certificate constructions. (2)

The Recipe (1) So. S algorithms (often) come with dual certificate constructions. (2) Explicitly compute spectrum of dual certificate. (3) Compress to lower dimensions using randomness to avoid losing information.