Random projections and depth hierarchy theorems Rocco Servedio

Random projections and depth hierarchy theorems Rocco Servedio Columbia University St. Petersburg, Russia May 2016

Two small-depth-circuit lower bounds Rocco Servedio Columbia University St. Petersburg, Russia May 2016

The goal of circuit complexity Goal: Strong lower bounds on size of unrestricted circuits computing explicit functions. Holy grail: size lower bound Unfortunately, we have no idea how to prove that an explicit nvariable Boolean function cannot be computed by unrestricted circuits of size 10 n. So, we look at restricted circuits: • Small-depth circuits • Monotone circuits (both results of this talk) (first result)

This talk: Two recent circuit lower bounds 1. Lower bound for monotone small-depth circuits of Maj gates that compute the “addition” function 2. Lower bound for small-depth circuits of AND/OR/NOT gates (non-monotone) computing “small-distance connectivity. ” Similarities: Both are • Lower bounds against small-depth circuits • Near-optimal (or actually optimal) Differences: • The problems (and proofs/techniques) are totally different.

Part I: Addition is exponentially harder than counting for shallow monotone circuits Joint work with Xi Chen Columbia Igor Oliveira Columbia/Charles University

Boolean circuit complexity A circuit with AND/OR/NOT gates: (de Morgan circuit) Size = # of wires. Depth = depth. (Unbounded fan-in. ) …. . Can consider circuits with other types of gates as well instead of AND/OR: • Maj circuit: can use gates that computes the majority function • “weighted threshold” circuit: can use “LTF gates” -- gates that compute “weighted threshold” functions, i. e. that output 1 iff

Monotone circuits: come in different flavors • Monotone de Morgan circuits: No negations, only AND/OR gates • Monotone Maj circuits: No negations, only Maj gates (and constants 0, 1) Maj • Monotone weighted threshold circuits: No negative weights allowed: each LTF gate computes some function where each

What do we know about monotone circuits? A fair bit! Celebrated lower bounds: • [Razborov 85, Andreev 85, Alon. Boppana 87, Karchmer. Wigderson 88, Tardos 88, Raz. Wigderson 92, Harnik. Raz 00, Pitassi. Goos 14]: strong lower bounds on monotone AND/OR-circuit size and depth for explicit monotone functions • [Ajtai. Gurevich 87]: Explicit monotone function in poly(n)-size AC 0 which does not have poly(n)-size monotone-AC 0 circuits O(1)-depth, poly(n)-size AND, OR, NOT gates …. . Monotone function Non-monotone O(1)-depth, poly(n)-size circuit that computes a monotone function that’s not computable by any O(1)-depth, poly(n)-size AND, OR, gates only …. .

What else do we know about monotone circuits? We also know non-trivial upper bounds: • [Ajtai. Komlos. Szemeredi 83, Valiant 84]: has poly(n)-size monotone AND/OR formulas (tree circuits) And more.

But we don’t know everything… Monotone-TC 0: Constant-depth, poly(n)-size circuits of monotone weighted threshold gates Monotone-NC 1: poly(n)-size monotone AND/OR formulas (= poly-size, log-depth, fan-in 2 monotone formulas) (= poly-size, log-depth, fan-in 2 monotone circuits) • [Yao 89, Hastad. Goldmann 90]: Monotone-NC 1 not contained in monotone-TC 0 log(n)-depth, O(n)-size not computed by any poly(n)-size, O(1)-depth circuit of monotone weighted threshold gates. . . . . …. .

A first question: Monotone-TC 0: Constant-depth, poly(n)-size circuits of monotone weighted threshold gates Monotone-NC 1: poly(n)-size monotone AND/OR formulas [Yao 89, Hastad. Goldmann 90]: Monotone-NC 1 not contained in monotone-TC 0. Question 1: Is monotone-TC 0 contained in monotone-NC 1 ? Equivalent to: Is the “univeral halfspace” computable in monotone-NC 1 ? Universal halfspace:

A close shave… Monotone-TC 0: Constant-depth, poly(n)-size circuits of monotone weighted threshold gates Monotone-NC 1: poly(n)-size monotone AND/OR formulas [Yao 89, Hastad. Goldmann 90]: Monotone-NC 1 not contained in monotone-TC 0. Question 1: Is monotone-TC 0 contained in monotone-NC 1 ? • [Beimel. Weinreb 05]: “almost Yes”: monotone-TC 0 is contained in n. O(log n)-size monotone-NC 1. • But…it is believed (at least by some people) that the answer is “No” – that n. O(log n) is the smallest possible size for monotone formulas computing the universal halfspace.

A second question Recall [Ajtai. Gurevich 87]: There is a monotone function in AC 0 which does not have poly(n)-size monotone-AC 0 circuits. O(1)-depth, poly(n)-size AND, OR, NOT gates not computable by any …. . AND, OR, gates only O(1)-depth, poly(n)-size …. . Monotone function in AC 0 Question 2: Could there be a monotone function in AC 0 which does not have poly(n)-size monotone circuits, period? O(1)-depth, poly(n)-size AND, OR, NOT gates poly(n)-size, any depth not computable by any …. . Monotone function in AC 0 AND, OR, gates only …. .

Question 1: Is monotone-TC 0 contained in monotone-NC 1 ? Question 2: Could there be a monotone function in AC 0 which does not have poly(n)-size monotone circuits, period? This talk will answer of these questions! (But it describes partial progress on both of them. )

Question 1 revisited Question 1: Can monotone-NC 1 compute monotone-TC 0? Possible approach, if you are an optimist: • Recall [AKS, Valiant]: monotone-NC 1 can compute . • So if constant-depth monotone circuits of Maj gates can compute monotone-TC 0, then monotone-NC 1 circuits can compute monotone-TC 0. (constant depth only polynomial blowup in size when we compose. ) First main result: This won’t work.

First result Define the function • The input is a list of • The output is 1 iff as follows: –bit strings (which we view as binary #s ) Example: input is For this input, have 150 + 120 + 203 } = 473 > 256

First result is computed by a single monotone threshold gate: Gates “can count” Theorem 1: Any depth-d monotone Maj circuit computing must have size “Addition is exponentially harder than counting for shallow monotone circuits”

Discussion Theorem 1: Any depth-d monotone Maj circuit computing must have size • Theorem is tight in two ways: • Any n-variable monotone weighted threshold function (such as ) can be computed by a poly-size depth-2 non-monotone Maj circuit [Goldmann. Hastad. Razborov 92, Goldmann. Karpinski 93, Hofmeister 96, Amano. Maruoka 05] • can be computed by a depth-d, sizemonotone Maj circuit • Theorem answers question of [GK 93, Hastad 10, Hastad 14] • [Hofmeister 92] proved the d=2 case

Second question and result Question 2: Could there be a monotone function in AC 0 which does not have poly(n)-size monotone circuits? O(1)-depth, poly(n)-size AND, OR, NOT gates poly(n)-size, any depth not computable by any …. . Monotone function in AC 0 AND, OR, gates only …. . Motivated by Theorem: [Ajtai. Gurevich 87] There is an n-variable monotone function such that • has a poly(n)-size, constant-depth AND/OR/NOT circuit; but • Any constant-depth AND/OR circuit for has size nw(1)

The Ajtai-Gurevich function n columns Input 111111101101011 1110001001000111101 1111110011100 0100101111010110111 …. . . . 11011001010101 log n rows iff (# of 1 s) > (n log n)/2 Theorem: [Ajtai. Gurevich 87] The above monotone function • has a poly(n)-size, constant-depth AND/OR/NOT circuit, but • any depth-d AND/OR circuit for has size at least where

Second main result: Strengthening of Ajtai-Gurevich Theorem: There is an -variable monotone function such that • has a poly(n)-size, depth-3 AND/OR/NOT circuit; • Any depth-d monotone-Maj circuit computing has size at least Compare with [Ajtai. Gurevich 87]: • stronger size bound (exponential for every d) • works against stronger circuits (Maj gates) • very different proof (no switching lemma)

Proving Theorem 2 from Theorem 1: Any depth-d monotone Maj circuit computing must have size Theorem 2: There is an -variable monotone function such that • has a poly(n)-size, depth-3 AND/OR/NOT circuit; • Any depth-d monotone Maj-circuit computing has size at least The function is where

A glimpse at the proof of Theorem 1: Any depth-d Maj circuit computing must have size Proof is a induction. Inductively construct distributions that are “hard” for deeper and deeper circuits.

A glimpse at the proof of Theorem 1 Let Initial pair of distributions Every has . : both over .

Initial hardness: against depth-1 monotone MAJ circuits Not hard to show: Any depth-1 monotone Maj circuit of size satisfies

Key step in the proof of Theorem 1 Key lemma: Suppose any depthmonotone Maj circuit over of size satisfies Then any depth- monotone Maj circuit of size satisfies where The case gives theorem. over

To construct from over the proof actually needs three pairs of distributions: over and over some other domain that’s intermediate between and Proof maintains very tight control over all three pairs of distributions.

Related questions: Average-case lower bounds? Gil Kalai has asked several “uniform distribution” questions about related circuit classes: Question: Can monotone-TC 0 approximate Rec-Maj-3? Question: (“approximate Ajtai-Gurevich”) Are there monotone functions in AC 0 that can’t be approximated by monotone-AC 0? Question: (“approximate Ajtai-Gurevich for TC 0”) Are there monotone functions in TC 0 that can’t be approximated by monotone-TC 0?

An observation and a question Theorem 1: Any depth-d monotone Maj circuit computing must have size Observation: “approximate” version of Theorem 1 does not hold already for depth-1 circuits of Maj gates, in a strong sense: • Any n-variable monotone weighted threshold function is 0. 99 approximated by a single Maj gate of size O(n) [De. Diakonikolas. Feldman. S 12]. Question: what about “d>1” version of the above? Can constantdepth monotone-Maj circuits approximate constant-depth monotone weighted threshold functions?

End of Part I Questions?

Part II: Near-Optimal Small Depth Lower Bounds for Small-Distance Connectivity Joint work with Xi Chen Columbia Igor Oliveira Columbia/Charles University Li-Yang Tan TTI

Small-distance connectivity The problem: STCONN(k(n)): Given an n-node graph G with two distinguished nodes s and t, does there exist an s-to-t path of length at most k(n)? At most k(n) edges on the path s t (Don’t worry about whether G is directed or undirected. )

The lower bound question Our question: Circuit complexity of STCONN(k(n)) – in fact, smalldepth circuit complexity of STCONN(k(n)). What size is needed for a depth-d AND/OR/NOT circuit to compute STCONN(k(n))? variables, one for each possible edge

Why ask this question? Strong motivation from study of space-bounded computation. Savitch’s algorithm for directed connectivity uses space. Any improvement gives . A simple reduction shows: If there is any k=k(n)<n such that there are poly(n)-size, o(log k)depth circuits for STCONN(k), then Savitch can be improved! Motivates study of small-depth circuits for STCONN(k).

Previously known bounds Large gap. Based on “repeated squaring” (Savitch)

Our result Theorem: For any circuit for and any must have size any depth-

The current landscape of upper and lower bounds Much smaller gap…and may not be closed anytime soon; getting here or here would imply

Our approach Two main ingredients: 1. Easy reduction that lets us get graphs out of the picture quickly and painlessly. Thanks to the reduction, it’s just a question about circuits. 2. New lower bounds on size of depth-d circuits that compute a particular depth-d’ circuit (a sort of depth hierarchy), proved using random projections. d < d’

First ingredient: The reduction. From graphs to circuits We’re interested in getting from node s to node t. Graphs correspond to bitstrings: presence/absense of an edge corresponds to variable being 1/0. s t

From graphs to circuits How about OR? s t

Easy to compose… s (nothing fancy – use a fresh variable for each edge) t

From monotone read-once formulas to series-parallel graphs

From monotone read-once formulas to series-parallel graphs formula f graph Gf If f has r levels of AND gates with fanins a 1, …, ar, then every shortest path in Gf has length a 1…ar. Given an assignment z, have f(z)=1 iff G(f, z) has an s-to-t path of length a 1…ar.

Goodbye, graphs Goal: Size lower bound on depth-d circuits computing STCONN(k). Thanks to reduction, this has become Goal: Size lower bound on depth-d circuits computing a monotone read-once formula f in which (product of AND-gate fanins across all depths) = k. Let’s cook up an f for which we could try to prove such a size lower bound.

The formula we want Some requirements: • Formula f better have depth > d (so it has a chance of being hard for circuits of depth d) • Product of AND-gate fan-ins across all depths must = k “Skewed Sipser functions”: • Every OR fan-in = w • Every AND fan-in = u<<w • d layers of ANDs (so k=ud) …. . . fan-in w . . . fan-in u<w …. .

All that remains: a straight-up circuit lower bound Skewed Sipser function has depth 2 d and size n=(uw)d. Recall u=AND-fan-in, w=OR-fan-in, k=ud. …. . . . . fan-in w. . . fan-in u<w …. . . . To prove the desired lower bound for STCONN(k), suffices to show: Theorem: Any depth-d circuit computing skewed Sipser function must have size a depth-(2 d) circuit!

What this might remind you of: Hastad’s depth hierarchy theorem [Hastad 86] proved that depth-d circuits require exp(n 1/d) size to compute the “unskewed” Sipser function of depth d+1. But there are important differences: • Need size lower bound for circuits of half the depth (instead of one less) • Need to handle “skewed” Sipser functions (small AND-fan-in) Indeed, if u = AND-fan-in = 2 (an interesting case!), circuits of depth (2 d-1) can compute the depth-2 d skewed Sipser function in fixed poly(n) size!

a depth-(2 d) circuit Theorem: Any depth-d circuit computing skewed Sipser function must have size Hastad’s depth hierarchy theorem: proved using random restrictions Our theorem (above): proved using random projections

What’s a random projection? Have n Boolean variables x 1, …, xn. Restriction: Each xi set to constant or “survives”: xi 0 1 xi (denoted *) Projection: Each xi set to constant or new formal variable xi 0 1 yj where { y 1, …, ym } are new formal variables Our proof: { y-variables } much smaller than { x-variables }. Distinct x-variables collide to same y-variable (depending on the structure of the skewed Sipser formula)

Structure of our argument Requirements for our theorem: 1. Skewed Sipser (depth 2 d) retains structure after random projection • Sequence of (d-1) projections each “peel off” two layers from skewed Sipser, but don’t “damage” it further 2. Small AC 0 d circuits collapse to “simple” function under random projection • Each sub-restriction collapses bottom depth-2 circuit to shallow decision tree (as usual) Key technical ingredient: random projection switching lemma

A peek at the random projection switching lemma • Unusual distribution over random projections • Unusual (and delicate) requirements: the condition is essential (the conclusion provably does not hold for some for )

Conclusion First part of talk: • Small-depth circuits of low-weight monotone LTF gates (Maj gates) can’t efficiently simulate a single monotone high-weight LTF gate (careful inductive construction of “hard distributions”) • Many “uniform distribution” questions for future work Second part of talk: • Near-optimal lower bounds on size of small-depth circuits for “small-distance connectivity” (random projection switching lemma) • What else can we do with random projections?

Thankyou!