On the Complexity of Approximating the VC Dimension

On the Complexity of Approximating the VC Dimension Chris Umans, Microsoft Research joint work with Elchanan Mossel, Microsoft Research June 2001

The VC Dimension • C collection of subsets of universe U • VC(C) = VC dimension of C: size of largest subset T U shattered by C • T shattered if every subset T’ T expressible as T (an element of C) • Example: C = {{a}, {a, c}, {a, b, c}, {b}} VC(C) = 2 {b, c} shattered by C • Plays important role in learning theory, finite automata, comparability theory, computational geometry

Complexity Questions Given C, compute VC(C) • since VC(C) log |C|, can compute in O(nlog n) time (Linial-Mansour-Rivest 88) • probably can’t do better: problem is LOGNP-complete (Papadimitriou-Yannakakis 96) Often C has a smallimplicit representation : C(i, x) is a polynomial-size circuit such that C(i, x) = 1 iff x belongs to set i • implicit version is 3 -complete (Schaefer 99) (as hard as a b c (a, b, c) for CNF formula )

Approximation Given C (circuit with N inputs), approximate VC(C) • approximation within N 1 - NP-hard (Schaefer 99) • this paper: • 3 -hard to approximate to within 2 - (for any 0) • approximable to within 2 in AM • AM-hard to approximate to within N (for some 0) PSPACE 3 2 AM NP (any 1 if optimal explicit dispersers exist) P

Why Interesting • first constant approximability threshold for optimization problem in the Polynomial Hierarchy • we locate threshold with unusual accuracy: • 3 -hard to within 2 - (N-(1/4 - )) (for any 0) • approximable in AM to within 2 -O(N-1/2) • main idea in 3 -hardness result: desired reduction is essentially a randomness extraction problem • AM-hardness result requires strong amplification of AM using dispersers • constant in disperser seed length matters

Outline for Rest of Talk • Arthur-Merlin protocol for 2 -approximation • 3 -hardness • Schaefer’s 3 -completeness proof • why we need “randomness extraction” • dispersers for simple distributions using listdecodable codes • AM-hardness • conclusions

2 -approximation in AM Sauer-Shelah(-Perles) Lemma: Let C be a collection of subsets of [n] such that |C| m n ( j) j=0 . Then VC(C) m+1. Mutual input: circuit C(i, x) (= 1 iff x in set i) • Merlin sends set of k elements X = {x 0, x 1, . . . , xk-1} • Arthur replies with a random k-bit string s • Merlin sends an index i Accept iff C(i, xj) = sj for j = 0, 1, 2, . . . , k-1 VC(C) k/2 ⇒ VC(C ∩ X) k/2 ⇒ Pr[C accepted] 1/2

Schaefer’s Reduction • (a, b, c) an instance of QSAT 3 with |a| = |b| = |c| = n • Circuit C encodes these sets over universe {0, 1}n x [n]: S( , v, w) = { } x v n if ( , v, w)=1 • if b c (a, b, c) = 1 then C includes sets: {a} x 0. . . 00000 {a} x 0. . . 00001 {a} x 1. . . . 111111 • and so set {a} x 1. . . . 111111 is shattered by C • ⇒ VC(C) n

Schaefer’s Reduction • Circuit C encodes these sets over universe {0, 1}n x [n]: S( , v, w) = { } x v n if ( , v, w)=1 • In general, set of form: {a} x VC(C) ones is shattered if VC(C) n, then {a} x 1. . . . 111111 is shattered, which implies b c (a, b, c) = 1 • For inapproximability, we want relaxed version of this statement to hold for some close to ½. if VC(C) n, then {a} x n ones which implies b c (a, b, c) = 1 ? ? ? is shattered,

A Randomness Extraction Problem • we have a distribution X ⊂ {0, 1}n with n entropy • we need: x∈X n bits seed EXT O(log n) bits uniform m = ( n) (1) bits • this gives us: c) (x ∈ X) c Note: ∧ (a, EXT(x, y), c) ⇔ b c (a, b, y • we only need a disperser • we need zero-error ! (need to hit all of {0, 1}m) BUT, X is in a special class of distributions. . .

Generalized Bit-Fixing Sources From what class of distributions is X ? • recall a set of form {a} x 001001010001 is shattered ↑ ↑↑ ↑ n ones • implies C includes following sets: {a} x ? ? 0? 0? ? ? 0 {a} x ? ? 0? 0? ? ? 1 {a} x ? ? 1 ? 1? ? ? 1 = {a} x X • projecting onto n red positions, we get uniform distr. • “generalized bit-fixing source of dimension n” ( Kahn-Kalai-Linial 88: no det. extraction if (1 - )n Ω(n/log n) )

Dispersers from Codes • we need: x∈X n bits seed EXT t = O(log n) bits uniform m = ( n) (1) bits so that EXT(X, Ut) = {0, 1}m, for all generalized bitfixing sources X of dimension n • binary list-decodable code ECC: {0, 1}m → {0, 1}n • Decode(R, i) gives ith codeword within distance at most (1 - )n from R EXT(x, i) ≝ Decode(x, i) Proof: ∀z∈{0, 1}m ∃x ∈X s. t. dist(ECC(z), x) (1 )n therefore, EXT(x, Ut) hits every z.

Dispersers from Codes x∈X n bits seed EXT t = O(log n) bits uniform m = ( n) (1) bits • using Guruswami-Sudan 00: Theorem : for all 1 ¾, exists an explicit zeroerror disperser EXT: {0, 1}n x {0, 1}2(1 - )log n + O(1) → {0, 1}m for generalized bit-fixing sources of dimension with m = n (1). • (notice n = n/2 + n degree is ≈ n 1/2 instead of ≈ n)

(2 - ) Approximation is 3 -Hard • (a, b, c) an instance of QSAT 3 with |a| = |b| = |c| = m • Circuit C encodes these sets over universe {0, 1}m x [n]: → = { } x S( , v, w) )=1 v if (∧ ( , EXT(v, y), wy) y • if b c (a, b, c) = 1, then {a} x 11111 shattered ⇒ VC(C) n • if VC(C) (1/2 + )n= n, {a} x n ones shattered, C contains sets {a} x x ∈ X , for some generalized bit-fixing source X of dimension n ⇒ (x ∈ X) cy ∧ (a, EXT(x, y), c) ⇔ b c (a, b, c)

N Approximation is AM-hard • language L in AM ⇔ exists poly-time computable RL: x ∈ L ⇒ Pr[ z RL(x, y, z) = 1] = 1 x ∉ L ⇒ Pr[ z RL(x, y, z) = 1] ≤ ½ • strong amplification using dispersers yields: ( 0) x ∉ L ⇒ Pr[ z RL(x, y, z) = 1] ≤ exp(|y| -|y|) • Circuit C encodes sets S(y, z) = y if RL(x, y, z)=1 • if x ∈ L, then 1111 is shattered ⇒ VC(C) = |y| • if x ∉ L, then #sets ≤ exp(|y| ) ⇒ VC(C) ≤ |y| • size of instance (N) depends on degree of disperser • to get gap of N 1 - , need near-linear degree disperser

Conclusions • fairly complete picture of approximability of VC dimension • list-decodable binary codes yield zero-error dispersers for a non-trivial class of distributions • Some improvements and generalizations: • Ta-Shma-Zuckerman-Safra 01 construct nearlinear degree dispersers (available on ECCC) • Using q-ary list-decodable codes, and generalization of Sauer’s Lemma, we obtain approximability threshold of q for a generalization of VC dimension (in final version)