Support Vector Machines Joseph Gonzalez The Big Idea

Support Vector Machines Joseph Gonzalez

The Big Idea

Maximum possible separation between positive and negative training examples

Geometric Intuition

Primal Version min ||w||2 +C ∑ξ s. t. (w. x + b)y ≥ 1 -ξ ξ≥ 0

DUAL Version max ∑α -1/2 ∑αiαjyiyjxixj s. t. ∑αiyi = 0 C ≥ αi ≥ 0 Where did this come from? Remember Lagrange Multipliers Let us “incorporate” constraints into objective Then solve the problem in the “dual” space of lagrange multipliers

Primal vs Dual min ||w||2 +C ∑ξ s. t. (w. x + b)y ≥ 1 -ξ ξ≥ 0 max ∑α -1/2 ∑αiαjyiyjxixj s. t. ∑αiyi = 0 C ≥ αi ≥ 0 Number of parameters? large # features? large # examples? for large # features, DUAL preferred many αi can go to zero!

DUAL: the “Support vector” version max ∑α - 1/2 ∑αiαjyiyjxixj s. t. ∑αiyi = 0 C ≥ αi ≥ 0

“Support Vector”s? max ∑α - α 1α 2(-1)(0+2) - 1/2 α 12(1)(0+1) - 1/2 α 22(1)(4+4) max α 1 + α 2 + 2α 1α 2 - α 12/2 - 4α 22 s. t. α 1 -α 2 = 0 C ≥ αi ≥ 0

“Support Vector”s?

Playing With SVMS