Separating Style and Content with Bilinear Models Joshua

Separating Style and Content with Bilinear Models Joshua B. Tenenbaum, William T. Freeman Computer Examples Barun Singh 25 Feb, 2002

PHILOSOPHY & REPRESENTATION Data contains two components: style and content Want to represent them separately Symmetric Bilinear Model: y : observed data a : style vector b : content vector I, j : components of style and content W : matrix of basis vectors (e. g. , “eigenfaces”) Asymmetric Bilinear Model: A : matrix of style-specific basis vectors More flexible model Easier to deal with Y : (SK) x C A : (SK) x J b: Jx. C

PROBLEMS TO BE SOLVED Given a labeled training set of observations in multiple styles and content classes, t extrapolate a new style to unobserved content classes Ø Fit asymmetric model (find A and b for known styles and contents) using SVD Ø Find style matrix that best explains data for incomplete style (i. e. , minimizes E given below) Ø Extrapolate using the estimated style matrix OLC used to solve overfitting problem Ø Parameters involved: l Ø l = 0 : Purely asymmetric model Ø l = : Purely symmetric model Ø

PROBLEMS TO BE SOLVED Given a labeled training set of observations in multiple styles and content classes, t classify content observed in a new style Ø Fit asymmetric model Ø Use separable mixture model (SMM) with EM algorithm to determine style matrix for new style Ø Ø Parameters: model dimensionality J, model variance s 2, max number of EM iterations tmax Select content class c that maximizes Pr(s’, c|y)

PROBLEMS TO BE SOLVED Given a labeled training set of observations in multiple styles and content classes, t translate from new content observed only in new styles into known styles or content classes Ø Fit symmetric model (find W, a, and b for known styles and contents) using iterated SVD procedure Ø Given a single image in a new style and content type, iterate to find the style and content vectors for the new image (given an initial guess for the new content vector):

TOY EXAMPLE - intro Image made of 4 pixels, each of which are either white or red. Style represents if the top or bottom rows are red or white Content represents if the left or right columns are red or white. SYMMETRIC MODEL Content Vectors ( b ) Output Images ( y ) Style Vectors ( a ) Basis Images ( W )

TOY EXAMPLE - intro Content Vectors ( b ) *Note: Images drawn as blocks, but represented as vectors, not matrices Output Images ( y ) Style-specific Basis Images ( A ) ASYMMETRIC MODEL

TOY EXAMPLE - extrapolation ? Extrapolate Style-specific Basis Images ( A ) Fitting the asymmetric model Content Vectors ( b )

FONTS EXAMPLE - extrapolation Training Set Content (Letter) Style (Font) Incomplete Style

FONTS EXAMPLE - extrapolation Symmetric Model Asymmetric Model Sym. W/ Asym. Prior (dim = 60) vs. Actual 10 20 30 40 model dimension 50 60

TOY EXAMPLE - classification 1: Fit asymmetric model to training set Basis Images ( A ) Style-specific Content Vectors ( b )

TOY EXAMPLE - classification 2: Use Separable Mixture Model w/ EM to classify Content Vectors ( b ) Basis Images ( A ) Style-specific s 2 = 0. 5 s 2 = 0. 6 s 2 = 0. 35 Resulting Images Actual

FACES EXAMPLE - translation Content : faces Style: ligting

finito