Grounding Language with Points and Paths in Continuous

Grounding Language with Points and Paths in Continuous Spaces Berkeley N L P Jacob Andreas and Dan Klein UC Berkeley

Formal grounding On June 26 th, Facebook stock cost $65 per share quote { date: 2014 -06 -26, stock: FB, price: $65 }

Perceptual grounding On June 26 th, Facebook stock rebounded after a bruising swoon ?

Perceptual grounding On June 26 th, Facebook stock rebounded after a bruising swoon

Perceptual grounding On June 26 th, Facebook stock rebounded after a bruising swoon A after B A, B A before B B, A rebounded { sgn(slope) = +1 } bruising { sgn(slope) = -1, abs(slope) = +2. 3 }

Continuous spaces everywhere On June 26 th, Facebook stock rebounded after a bruising swoon A deep red sunset Keep a little to the left of the post Beat the eggs gently, until they form stiff peaks

Three tasks Color Time series Navigation

Predicting colors H V S dark pastel blueblue

Regression model H V S dark pastel blueblue

Regression model dark pastel blue H 0 0 216 S 0 + -37 + 80 V -40 -25 90 216 = 43 75

Regression model dark pastel blue

Regression model H 216 S 43 V 75 {dark, dark pastel, pastel blue} blue

Experiment setup

Sample predictions electric green pale blue dark brown indigo

Prediction error 0. 4 0. 3 Baseline 0. 2 Last word Full model 0. 1 0 Hue Sat Val Mean

A guessing game pale blue

A guessing game 1 0. 78 0. 6 0. 81 0. 86 0. 50 Baseline Last word Full model 0. 4 Human 0. 2 0 Prediction accuracy

Predicting time series 1 2 stocks rebounded after a bruising swoon 2 1

Predicting time series stocks rebounded after a bruising swoon

Predicting time series sgn(slope): -1 abs(slope): 3. 1 curvature: 0. 5 2 {stocks, stocks rebounded} sgn(slope): 1 abs(slope): 2. 7 curvature: -0. 1 1 {after, after a, a bruising, swoon}

Learning & inference • Need parameters for linear prediction model & log-linear alignment model: easy with EM • For small number of path segments, possible to sum exactly over latent alignments • Otherwise, approximation of your choice

Experiment setup Market rallies to new highs

Sample predictions Reference Predicted [U. S. [aseconomicworriespersist]1 U. S. stocksend endlower] lower 2 as

A guessing game 0. 8 0. 6 0. 72 0. 59 0. 61 0. 5 Baseline No alignment 0. 4 Full model Human 0. 2 0 Prediction accuracy

Peeking at parameters sgn(slope) abs(slope) rise 0. 27 -0. 78 swoon -0. 57 0 sharply -0. 22 0. 28

Following instructions … and then we're going to turn north again and immediat-- well a distance below that turning point there's a fenced meadow but you should be avoiding that by quite a distance okay so we've turned and we're going up north again continue straight up north and then we're going to turn to the west on a curvature right sort of …

Navigation results 0. 6 0. 5 0. 4 Branavan 0. 3 Vogel This work 0. 2 0. 1 0 Precision Recall F-measure

Conclusions • New model for predicting grounded representations of meaning in arbitrary realvalued spaces • Beats strong baselines on a diverse range of tasks • Code and data available online at http: //cs. berkeley. edu/~jda