More precise fuzziness more fuzzy precision Katrin Erk

More precise fuzziness, more fuzzy precision Katrin Erk University of Texas at Austin COMPOSES workshop August 14, 2016 Joint work with: Ray Mooney, Islam Beltagy, Gemma Boleda, Pengxiang Cheng, Chau Cuong, Dan Garrette, Gerhard Kremer, Sebastian Padó, Stephen Roller, Stefan Thater

Where do we need both fuzziness and precision? • In-depth text understanding: • Precision for understanding structure • Fuzziness for the lexical/phrasal level • Inference needs to integrate both • Same question in two areas: • Computational: single-document understanding • Linguistic: How can people learn word meanings and reason over what they have learned?

Single-document understanding • In-depth analysis of a single document: • Reading comprehension/single document QA: CNN/Daily Mail (Hermann et al 2015) • Evidence-based medicine: reading individual journal papers • (Textual entailment) • Redundancy is our friend – but what if we don’t have it? • Have to deal with many language phenomena • Have to deal with rare events

Alignment and inference: RTE, STS, QA (Beltagy 2016) Formula 1: Text Align predicates and discourse entities, formulate alignment as weighted rules Formula 2: Query Probabilistic inference over weighted formulas

“Montague meets Markov”: Deep sentence structure, flexible word meaning (Beltagy et al 2013; 2016) • Sentence representation: First-order logic • A man walks into a bar • Distributional ratings of lexical entailment translated into weighted inference rule: • Probabilistic inference: • Markov Logic Networks (MLNs)

Probabilistic semantics • Uncertainty about the world we are in • Probability distribution over worlds • Propositional logic: worlds = truth assignments • Nilsson 1986 • Probability that a sentence φis true depends on the worlds w in which it is true: • Markov Logic Networks build on this idea • Particular way of determining probability of worlds: from weighted formulas

Undirected graphical models: interaction between random variables 0. 3 0. 4 Assignment x 0. 9 0. 4 0. 1 0. 8 • • 0. 9 0. 2 Nodes are random variables Probability of assignment x of values to all random variables Depends on values for all cliques in the graph Phi: potential functions

Markov Logic Networks: computing with weighted clauses All cats chase all dogs: chase(a, b)F cat(a) T Assume 2 constants: a, b dog(b) F chase(b, b)T chase(a, a) T dog(a) F cat(b) F T chase(b, a) Node = random variable = ground literal. Clique = ground literals that form a ground clause Assignment x: truth values = possible world Value x. C on a clique: depends on whether the ground clause is true in x.

Alignment rules are paraphasing rules • Many existing paraphrase collections, but all too small (Lin and Pantel 2001, Szpektor et al 2008, Berant et al 2010, 2011, Ganitkevitch et al 2013) • Our system: paraphrase rules on the fly • Use alignment to find rules: Phrases that, if paraphrases, would make the Query entailed • When gold labels at passage level are available (entails/does not): Train classifier on lexical/phrasal entailment • Features: • • Distributional information Word. Net Average, max over word features for phrasal entailment Compositional distributional phrase similarity (Paperno et al 2014) did not help

More precise fuzzy representations for phrasal entailment? • Can compositional distributional models be trained to produce good phrasal entailment? (Kruszewski et al 2015, Henderson et al ACL 2016) • End-to-end deep learning systems seem to be good at paraphrases, as qualitative analyses of attention show: • Cheng/Dong/Lapata 2016: textual entailment using SNLI • Rocktäschel et al 2015: textual entailment using SNLI • Hermann et al 2015: text comprehension (single-document QA) • Any way to transfer that?

Inference: What is the right mix of fuzziness and precision? • For in-depth text understanding, we will need to handle negation and other scope phenomena in a principled fashion • We have used Markov Logic Networks (MLNs), Probabilistic Soft Logic (PSL), custom graphical models • The Achilles heel of MLNs: network size • The problem: Generating all groundings of a given set of formulas • Not just MLNs, PSL too • Our solution: • Pre-pruning of the network based on information in the Text • Only adding inference rules likely to be useful (based on alignment) • No rule chaining

Other probabilistic inference options • Encoding logical rules into the architecture of a neural network (Towell et al, 1990, Eliassi-Rad 2001): also restricted to propositional rules • Training a distributed representation to mostly obey particular logical rules (Rocktäschel et al 2015, Hu et al. 2016): so far, relatively simple rules • Generative models via probabilistic programming (Goodman et al. ): need to restrict model size or search space • Needed: ideas for scalable weighted reasoning

Can humans learn word meaning from textual context? How to learn a word: • Getting an explanation (for example in school, or from an encyclopedia): A samovar […] is a heated metal container traditionally used to heat and boil water in and around Russia […] (Wikipedia, July 29, 2016) • Perceiving a category member: “Look, a samovar!” • Learning from text Model of word meaning should allow for all these modes of word learning Image: Benito bonito [CC BY-SA 3. 0]

Three modes of word learning: How to model? • Learning from definitions • Modeling: Update to information state (Veltman 1996): set of worlds that, as far as the agent knows, could be the actual one • Learning from distributional evidence (Erk 2016) • Update probabilistic information state (van Benthem et al. 2009, Zeevat 2013): probability distribution over worlds in information state • Nilsson 1986, probabilistic logic (also in MLNs):

Three modes of word learning: How to model? • Learning from grounded situations: • Generative model for probabilistic information state (Goodman and Lassiter 2013) • Word meaning representation in a conceptual space -- think probabilistic Gärdenfors • For experimental evaluation: feature norms (Mc. Rae et al, Vinson and Vigliocco)

Three modes of word learning: How to model? • Upon observing a word: Probabilistically generate an entity using • its concept representation • the textual context • Fixed features: those that are observed • Update the concept by the entity • Learning = disambiguation

Again: how to do inference? • Humans can do one-shot learning We found a cute, hairy wampimuk sleeping behind the tree. Doesn’t always work this neatly: A study showed that a tiny 50 gramme (1. 76 oz) wampimuk heated up 1 degree every minute and a half from the sun. • How do humans do that? • “Overhypotheses” (Goodman 1955, Kemp et al 2007) • e. g. : Names of solid objects generalize by shape, names of nonsolids by material (Colunga & Smith 2005)

Again: what are the right fuzzy representations for context? • Inferring the meaning of a word in context • We had humans do this task as lexical substitution (Kremer et al 2014) • My fear is that she would live, and I would learn that I had lost her long before Emil Malaquez translated her into a thing that can be kept, admired, and loved. • Substitutes include: retain, store, own, possess, enshrine, stage • The last four: Not Word. Net related to the target!

What is this wider context? • Some sort of abstract situation type “Pulp romance: woman as precious object”? • U-semantics frames? Fillmore 1985 • Events that influence human sentence understanding? Mc. Rae & Matsuki 2009 • Some kind of narrative schema (which we could model)? Chambers & Jurafsky 2009, Pichotta & Mooney 2016 • How many such abstract situations are there? Too many to cluster

Summary of questions • More precise fuzzy representations • Better phrase representations for phrasal entailment • Representing abstract situations? • Right mix of fuzziness and precision for inference • Scaling up while retaining expressivity? • Best way of integrating alignment and inference? • Learning the right overhypotheses for word meaning?