Reasoning With Neural Tensor Networks for Knowledge Base

Reasoning With Neural Tensor Networks for Knowledge Base Completion Author: Richard Socher Presented by Batselem 숭실대학교 2017. 9. 13

Overview Introduction to Knowledge Base Reasoning Knowledge Base Completion Neural Tensor Network Knowledge Base Completion Experiment 2 / 65

Knowledge Base A knowledge base is a special kind of database for knowledge management. • Designed for storing complex information advice User Interface Non-expert user query Knowledge from an expert Inference Engine (추론 시스템) Knowledge base Captures human knowledge and places it into a computer system • Used to solve complex problems that require a high level of human expertise. 3 / 65

Knowledge Base Reasoning Knowledge base consists of a large amount of facts (triples) • Represented as triples of a relation between two entities T = (e 1, R, e 2) Entity 1 Relation Shakespeare author. Of Entity 2 Hamlet Reasoning over a large knowledge base (KB) rdf: type ex: Publication ex: book 3 ex: Article e ng rdf type s: ra su rdf: rd fs: ex: book 2 b. C la ss O f ex: book 1 ex: publishes Knowledge Base Facts: (1) ex: book 1 rdf: type ex: Publication. (2) ex: book 2 rdf: type ex: Article. (3) ex: Article rdfs: sub. Class. Of ex: Publication. (4) ex: publishes rdfs: range ex: Publication. (5) ex: MITPress ex: publishes ex: book 3. (6) … ex: MITPress 4 / 65

Knowledge Base Reasoning Knowledge base consists of a large amount of facts (triples) • Represented as triples of a relation between two entities T = (e 1, R, e 2) Entity 1 Relation Shakespeare author. Of Entity 2 Hamlet Reasoning over a large knowledge base (KB) ss O ex: book 3 Logic (IF-THEN) Rules: IF X is subclass of Y and Z is type of X THEN Z is type of Y ng e la rdf s: ra su e typ : f rd pe : ty f ex: book 2 rd rdf: type ex: Article f ex: Publication rd fs: Infer new relations rdf: type b. C ex: book 1 ex: publishes ex: MITPress 5 / 65

Knowledge Base Completion Knowledge bases are an important resource for question answering and other natural language processing (NLP) tasks • Question Answering (Apple Siri, IBM Watson), • Information Retrieval • Providing structured knowledge to users. Problems with KB: • Suffer from incompleteness • Lack of ability to reason over their discrete entities and relationships This paper adopts the goal of predicting the likely truth of additional facts based on existing facts in the knowledge base predict KB: Existing facts KB: Additional facts 6 / 65

Knowledge Base Completion Introduce a model that can accurately predict additional true facts using only an existing database. • (e 1, R, e 2) = (Ben Affleck, occupation, Film. Maker) is true or false Score Argo Violet Anne director played. In child. Of gender male Ben Affleck lived. In Los Angeles Actor Relation Neural Tensor Network occupation won Oscar Award Film. Maker (Tom cruise, played. In, Mission Impossible) (Tom cruise, occupation, Actor) (Matt Damon, played. In, Martian) (Matt Damon, occupation, Actor) …. 7 / 65

Neural Tensor Network Each relation is described by a neural network and pairs of entities are given as input to the model. Each entity has a vector representation, which can be constructed by its word vectors. The model returns a high score if they are in that relationship and a low one otherwise. • This allows any fact, whether implicitly or explicitly mentioned in the database to be answered with a certainty score. Large text data Word 2 vec 8 / 65

Neural Tensor Network It represents each entity as the average of its word vectors, . • an entity vector as the composition of its word vectors. For example Vector. Bank_of_China= 0. 5*(Vector. Bank+ Vector. China) Vector. Bank Vector. China The goal of their approach is to be able to state whether two entities (e 1, e 2) are in a certain relationship R. • For instance, whether the relationship (e 1, R, e 2) = (Bengal tiger, has part, tail) is true and with what certainty. To this end, they define a set of parameters indexed by R for each relation’s scoring function 9 / 65

Neural Tensor Network The Neural Tensor Network (NTN) replaces a standard linear neural network layer with a bilinear tensor layer • that directly relates the two entity vectors across multiple dimensions. • The model computes a score of how likely it is that two entities are in a certain relationship by the following NTN-based function g(e 1, R, e 2): 10 / 65

Neural Tensor Network R Reference: An MLP-based representation of neural tensor networks for the RDF data models 11 / 65

Neural Tensor Network Training objective: T(i)c = (e(i)1, R(i), ec) is a triplet with a random entity corrupted from a correct triplet T (i) = (e(i)1, R(i), e(i)2) • Score the correct relation triplet higher than its corrupted one up to a margin of 1. • For example, given a correct triplet (Pablo Picaso, nationality, Spain), a potential negative example is (Pablo Picaso, nationality, United States). • For each correct triplet we sample C random corrupted triplets. 12 / 65

Neural Tensor Network The model is trained by taking derivatives with respect to the five groups of parameters. Cost Function Update W using Backpropagation: 13 / 65

Experiment Datasets Relation Triplets Classification • They randomly switch entities from correct testing triplets resulting in an equal number of positive and negative examples. We predict the relation (e 1, R, e 2) holds if g(e 1, R, e 2) ≥ TR 14 / 65

Experiment 15 / 65

Experiment Given place of birth is Florence and profession is historian, relation model can accurately predict that Francesco Guicciardini’s gender is male and his nationality is Italy. These might be inferred from two pieces of common knowledge: (i) Florence is a city of Italy; (ii) Francesco is a common name among males in Italy. 16 / 65

Q&A Thank you 17 / 65