COMET Commonsense Transformers for Automatic Knowledge Graph Construction

COMET : Commonsense Transformers for Automatic Knowledge Graph Construction Antoine Bosselut☆★ Hannah Rashkin☆★ Maarten Sap☆★ Chaitanya Malaviya ★ Asli Celikyilmaz＊ Yejin Choi ☆★ ★Allen Institute for Artificial Intelligence, Seattle, WA, USA ☆ Paul G. Allen School of Computer Science & Engineering, Seattle, WA, USA ＊Microsoft Research, Redmond, WA, USA ACL 2019 자연어처리 연구실 한장훈 Natural Language Processing Lab 1

서론 COMET 목차 ATOMIC CONCEPTNET 결론 Natural Language Processing Lab 2

COMET (Commonsense Transformers for Automatic Knowledge Graph Construction) Main Contribution • KB 구축에 대한 generative 한 approach를 제안 • Commonsense knowledge tuple을 produce하는 트랜스포머(언어모델) 기반의 framework를 제안. • ATOMIC과 Concept. Net 두 가지 도메인에 대한 분석. Natural Language Processing Lab 5

COMET Task • Knowledge base 정보를 학습하여 새로운 knowledge를 생성 • 기존 training knowledge base 에 관한 정보가 language tuple로 주어짐 {s, r, o} • s: subject r: relation o: object • s=“take a nap", r=Causes, o=“have energy“ (ex : Concept. Net) • 주어진 s와 r을 통해 o를 generate 함 • Ex) generated o =“Feel good” Natural Language Processing Lab 6

COMET Model • 트랜스포머 기반 언어모델 사용 => GPT Natural Language Processing Lab 7

COMET Transformer block • GPT는 12개의 디코더 트랜스포머 블록으로 구성 • FFN 은 2 layer feedfoward network • Multiattn은 12개의 attention head 사용 Natural Language Processing Lab 8

COMET Natural Language Processing Lab 9

COMET Dataset • Knowledge base를 사용 • ATOMIC , Concept. Net • domain-agnostic 하기 때문에 다른 commonsense knowledge도 사용가능 Natural Language Processing Lab 10

ATOMIC (An Atlas of Machine Commonsense for If. Then Reasoning) 특징 • Taxonomic , encyclopedic knowledge를 가진 기존 knowledge base와는 다르게 추론적인 if-then Knowledge 에 초점을 둠. • scale, coverage, and quality를 중점으로 지식베이스를 구축. • 기존 corpora에서 comonsense를 추출하는게 아닌 crowdsourcing experiment에 초점을 둠. Natural Language Processing Lab 11

ATOMIC Relation type 분류 1. 예측되는 내용에 따른 분류 • If-Event-Then-Mental-State • If-Event-Then-Event • If-Event-Then-Persona 2. 인과 관계(causal reation)에 따른 분류 • (1) causes, (2) effects, (3) stative. Natural Language Processing Lab 12

ATOMIC 1. If-Event-Then-Mental-State • 사건에 대한 생각의 전, 후 관계를 의미한다. • Event : X compliments Y • Relation_Intents: X wants to be nice 2. If-Event-Then-Event • 사건에 대한 전, 후 사건을 의미한다. • Event : X makes Y’s coffee • Relation_x. Need: X needs to put coffee in the filter 3. If-Event-Then-Persona • 사건의 주체가 어떻게 인식되는지를 의미한다. • Event : X calls the police • Relation_x. Attr: X is seen as “lawful” Natural Language Processing Lab 13

ATOMIC Natural Language Processing Lab 14

ATOMIC Data 1. Base event 로 24 K의 event phrase를 다양한 말뭉치로부터 추출(stories, books, Google Ngrams, and Wiktionary idioms) 2. Crowdsourcing framework -> Free-form text annotation setup 통해 commonsense knowledge를 모 음 3. 총 877 K 의 Tuple Base Model • 인코더 디코더 모델 (BI-GRU , unidirectional GRU) • event와 relation 이 주어졌을때 target generation. • Cross entorpy를 통한 학습. • Single learning: 9 ENC 9 DEC • Mutitask Learning 1. EVENT 2(IN)VOLUNTARY : one encoder 4 voluntary decoder, another encoder five involuntary decoder. 2. EVENT 2 PERSONX/Y : agent, theme 3. EVENT 2 PRE/POST : couse, effect Natural Language Processing Lab 15

COMET-ATOMIC 실험결과 Automatic evaluation • • PPL = perplexity N/T sro= all generate tuple that are novel N/T o= have novel object N/U o=novel object of unique object Natural Language Processing Lab 16

COMET-ATOMIC 실험결과 Human evaluation Precision at 10 % Natural Language Processing Lab 17

Concept. Net 5. 5: An Open Multilingual Graph of General Knowledge • Word 와 phrase를 labeled weighted edge로 연결한 knowledge graph • subject, relation label , object 로 이루어짐. • Ex) a dog has a tail -> (dog, Has. A, tail) • Word 2 vec, Glove 와 유사하지만 성능 좋은 word embedding 형성 가 능 • freely-available semantic network • Multilinugual • 21 million edges, over 8 million nodes • 83 languages in which it contains at least 10, 000 nodes • 34개의 relation을 가짐. Natural Language Processing Lab 19

Concept. Net 5. 5 Knowledge source • Facts acquired from Open Mind Common Sense corpus(OMCS) • Information extracted from parsing Wiktionary (18. 1 million edges) • Games with a purpose designed to collect common Knowledge • Open Multilingual Word. Net • Japanese-multilingual dictionary • Open. Cyc • A subset of DBPedia Natural Language Processing Lab 20

Concept. Net 5. 5 Relation example • /r/Related. To : 관계는 있는데 어떻게 관계 있는지 모를경우 Ex) learn ↔ erudition • /r/Is. A : A 가 B의 subtype 이거나 특정 인스턴스일 경우 Ex) car → vehicle; Chicago → city • /r/part of : A가 B의 part일 경우 Ex) gearshift → car 출처 https: //github. com/commonsense/conceptnet 5/wiki/Relations Natural Language Processing Lab 21

COMET – Concept. Net 실험결과 Data • Test : 1200 tuple • Training set : 100 k tuple Baseline Model • Bi. LSTM model (by Saito et al. ) sr→o , or → s • LSTM – s (sr→o) Detail • Score : correct by the pre-trained Bilinear AVG model (by Li et al. 2016) • COMET-RELTOK : Is. A ≠ is a • Decoding 방법은 greedy Natural Language Processing Lab 22

Comet – Concept. Net 실험결과 • Pretrain한 모델이 정보를 가짐. • 망고의 경우 train에서 (mango, Used. For, salsa) 가 주어졌을때 • Comet – pretrain (mango, isa, spice) 라고 나온 반면 • Comet (mango, isa, fruit) 로 생성됨. Natural Language Processing Lab 23

참고문헌 • T 2 KG: An End-to-End System for Creating Knowledge Graph from Unstructured Text • Commonsense knowledge base completion. • Wordnet: A lexical database for English • Query Expansion with Concept. Net and Word. Net: An Intrinsic Comparison • Commonsense knowledge base completion and generation. • https: //mosaickg. apps. allenai. org/ • http: //conceptnet. io/ Natural Language Processing Lab 25

부록 (왼쪽: Atomic , 오른쪽 : Concept. Net generated object) Natural Language Processing Lab 26