Improving Relation Extraction with Knowledgeattention 2019 11 14

Improving Relation Extraction with Knowledge-attention 2019. 11. 14 Xiaoyuan Liu

Motivation • Traditional feature-based and kernel-based approaches require extensive feature engineering • Deep neural networks normally require sufficient labeled data to train their numerous model parameters • Incorporating prior knowledge from external lexical resources into deep neural network can reduce the reliance on training data and improve relation extraction performance.

Related Work • Self Attention • Position-aware Attention

Self Attention

Self Attention

Self Attention

Self Attention

Self Attention

Self Attention

Self Attention

Self Attention

Self Attention

Self Attention

Self Attention

Self Attention

Self Attention

Position-aware Attention

Knowledge-attention • Knowledge Attention Encoder • Integrate Knowledge-attention with Self-attention

Relation indicators generation • Frame. Net • Thesaurus. com

Knowledge Attention

Knowledge Attention • Small outputs for irrelevant words • Resulted output will be close to the related relation indicators and further apart from other relation indicators in relational semantic space • Effective in capturing the linguistic clues of relations represented by relation indicators in the relational semantic space.

Multi-head knowledge-attention

Position-aware Attention

Position-aware Attention

Position-aware Attention

Position-aware Attention

Integrate Knowledge-attention with Selfattention • Multi-channel Attention • Softmax Interpolation • Knowledge-informed Self-attention

Multi-channel Attention

Softmax Interpolation

Knowledge-informed Self-attention

Knowledge-informed Self-attention

Experiment • Results on TACRED dataset • Results on Sem. Eval 2010 -Task 8 dataset • Ablation study

Results on TACRED dataset

Results on TACRED dataset

Results on Sem. Eval 2010 -Task 8 dataset

Ablation study on TACRED dataset

Error Analysis • For knowledge-attention encoder, 58% errors are false negative (FN) due to the limited ability in capturing long-distance dependencies and some unseen linguistic clues during training • For our integrated model 4 that takes the benefits of both self-attention and knowledgeattention, FN is reduced by 10% • False positive (FP) is not improved due to overfitting that leads to wrong predictions • Many errors are caused by multiple entities with different relations co-occurred in one sentence.

Example