Question Classification in EnglishChinese CrossLanguage Question Answering An

  • Slides: 21
Download presentation
Question Classification in English-Chinese Cross-Language Question Answering: An Integrated Genetic Algorithm and Machine Learning

Question Classification in English-Chinese Cross-Language Question Answering: An Integrated Genetic Algorithm and Machine Learning Approach Min-Yuh Day 1, 2, Chorng-Shyong Ong 2, and Wen-Lian Hsu 1, *, Fellow, IEEE Institute of Information Science, Academia Sinica, Taiwan 2 Department of Information Management , National Taiwan University, Taiwan 1 {myday, hsu}@iis. sinica. edu. tw; ongcs@im. ntu. edu. tw IEEE IRI 2007, Las Vegas, Nevada, USA, Aug 13 -15, 2007. 1/

Outline n n n Introduction Research Background Methods n n Hybrid GA-CRF-SVM Architecture Experimental

Outline n n n Introduction Research Background Methods n n Hybrid GA-CRF-SVM Architecture Experimental Design Experimental Results and Discussion Conclusions Min-Yuh Day (NTU; SINICA) 2

Introduction n Question classification (QC) plays an important role in cross-language question answering (CLQA)

Introduction n Question classification (QC) plays an important role in cross-language question answering (CLQA) n n QC: Accurately classify a question in to a question type and then map it to an expected answer type “What is the biggest city in the United States? ” Question Type: “Q_LOCATION_CITY” Extract and filter answers in order to improve the overall accuracy of a cross-language question answering system Min-Yuh Day (NTU; SINICA) 3

Introduction (cont. ) n Question informers (QI) play a key role in enhancing question

Introduction (cont. ) n Question informers (QI) play a key role in enhancing question classification for factual question answering n n QI: Choosing a minimal, appropriate contiguous span of a question token, or tokens, as the informer span of a question, which is adequate for question classification. “What is the biggest city in the United States? ” Question informer: “city” is the most important clue in the question for question classification. Min-Yuh Day (NTU; SINICA) 4

Introduction (cont. ) n Feature Selection in Machine Learning n n n Optimization problem

Introduction (cont. ) n Feature Selection in Machine Learning n n n Optimization problem that involves choosing an appropriate feature subset. Hybrid approach that integrates Genetic Algorithm (GA) and Conditional Random Fields (CRF) improves the accuracy of question informer prediction in traditional CRF models (Day et al. , 2006) We propose an integrated Genetic Algorithm (GA) and Machine Learning (ML) approach for question classification in cross-language question answering. Min-Yuh Day (NTU; SINICA) 5

Research Background n Cross Language Question Answering n International Question Answering (QA) contests n

Research Background n Cross Language Question Answering n International Question Answering (QA) contests n TREC QA: 1999~ n n QA@CLEF: 2003~ n n European languages in both non-English monolingual and cross-language NTCIR CLQA: 2005~ n n Monolingual QA in English Asian languages in both monolingual and cross-language Question Classification n n Rule-based method Machine Learning based method Min-Yuh Day (NTU; SINICA) 6

Research Background (cont. ) n Two strategies for question classification in English-Chinese cross-language question

Research Background (cont. ) n Two strategies for question classification in English-Chinese cross-language question answering 1) Chinese QC (CQC) for both English and Chinese queries. n English source language has to be translated into the Chinese target language in advance. 2) English QC (EQC) for English queries and Chinese QC (CQC) for Chinese queries. n We focus on question classification in English. Chinese cross-language question answering n Bilingual QA system for English source language queries and Chinese target document collections. Min-Yuh Day (NTU; SINICA) 7

Methods n Hybrid GA-CRF-SVM Architecture n n n GA for CRF Feature Selection GA-CRF

Methods n Hybrid GA-CRF-SVM Architecture n n n GA for CRF Feature Selection GA-CRF Question Informer Prediction SVM-based Question Classification using GACRF Question Informer Min-Yuh Day (NTU; SINICA) 8

Hybrid GA-CRF-SVM Architecture Question GA GA : Feature Selection GA for CRF Feature Selection

Hybrid GA-CRF-SVM Architecture Question GA GA : Feature Selection GA for CRF Feature Selection Near Optimal Feature Subset of CRF GA-CRF Question Informer Prediction Near Optimal CRF Prediction Model CRF-based Question Informer Prediction SVM-based Question Classification Question Informer SVM-based Question Classification Question Type 9

GA-CRF Learning Encoding a Feature Subset of CRF with the structure of chromosomes Initialization

GA-CRF Learning Encoding a Feature Subset of CRF with the structure of chromosomes Initialization Training dataset Population Evaluate (Fitness Function) Stopping criteria Satisfied? x: Feature subset F(x): Fitness Function CRF model 10 -fold Cross Validation Yes No GA Operators: Reproduction, Crossover, Mutation Test dataset Near Optimal Feature Subset of CRF Near Optimal CRF Prediction Model CRF-based Question Informer Prediction GA-CRF Question Informer Prediction 10

Experiment Design n Data set for English Question Classification n Training dataset (5288 E)

Experiment Design n Data set for English Question Classification n Training dataset (5288 E) n n n Test dataset (CLQA 2 T 150 E) n n 4, 204 questions from UIUC QC dataset (E) + 500 questions from the NTCIR-5 CLQA development set (E) + 200 questions from the NTCIR-5 CLQA test set (E) + 384 questions from TREC 2002 questions (E) 150 English questions from NTCIR-6 CLQA’s formal run Data set for Chinese Question Classification n Training dataset (2322 C) n n n 1238 question from IASL (C) + 500 questions from the NTCIR-5 CLQA development set (C) + 200 questions from the NTCIR-5 CLQA test set (C) + 384 questions from TREC 2002 questions (translated) (C) Test dataset (CLQA 2 T 150 C) n Min-Yuh Day (NTU; SINICA) 150 Chinese questions from NTCIR-6 CLQA’s formal run 11

Experiment Design (cont. ) n Features for English Question Classification n Syntactic features n

Experiment Design (cont. ) n Features for English Question Classification n Syntactic features n Word-based bi-grams of the question (WB) n First word of the question (F 1) n First two words of the question (F 2) n Wh-word of the question (WH) n i. e. , 6 W 1 H 1 O: who, what, when, where, which, why, how, and other n Semantic features n Question informers predicted by the GA-CRF model (QIF) n Question informer bi-grams predicted by the GACRF model (QIFB) Min-Yuh Day (NTU; SINICA) 12

Experiment Design (cont. ) n Features for Chinese Question Classification n Syntactic features n

Experiment Design (cont. ) n Features for Chinese Question Classification n Syntactic features n Bag-of-Words n character-based bi-grams (CB) n word-based bi-grams (WB). n Part-of-Speech (POS) n Semantic Features n How. Net Senses n How. Net Main Definition (HNMD) n How. Net Definition (HND). n Tong. Yi. Ci Ci. Lin (TYC) Min-Yuh Day (NTU; SINICA) 13

Experiment Design (cont. ) n Performance Metrics n Accuracy n MRR (mean reciprocal rank)

Experiment Design (cont. ) n Performance Metrics n Accuracy n MRR (mean reciprocal rank) where ranki is the rank of the first corrected question type of the ith question, and M is total number of questions. Min-Yuh Day (NTU; SINICA) 14

Experimental Results n Question informer prediction n Using GA to optimize the selection of

Experimental Results n Question informer prediction n Using GA to optimize the selection of the feature subset in CRF-based question informer prediction improves the F-score from 88. 9% to 93. 87%, and reduces the number of features from 105 to 40. n n n Training dataset (UIUC Q 5500) Test dataset (UIUC Q 500) The accuracy of our proposed GA-CRF model for the UIUC dataset is 95. 58% compared to 87% for the traditional CRF model reported by Krishnan et al. (2005) n Min-Yuh Day (NTU; SINICA) The proposed hybrid GA-CRF model for question informer prediction significantly outperforms the traditional CRF model. 15

Experimental Results n English Question Classification (EQC) using SVM Min-Yuh Day (NTU; SINICA) 16

Experimental Results n English Question Classification (EQC) using SVM Min-Yuh Day (NTU; SINICA) 16

Experimental Results of Chinese Question Classification (CQC) using SVM with different features Top 1

Experimental Results of Chinese Question Classification (CQC) using SVM with different features Top 1 Accuracy (Fine) Top 1 Accuracy (Coarse) Top 5 MRR (Fine) Top 5 MRR (Coarse) POS 53. 33% 65. 33% 0. 5732 0. 7533 POSB 60. 00% 74. 00% 0. 6469 0. 7970 HNMD 71. 33% 81. 33% 0. 7480 0. 8832 CB 74. 00% 84. 67% 0. 7934 0. 9130 HNMDB 74. 00% 86. 00% 0. 7916 0. 9117 C 74. 67% 84. 67% 0. 7979 0. 9152 TYCB 74. 67% 86. 00% 0. 7880 0. 9062 HND 74. 67% 86. 67% 0. 7860 0. 9102 W 76. 00% 88. 00% 0. 7901 0. 9208 HNDB 76. 67% 88. 00% 0. 8000 0. 9162 WB 77. 33% 88. 00% 0. 8067 0. 9162 TYC 77. 33% 88. 67% 0. 8019 0. 9240 Feature Used Min-Yuh Day (NTU; SINICA) 17

Experimental Results (cont. ) n Chinese Question Classification (CQC) using SVM Min-Yuh Day (NTU;

Experimental Results (cont. ) n Chinese Question Classification (CQC) using SVM Min-Yuh Day (NTU; SINICA) 18

Conclusions n n n We have proposed a hybrid genetic algorithm and machine learning

Conclusions n n n We have proposed a hybrid genetic algorithm and machine learning approach for cross-language question classification. The major contribution of this paper is that the proposed approach enhances cross-language question classification by using the GA-CRF question informer feature with Support Vector Machines (SVM). The results of experiments on NTCIR-6 CLQA question sets demonstrate the efficacy of the approach in improving the accuracy of question classification in English-Chinese cross-language question answering. Min-Yuh Day (NTU; SINICA) 19

Applications: ASQA (Academia Sinica Question Answering http: //asqa. iis. sinica. edu. tw System) n

Applications: ASQA (Academia Sinica Question Answering http: //asqa. iis. sinica. edu. tw System) n ASQA (IASL-IIS-SINICA-TAIWAN) n n ASQA is the best performing Chinese question answering system. The first place in the English-Chinese (E-C) subtask of the NTCIR-6 Cross-Lingual Question Answering (CLQA) task. (2007) The first place in the Chinese-Chinese (C-C) subtask of the NTCIR-5 Cross-Lingual Question Answering (CLQA) task. (2005) Min-Yuh Day (NTU; SINICA) 20

Q&A Question Classification in English-Chinese Cross-Language Question Answering: An Integrated Genetic Algorithm and Machine

Q&A Question Classification in English-Chinese Cross-Language Question Answering: An Integrated Genetic Algorithm and Machine Learning Approach Min-Yuh Day 1, 2, Chorng-Shyong Ong 2, and Wen-Lian Hsu 1, *, Fellow, IEEE Institute of Information Science, Academia Sinica, Taiwan 2 Department of Information Management , National Taiwan University, Taiwan 1 {myday, hsu}@iis. sinica. edu. tw; ongcs@im. ntu. edu. tw Min-Yuh Day (NTU; SINICA) IEEE IRI 2007, Las Vegas, Nevada, USA, Aug 13 -15, 2007. 21

Interactive CrossLanguage Question Answering Searching passages vs searchingInteractive CrossLanguage Question Answering Searching passages vs searching
4 Question Answering QA Question Answering Query NLP4 Question Answering QA Question Answering Query NLP
Question Answering over Linked Data Outline Question AnsweringQuestion Answering over Linked Data Outline Question Answering
Question Answering Question Answering Goal Automatically answer questionsQuestion Answering Question Answering Goal Automatically answer questions
Memory Networks for Question Answering Question Answering ReturnMemory Networks for Question Answering Question Answering Return
Question Answering What is Question Answering Dan JurafskyQuestion Answering What is Question Answering Dan Jurafsky
Sistemi di Question Answering Sistemi di Question AnsweringSistemi di Question Answering Sistemi di Question Answering
Media Regulation Answering Questions Answering An InfluenceRegulation QuestionMedia Regulation Answering Questions Answering An InfluenceRegulation Question
Classification Classification Classification Classification Features and labels GreenClassification Classification Classification Classification Features and labels Green
Classification 5 3 Classification of Biodiversity Classification ClassificationClassification 5 3 Classification of Biodiversity Classification Classification
Classification of Living Things CLASSIFICATION CLASSES Classification CLASSIFICATIONClassification of Living Things CLASSIFICATION CLASSES Classification CLASSIFICATION
Essentials of Oral Defense EnglishChinese Translation This lectureEssentials of Oral Defense EnglishChinese Translation This lecture
LCSbased EnglishChinese Translingual Retrieval Bonnie Dorr GinaAnne LevowLCSbased EnglishChinese Translingual Retrieval Bonnie Dorr GinaAnne Levow
Conversion in EnglishChinese Translation 1 The students workConversion in EnglishChinese Translation 1 The students work
EnglishChinese Handbook for Martial Art Coaches 2019 914EnglishChinese Handbook for Martial Art Coaches 2019 914
EnglishChinese Handbook for Martial Art Coaches 2019 920EnglishChinese Handbook for Martial Art Coaches 2019 920
EnglishChinese Handbook for Martial Art Coaches 2019 914EnglishChinese Handbook for Martial Art Coaches 2019 914
Final exam orientation EnglishChinese Translation 2 Prepared byFinal exam orientation EnglishChinese Translation 2 Prepared by
Improving interaction with the user in Crosslanguage QuestionImproving interaction with the user in Crosslanguage Question
ANSWERING POLICY QUESTIONS Welcome to Answering Policy QuestionsANSWERING POLICY QUESTIONS Welcome to Answering Policy Questions