QA System First Stage Classification Project by Abdullah

Q/A System First Stage: Classification Project by: Abdullah Alotayq, Dong Wang, Ed Pham

Query Processing • Classification Package: Mallet • Classifiers: Maxent, Decision. Tree, C 45, Naive. Bayes, Ada. Boost, Winnow, Balanced Winnow, Bagging Trainer. etc

Main Techniques

Features Semantic Morphological Neighboring (Syntactic)

Stemming • nltk stemmer

N-grams • Bigrams:

• Trigrams: – Poor Classification results • 0. 48 • 0. 478 • Not A good strategy.

NER (Named Entity Recognition) • nltk NER • pre-trained model to do this task. • 6 types of NE

Frequencies Training Data: Type Freq. GSP 22 FACILITY 3 GPE 1203 PERSON 600 LOCATION 21 ORGANIZATION 622

Test Data: Type Freq. GSP 2 FACILITY 0 GPE 90 PERSON 35 LOCATION 3 ORGANIZATION 42

NO Named Entity detected • In training data: 3533, namely 64. 8% • In test data, 353, 70. 6%. -> data sparseness problem

NER Results & Future work • Test data accuracy= 0. 802 • we might try other NE tools, which would give more NE types and cover more percentage on training and test data.

Binary and Real Values • Testing for potential improvement. • Best performing classifiers: For Binary: – Balanced. Winnow: Test data accuracy= 0. 804 – Max. Ent: Test accuracy mean = 0. 78 For Real Values: - Balanced. Winnow: Test data accuracy= 0. 784 - Max. Ent: Test data accuracy= 0. 758

Data set 1: Type Binary Real Values NER Binary NER Real Values Bigrams Binary Bigrams Real Values Trigrams Binary Trigrams Real Values Trainer Balanced. Winnow Decision. Tree Max. Ent Naive. Bayes Max. Ent Naive. Bayes Balanced. Winnow Results 0. 804 0. 68 0. 756 0. 546 0. 784 0. 42 0. 758 0. 54 0. 802 0. 5 0. 772 0. 54 0. 48 0. 768 0. 538 0. 702 0. 624 0. 76 0. 698 0. 624 0. 76 0. 478

Data set 2: Type Binary Real Values Stemmed Binary Stemmed Real Values Trainer Results Balanced. Winnow 0. 74 Max. Ent 0. 74 Naive. Bayes 0. 72 Balanced. Winnow 0. 784 Max. Ent 0. 75 Naive. Bayes 0. 71 Balanced. Winnow 0. 78 Max. Ent 0. 76 Naive. Bayes 0. 76 Balanced. Winnow 0. 75 Max. Ent 0. 77

Proposed future improvement • Word. Net Senses • Class-Specific Related Words

Issues • Performing poorly on some refinements. – Low accuracy scores: • 0. 42 • 0. 54 • Memory consuming classifiers. – Classifiers showed some error messages.

Successes • Made progress in creating the system. • Had some hands-on experience dealing with classifiers, and NLP packages. • Learned ways to improve classification results.

Readings that helped • Employing Two Question Answering Systems in TREC-2005, Sanda Harabagiu & others.

Software packages participated • • • Mallet NLTK Porter-stemmer Self-written code files Stanford Parser, Berkeley Parser