Question Classification with Machine Learning Jason Liang YSP

Question Classification with Machine Learning Jason Liang, YSP Student, Westford Academy Stephanie Martinez, YSP Student, Prospect Hill Academy Charter School Shuyang Cao, Computer Science, Northeastern University Professor Lu Wang, Computer Science Department, Northeastern University

Introduction ● ● Question taxonomies today are often based on interrogative words or are specific to a field ○ Aren’t as useful for open-ended and diverse questions ⇒ creates a need ○ Purpose of project is for machine to identify intent of the user - more useful State-of-the-art classification models require huge computing resources ■ Fasttext was used ○ Can run on computers with limited computing capability ○ Employs multiple techniques and allows for much customization of model parameters

Question Taxonomies ● Include categories of questions ● Often specific to a field - like education ○ Not as helpful for diverse setting Fig. 1: Original taxonomy that we used Fig. 2: Bloom’s Taxonomy

Categorizing Questions Question How does the Turkish populace view EU membership? What does chronic pain do to the nation? What groups have endorsed egg legislation in the past? Who is affected the most? How did the increase in grazing for BLM compare to that of FS? Table 1: Example from the 500 Northeastern Question Set used Category Certainty

Northeastern Questions Compared Pt. 1 Fig. 3, 4 and 5: Data collected from first set of 500 Northeastern Questions using original taxonomy

Updated Taxonomy Table 2: New and updated taxonomy Category Definition Examples 1. Verification Is X true? “Did Congress pass the law? ” 2. Disjunctive Is X or Y the case? “Did Congress pass the law in 2016 or 2017? ” 3 -6. Concept question Who? What? When? Where? What does X mean? “What does the OCE do? ” 5. Extent How much? How many? To what extent? “How much does the appropriation offer for the plan? ” "To what extent is the Renewable Fuel Standard accurate nationwide? " 7. Example What is an example? “What are some examples to support or contradict this? ” 8. Comparison How is X similar to or different from Y? “What is similar about the two proposals? ” 9 -16. Judgmental What can be inferred from the given data? What value does the answerer give to an idea? “What does GAO think of the new measurement? ” 10. Causal antecedent What state causally led to another state? “What has escalated the ongoing conflict in the Southeast? ” “How are younger parents correlated with child poverty? ” 11. Causal consequence What are the consequences of a state? “What are the negative consequences for the services if they do not evaluate their programs? ” “How does the DATA Act affect OIGs? ” 12. Goal orientation What are the goals behind an agent’s action? “What is the purpose of this report? ” 13 -14. Procedural & Enablement What process allows an agent to reach a goal? What resource allows an agent to reach a goal? “What do insurers do now to avoid insuring individuals with higher risk? ” 15. Expectation Why did some out-of-expectation events happen? “Why does persecution against gang violence fail? ”

Change in Taxonomy ● ● Merge 3 (Concept completion) & 6 (Definition questions) into 3 -6 Remove 4 (Feature specifications) Merge 9 (Interpretation) & 16 (Judgmental) Merge 13 (Procedural) & 14 (Enablement) Question Final Decision Jason Steph Shuyang What are the focuses of that program? 3 or 6 12 3 4 What important factor regarding passengers has the DOT not considered according to GAOs findings? 9 or 16 16 9 3 What must employers do besides obtain a certification in order to hire H-2 A workers? 13 or 14 14 3 13 Table 3: Example from the 500 Northeastern Question Set used

Northeastern Questions Compared Pt. 2 Fig. 6, 7 and 8: Data collected from second set of 500 Northeastern Questions using new taxonomy Fig. 9, 10 and 11: Data compared from Northeastern Questions using old and new taxonomy

Yahoo! And Reddit Questions Compared Fig. 12 and 13: Data compared from 600 Yahoo! Questions using new taxonomy Fig. 14 and 15: Data compared from 600 Reddit Questions using new taxonomy

Training the Fasttext Model ● ● ⅘ and ⅕ split of high agreement questions Supervised Learning Preprocessing the Data Bigrams, learning rate, and more Figure 17: Sample Output

Conclusion ● Accuracy of 43. 09% ● Randomly generating the label had an accuracy of 23. 41% ● 19. 68% increase with training Future Works ● More time for a larger data set ⇒ greater accuracy ● Train the model to create templates ● Train the model to generate questions of each type in the taxonomy

Poster

References [Example of Word Vectors]. (n. d. ). https: //devopedia. org/images/article/220/3225. 1569667846. png Olney, A. M. , Graesser, A. C. , & Person, N. K. (2012). Question Generation from Concept Maps. Dialogue and Discourse, 75– 99. https: //pdfs. semanticscholar. org/84 b 5/33 b 04 ca 5958664 fbcc 003 be 43852 c 59 df 1 b 4. pdf Dalton, J. & Smith, D. , (1986). Extending Children’s Special Abilities: Strategies for primary classrooms (pp. 36 -37). http: //www. mandela. ac. za/cyberhunts/bloom. htm Agarwal, M. , Shah, R. , & Mannem, P. (2011). Automatic Question Generation using Discourse Cues. Proceedings of the Sixth Workshop on Innovative Use of NLP for Building Educational Applications, 1– 9. https: //3. basecamp. com/3761917/buckets/17375195/uploads/2735558915 reddit. (n. d. ). Reddit. https: //www. reddit. com/ Homepage. (n. d. ). Yahoo Answers. https: //answers. yahoo. com/? guccounter=1&guce_referrer=a. HR 0 c. HM 6 Ly 93 d 3 cu. Z 29 v. Z 2 xl. Lm. Nvb. S 8&guce_referrer_sig=AQAAANQ 97 Cf. Rc 1 d 0 UFMj. PT 6 o. S VKJTj. Sckr. Cn 191 q. JIw 7 Ct. Tne. OG 4 ut. LP 9 i. MOKPVr. XTl. AWJ 1 d. WOAms. USHy 18 m. Hj 97 ZWXKd. Qnecjpk. VCSBGI 7 jk 1 bm. WZc 3 MAojv 62 Rb. Qx. Uy. Tv. VBh. GVql. UMK 2 JM 8 dcg 19 DUg. QR 3 g YEjk. F 1 g 1 X 72 -Po. SG 9 m Oxford. Sparks. (2017, January 11). What is Machine Learning? [Video]. You. Tube. https: //www. youtube. com/watch? v=f_uw. KZIAe. M 0 [Robot Learning]. (n. d. ). Artificial Intelligence: Preparing Students for the Future with AI. https: //www. gettingsmart. com/wpcontent/uploads/2019/09/4 f 06 b 7 b 2 -753 e-48 b 3 -bb 8 f-777 ca 2653210_30212411048_2 a 1 d 7200 e 2_b. jpg

Acknowledgements Department of Computer Science Professor Lu Wang - Assistant Professor Shuyang Cao - Ph. D Student in Computer Science Department of STEM Claire Duggan - Director of Programs and Operations Natasha Zarour, Salima Amiji, Nicholas Fuchs - YSP Coordinators

Any Questions?