CRCV REU 2019 Week 8 Aaron Honculada Current

Current Approach • Parsing Question into Subqueries • How to represent video to utilize

Literature Review • Neural-Symbolic Concept Learner • Visual Perception Module • Mask R-CNN The

Neural-Symbolic Concept Learner • Semantic Parsing • Question to program • Hierarchy of operations

Neural-Symbolic Concept Learner • Domain Specific Language • Requires unique referral object The Neuro-Symbolic

Dependency Parsing • Sentence to Dependency Tree • Dependency from Grammatical Relation • Possible

Dependency Parsing Example • What could happen to the newborn , Emily , after

Dependency Parsing • Locate keywords in questions • With keywords formulate subqueries • Biaffine

Video Representation • Cannot fully answer from visual concepts alone • Contextualize objects in

Slides: 9

Download presentation

CRCV REU 2019 Week 8 Aaron Honculada

Current Approach • Parsing Question into Subqueries • How to represent video to utilize the information

Literature Review • Neural-Symbolic Concept Learner • Visual Perception Module • Mask R-CNN The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision (Tenenbaum, ICLR 2019)

Neural-Symbolic Concept Learner • Semantic Parsing • Question to program • Hierarchy of operations • concepts as parameters The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision (Tenenbaum, ICLR 2019)

Neural-Symbolic Concept Learner • Domain Specific Language • Requires unique referral object The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision (Tenenbaum, ICLR 2019)

Dependency Parsing • Sentence to Dependency Tree • Dependency from Grammatical Relation • Possible Multi-task model • VQA • Generate Dependency Tree

Dependency Parsing Example • What could happen to the newborn , Emily , after Izzie administers the ECMO ?

Dependency Parsing • Locate keywords in questions • With keywords formulate subqueries • Biaffine Classifier and Bi. LSTM Deep Biaffine Attention for Neural Dependency Parsing (Dozat, ICLR 2017)

Video Representation • Cannot fully answer from visual concepts alone • Contextualize objects in scene • Color • Belongs to person or scene • Places 365 • Scene Attributes • Indoor/ Outdoor