VQA Visual Question Answering Presented by Vivek Pradhan

VQA: Visual Question Answering Presented by: Vivek Pradhan

Problem Definition • Given an image we want to answer an open ended question about the image.

Best Performing Model ØQuestion and image encodings are fed to a LSTM to generate the answer. ØScene level and object level information is available since it works well on these questions.

Experiments 1. Why don’t humans have perfect accuracy? 2. What does the model really learn?

Human Performance Ø Humans do not do a perfect job. Ø Captions give a lot of information. The model does not outperform humans with captions. Ø It will be interesting to see if the set of answers that humans get right with captions significantly overlap with the model’s correct answers.

Annotator Accuracy

Naïve Baselines - Performance Method Accuracy Most Popular answer by question type 36. 18% Answer chosen by answer to NN 40. 61% Ø Clearly it is possible to learn some shortcuts to solve this task. Ø Captions Ø Co-Occurrence Ø Likely Answer by Scene Category Ø Likely Answer by Object Detection

Possible Extensions • Will the addition of object labels and scene categories help? • Can we use a object detection model and scene classification model to provide better signals than VGG?

What does the model learn?

Model Learns Scene Label • In a baseball game. A ball is most likely object to be hit. • In a beach. Likely to be surfing.

What does the model learn?

Model Learns Co-Occurrence of objects ØWhat objects are likely to be on top of a car.

What does the model learn?

Model learns actions associated with objects ØA dog is likely to be eating. ØA pizza is likely to be eaten by a man.

Conclusion ØThe model is learning many shortcut ways of answering questions. ØIt might be benefited by getting a list of objects and the scene category of the image, this can be tested using ground truth annotations.