Storied Navigation Edward Shen Hector Beltran Donyun Kim

Storied Navigation Edward Shen Hector Beltran Donyun Kim 12/15/06

Story Analogy character A character B character type student teacher situation not knowing where to go for the future willing to give guidance to character A emotion / attitude anxious, lost, hoping for help, pessimistic caring, insightful, appreciatory, advisory character A character B character type Student teacher situation Thought that she was clear where to go for the future willing to give suggestion to character A emotion / attitude Confused, hesitated encouraging, insightful, advisory

Idea n Build system that allows users to tell different stories from sequences of video clips with meaningful relationships between clips (using analogies to other stories)

Original System Design What’s the story about? Presidential election 1. Who were the candidates? 2. When was the election? 3. What was the outcome of the election? 4. What were the issues? 1. How do you feel about him? 2. What has he done in the past? Video Corpus

Challenges for Q/A scenario How to derive context-relative questions based on provided information n How to derive plausible annotations for a video sequence based on the context of using it n How to design representation for video sequence such that the above can be conducted more easily n

Generating Questions Assume video sequences annotated with Roger Schank’s form of indexing stories (theme, goal, plan, result, lesson) n Questions come from: n ¨ Generic question templates ¨ Common sense reasoning ¨ Story analogies

Curiosity Ask questions people tend to be more curious about n Introduce new pieces of information that produce effect of new direction of navigation n

Curiosity Relationship between people n Why do people do what they do? n ¨ External forces ¨ Values/Beliefs ¨ Internal desire

New System Design n Different users ¨ Users that know video corpus very well ¨ Users that are seeing videos for first time n Help new users create stories by using information first set of users have already entered (system developers)

New system design System Developer 1 Q 2 Q 3 2 Q 4 Q 5 3 Q 7 Q 8 Q 9 Q 5 Q 6 4 4 Using semantic similarity and common sense End User 2 5 Why is Mary sad? View suggested questions

System Developer 1. System Developer types in story Gustave is one of the dancers. He likes to suggest ideas for the dance. He gets very frustrated that he can only do what he is told. 3. Question set developed for each sequence 1 Q 2 Q 3 2. Sentence/Sentences attached to clips 1 2 3 4. Generate Q/A sets between video sequences 1 2 Q 1 Q 2 Q 3 Q 4 3 Q 1 Q 2 Q 3 4 Q 1 Q 2 2

Steps 1 – Segment Stories Gustave is one of the dancers. He likes to suggest ideas for the dance. 3/5 He gets very frustrated that he is only allowed to do what he is told. 4/5 At one point he was crying. This happened during the first week. 4/5 The second week was different. 3/5 Dancers were encouraged to design their own dance. 4/5 Gustave had a wonderful time. 1. Always include topic sentence 2. Try topic sentence with new sentence, if sentence does not return 4/5 same clips as previous sentence/segment, start new segment. Otherwise, add to previous sentence/segment

Step 2 – Assign clips 1 Gustave is one of the dancers. He likes to suggest ideas for the dance. 2 Gustave is one of the dancers. He gets very frustrated that he is only allowed to do what he is told. At one point he was crying. This happened during the first week. The second week was different. 3 Gustave is one of the dancers. Dancers were encouraged to design their own dance. Gustave had a wonderful time.

Step 3 – Question Generation Gustave is one of the dancers. 1 to suggest ideas for the dance. He likes How does Gustave dance? What are Gustave’s ideas? Common Sense 2 1. General templates What, when, how, why 2. Emotions – why…? Ever feel the opposite? 3. Conflict – ask for resolution Gustave is one of the dancers. He gets very frustrated that he is only allowed to do what he is told. At one point he was crying. This happened during the first week. The second week was different. Why was Gustave frustrated? Is Gustave ever happy? 3 the conflict resolved? How was Gustave is one of the dancers. Dancers were encouraged to design their own dance. Gustave had a wonderful time. Why did Gustave have a wonderful time?