Scene Maker Multimodal Visualisation of Natural Language Film

Scene. Maker: Multimodal Visualisation of Natural Language Film Scripts Eva Hanser Prof. Paul Mc Kevitt Dr. Tom Lunney Dr. Joan Condell Dr. Minhua Eunice Ma School of Computing & Intelligent Systems Faculty of Computing & Engineering University of Ulster, Magee, Northern Ireland School of Computing and Mathematics Faculty of Business, Computing and Law University of Derby, England hanser-e@email. ulster. ac. uk, {p. mckevitt, tf. lunney, j. condell}@ulster. ac. uk m. ma@derby. ac. uk

PRESENTATION OUTLINE Aims & Objectives Related Projects Scene. Maker Design and Implementation Relation to Other Work Conclusion and Future Work

: AIMS & OBJECTIVES AIMS Input: Screenplay Scene. Maker System Output: Animation • Automatically generate well-designed and affective virtual scenes from screenplays • Realistic visualisation of emotional aspects • Multimodal representation with 3 D animation, speech, audio and cinematography • Enhance believability of virtual actors and scene presentation

: AIMS & OBJECTIVES • Processing/inferencing emotions and semantic information within story context • Common sense, affective and cinematic knowledge ontologies reflecting human cognitive reasoning rules • Automatic genre recognition from text • Design, implementation and evaluation of Scene. Maker

: RELATED PROJECTS SEMANTIC TEXT PROCESSING INT. M. I. T. HALLWAY -- NIGHT Lambeau and Tom come around a corner. His P. O. V. reveals a figure in silhouette blazing through the proof on the chalkboard. There is a mop and a bucket beside him. As Lambeau draws closer, reveal that the figure is Will, in his janitor's uniform. There is a look of intense concentration in his eyes. LAMBEAU Excuse me! WILL Oh, I'm sorry. LAMBEAU What're you doing? WILL (walking away) I'm sorry. Screenplay Extract from ‘Good Will Hunting (1997)’ • Standardized format and language of screenplays • Automatic annotation of formal screenplay elements (Jhala 2008) • Semantic information on location, timing, props, actors, events, manners, dialogue and camera direction

: RELATED PROJECTS VISUAL AND EMOTIONAL SCRIPTING • Emotion recognition from text: keyword spotting, lexical affinity, statistical models, fuzzy logic rules, machine learning, commonsense knowledge, cognitive models • XML-based annotations defining visual appearance of animated characters and scenes: BEAT – Behaviour Expression Animation Toolkit (Cassell et al. 2001) MSML – Movie Script Markup Language (Van Rijsselbergen et al. 2009) <GAZE word=1 time=0. 0 spec=AWAY_FROM_HEARER> <GAZE word=3 time=0. 517 spec=TOWARDS_HEARER> <R_GESTURE_START word=3 time=0. 517 spec=BEAT> <EYEBROWS_START word=3 time=0. 517>

: RELATED PROJECTS MODELLING AFFECTIVE BEHAVIOUR • Automatic physical transformation and synchronisation of 3 D models reflecting emotion • Manner influences intensity, scale, force, fluency and timing of an action • Multimodal annotated affective video or motion captured data (Gunes and Piccardi 2006) Greta (Pelachaud 2005) Personality & Emotion Engine (Su et al. 2007)

: RELATED PROJECTS VISUALISING 3 D SCENES • Words. Eye – Scene composition (Coyne and Sproat 2001) • Script. Viz – Screenplay visualisation (Liu and Leung 2006) • CONFUCIUS – Action, speech & scene animation (Ma 2006) • CAMEO – Cinematic and genre visualisation (Shim and Kang 2008) Words. Eye Script. Viz CONFUCIUS CAMEO

: RELATED PROJECTS AUDIO GENERATION • Emotional speech synthesis (Schröder 2001) - Prosody rules • Music recommendation systems - Categorisation of rhythm, chords, tempo, melody, loudness and tonality - Sad or happy music and genre membership (Cano et al. 2005) - Associations between emotions and music (Kuo et al. 2005)

: DESIGN AND IMPLEMENTATION KEYOBJECTIVES • Context consideration through natural language processing, common sense knowledge and reasoning methods • Extract genre and moods from screenplays • Influence on all elements of visualisation • Enhance naturalism and believability • Text-to-animation software prototype, Scene. Maker

: DESIGN AND IMPLEMENTATION ARCHITECTURE OF SCENEMAKER Screenplay Script Editor Animation Player Text & Language Processing Context Interpretation Genre } Emotion Action } Multimedia Generation

: DESIGN AND IMPLEMENTATION SOFTWARE AND TOOLS Natural Language Processing & Script Segmentation Movie Script Gate (1) ANNIE Onto-Gazetteer Genre Movie Ontology RDFS/OWL Script Format Ontology LVSR(2) Lexical Visual Semantic Representation Context + Emotion Reasoning Event Synchronisation 3 D Rendering + Multimedia Concept Net(3) MSML(5) Unity(6) /SMIL 3 D Engine (Java. Script, XML) Common Sense Knowledge Word. Net. Affect(4) Automatic Sound & Music Selection 3 D Models (3 D Studio Max) Festival(7) Speech Synthesiser (1) http: //gate. ac. uk (2) Ma 2006 (3) Liu and Singh 2004 (4) Strapparava and Valitutti 2004 (5) Van Rijsselbergen et al. 2009 (6) http: //unity 3 d. com (7) http: //www. cstr. ed. ac. uk/projects/festival

: DESIGN AND IMPLEMENTATION EVALUATION OF SCENEMAKER Evaluating 4 aspects of Scene. Maker: Aspect Evaluation Correctness of screenplay analysis & visual interpretation Hand-animating scenes Effectiveness of output scenes Existing feature film scenes Suitability for genre type Scenes of unknown scripts categorised by readers Functionality of interface Testing with drama students and directors

RELATION TO OTHER WORK

CONCLUSION AND FUTURE WORK • Automatic expressive multi-media animation of screenplays • Focus on: – automatic reasoning about story context and emotional interpretation – based on world knowledge and context memory – emotions influencing scene compositions and event execution – scene direction refined by genre-specifics • Analysis of script format to access semantic information • Automatic genre specification from script • Heightened expressiveness, naturalness and artistic quality • Assist directors, actors, drama students, script writers • Future work: Implementation & Testing of Scene. Maker

Thank you. QUESTIONS OR COMMENTS?