Designing Tools Enhance Interactive Experiences the development process

Designing Tools Enhance Interactive Experiences & the development process Magy Seif El-Nasr College of IST Penn State University

Interactive Entertainment: Industry Impact

Interactive Entertainment: Industry Impact

Interactive Entertainment: Industry Impact Game industry profits: n n n 2002: $1 billion more than movie industry 2004: EA reported $1. 43 billion quarterly earning 2005: Nintendo $4 billion and expects to sell at least 2 million Wiis in US alone

Importance of Game Engines + tools Training Entertainment Devil May Cry II Education Virtual Aquarium, NCSU Leaders project, ICT

code Tools

Assembly. radix 16 ; all numbers hexadecimal start: mov #41, @#8264 ; system variable loop: jsr r 4, print 1. asciz "HELLO WORLD! " br loop print 1: jsr pc, @#1248 ; print string (ROM call) rts r 4 C# System. Writeline (“Hello World”);

Learning and Computer literacy Impact Several Researchers believe that “video games provide an easy lead-in to computer literacy” – Cassell 2000 High Schools students and Modding Books on Modding for teens gaining popularity

Interactive Entertainment: Motivation and Learning “I want to learn how to make an awesome video game. I would like to learn everything about technology or at least more than I did” - middle school girl “I want to make my characters talk, build a world, and make interesting stories” - another middle school girl

Impact and Importance of the tools Facilitate development: games, training, health therapy application Facilitate learning: media, computing literacy Facilitate Creativity

Two thrusts of my research Tools for building engaging interactive experiences Game Engines + Tools as learning environments

Two thrusts of my research Tools for building engaging interactive experiences Game Engines + Tools as learning environments Focus: visual design and character design Based on cinematic and theatric theory Using CI and AI techniques

My Research Tools n n Lighting Camera (in progress) Character (in progress) Dance (in Progress)

Why Lighting? Lighting is Everything

Mii • • BS in Computer Science Directed in Theatre Graphics Design in Advertisement MS in Computer Science Studied Psychology and Neuroscience Ph. D in Computer Science 2 years studied acting & lighting design at Northwestern

Why Lighting? Dramatic Tension Visual Focus Style Feel of the space Time of Day Period visibility

Lighting in Traditional Media

Game Lighting Static Lighting Design Manually setting light layout (light maps) x Advantages: n n realistic Controllable = Images from Lightmaps (static shadowmaps) article written by Kurt Miller from: http: //www. flipcode. com/articles/article_lightmaps. shtml Disadvantages: n n Don’t adapt to variations in the environment Requires much memory Image from Max Payne

Game Lighting Static + Dynamic Lighting Manually scripting Lighting effects Dynamic Character Lighting Real-time Shadows Dynamic user control of some lights

Game Lighting Static + Dynamic Lighting

Game Lighting Static + Dynamic Lighting Advantages: More realistic Adaptable to change Disadvantages: Effects are scripted and rely on very careful design Restricted Based on realism Compositional Balancing is done at design time

Problems: No modulation Screenshot from Mission 21 Devil May Cry Saturation Graph for Screenshot

Problems (Blade of Darkness): No adaptation to game Play

Problem 1: unpredictability Develop a lighting plot or setup based on: Narrative configuration: n n Story Conflict/dramatic tension } Timing and dramatic progression Physical Configuration: n n Camera orientation and position Characters positions and orientations Requiring dynamic adaptation depend on user

Two Goals Dynamic Adaptive Lighting: Better gameplay Facilitate new Interactive models High-Level authoring of lighting: Faster prototyping Exploration of design space

Developing such as a Design Tool is Hard Why? • Technical Issues of Dynamic Lighting • Artistic control for style • low-level balancing of context and lighting parameters but provide continuity

ELE – Expressive Lighting Engine A lighting system that: Intelligently adjusts lighting in real-time accommodate context and effect Based on cinematic & theatric theory Allow artist to control lighting at a high-level

ELE Artistic Constraints Previous State Current State WAMP (World Action Message Protocol) Allocation Subsystem Game/Rendering Engine Angle Subsystem Color Subsystem LAMP (Lighting Action Message Protocol) use optimization to find best solution given context, desired effects, state, and artists’ constraints The three subsystems:

ELE Artistic Constraints Previous State Current State WAMP (World Action Message Protocol) Allocation Subsystem Game/Rendering Engine Angle Subsystem Color Subsystem LAMP (Lighting Action Message Protocol) use optimization to find best solution given context, desired effects, state, and artists’ constraints The three subsystems:

Automatic light allocation find best allocation given artistic constraints: n n n Modeling Depth Visibility Visual Continuity Visual Focus Low vs. high key

Algorithm for dynamic allocation 1. 2. 3. Calculate visible area Divide the zone into overlapping areas Allocate a number of lights to areas, given that minimize: visibility depth modeling Visual continuity

Layout of lights

ELE Artistic Constraints Previous State Current State WAMP (World Action Message Protocol) Allocation Subsystem Game/Rendering Engine Angle Subsystem Color Subsystem LAMP (Lighting Action Message Protocol) use optimization to find best solution given context, desired effects, state, and artists’ constraints The three subsystems:

Lighting angle Selection Select azimuth, elevation angles, given artistic constraints : n n n Visual Continuity Motivation of direction Visibility Modeling Mood ELE finds best light angles to meet goals

Automating cinematic lighting design Choosing key light azimuth angle Optimizes: visual mood motivation continuity k is the key light angle, k- is angle previous frame m is the mood angle desired, V is visibility v is the cost of deviation from best visibility angle - is the cost of visual continuity l is the cost of deviation from realistic direction m is the cost of deviation from ideal mood angle visibility & modeling • • •

Evaluation of key light angle visibility & modeling Light s Nose line k Camera From: Millerson’s The technique of lighting for television and Film, 1991 We described these rules as :

ELE: Angle Subsystem Showing character’s facial expressions and gestures

ELE: Angle Subsystem Showing character’s facial expressions and gestures, high tension

ELE: Angle Subsystem Mood Angle = side angle Emphasizing mood and mystery

ELE: Angle Subsystem Mood Angle = side angle Emphasizing mood

ELE Artistic Constraints Previous State Current State WAMP (World Action Message Protocol) Allocation Subsystem Game/Rendering Engine Angle Subsystem Color Subsystem LAMP (Lighting Action Message Protocol) use optimization to find best solution given context, desired effects, state, and artists’ constraints The three subsystems:

Choosing Colors Color (Hue + saturation + intensity) Compose colors for different areas on the set

Choosing Colors Adjust colors to accommodate desired artistic constraints: n Depth n Dramatic Intensity n Dramatic focus n low vs. high key setting n Specific author-suggested Hue, Saturation, Lightness, color Warmth for focus, non-focus, and background Palette restrictions specifying style + maintain visual continuity and style

Choosing Colors Optimize: depth contrast Palette constraints Visual continuity Artist’s desired color parameters

Choosing Colors Saturation Hue Lightness Warmth

Calculating Color Warmth Based on warmth perception Warm Used linear fit to psychophysical data: Cool

ELE: Color Subsystem Lightness (NF) Contrast (F – NF) Warmth (NF) warm tones, low color contrast for low tension, realistic colors

ELE: Color Subsystem cool tones, low color contrast for night scenes, or emphasize character

ELE: Color Subsystem warm, high intensity contrast for high tensions scenes

Two Goals Dynamic Adaptive Lighting: Better gameplay Facilitate new Interactive models High-Level authoring of lighting: Faster prototyping Exploration of design space

Use Light as tension meter Dynamic Intelligent Cinematic Lighting can make a difference in games

Demo (Dynamic Lighting)

Demo (Static Lighting)

Results of Pilot Study – during CHI 2005 Non-players liked the visual representation Liked representation of information through lighting Some players said: it made the game too easy Some gamers said: lighting was disturbing

Adapting Difficulty of Spotting enemies Adaptive Visual Focus can make a difference in games

Two Goals Dynamic Adaptive Lighting: Better gameplay Facilitate new Interactive models High-Level authoring of lighting: Faster prototyping Exploration of design space

Elevator

Elevator EDISON Edge Detection and Image Segmentati. ON [Christoudias et al. 2002] Mean-shift approach [Comaniciu and Meer 2002] Design Galleries [Marks et al. 1997]

User adjusting Lighting in Games like The Sims or The Movies A comment from an educator: “This can be used as a way of promoting Media Literacy”

Classes – Using the Tools 400 -level interdisciplinary course on Interactive Narrative, tools: n n C# Flash ELE Interactive Narrative Engine

Classes – Using the tools 400 -level Game Design, tools: n n n Unreal + ELE Half Life 2 Torque

Classes – Exploring theory 400 -level Interdisciplinary course on Design of Immersive Environments

Future Directions Tools for building engaging interactive experiences More on lighting Lighting: industry partners Continue on Projects: n n n Camera Character Dance

Current Projects Ambient Intelligence and Dance designing tools to allow artists to dictate aesthetic effect Intelligent Systems Pressure + Physiological Sensors

Current Projects Comedia dell’arte in Second Life Character, gesture, and camera tools Virtual Actor Training

Current Projects Lockheed Martin Lighting visual design camera character Living Hypothesis

Future Directions Game Engines + Tools as learning environments Use more of the research tools Impact of the use of the research tools Specifically our work on Games and Modding

Classes – Using Game Modding Middle/high school Gaming for Girls Future Work: n n n Camera tool Lighting Character

In Conclusion

Conclusion Game Industry is a growing industry Game tools are important Devil May Cry II Leaders project, ICT In my research I develop new tools to n n Enhance design process Enhance interactive experience Application of these tools: Education, Training, and other applications

Questions? http: //faculty. ist. psu. edu/ seifel-Nasr/ magy@ist. psu. edu

Modding and Learning Transfer of Computer Science, Math, Art Concepts Different engines promote different concepts, and require different pre-reqs Increase self efficacy of high-school female students Materialize abstract concepts Learn something about themselves Different engines are popular among different groups

Concepts Promoted thru Modding Essential Concepts n Assumptions of engines need to be understood, e. g. Parallel processing in Warcraft III n Concepts you need know to operate the engine, Co-ordinate system Color, contrast, etc. to use ELE Assignment-dependant concepts, e. g. n Character Modeling

Modding and Learning Transfer of Computer Science, Math, Art Concepts Different engines promote different concepts, and require different pre-reqs n n n Graphics: Vector Math, Geometry, Animation Programming: Parallel processing, event programming, Object Oriented Programming Arts: Architecture Design, map design, visual Composition, Lighting, camera movement, etc.

Game Projects involved Project Management Group Work iteration design Prototyping Critique Game Design concepts: n n n Mechanics Rules etc

What do students learn when they are building games using the current tools?