The Science of Play Testing EAs Methods for

The Science of Play Testing: EA’s Methods for User Research Veronica Zammitto Game User Researcher

Outline: Game User Experience Evaluations > Case Study 1: NBA Live 10 > Case Study 2: NHL 11 > Take Away > Q&A > Veronica Zammitto

Game User Experience Evaluation Common Current Techniques New Techniques Qualitative Methods: • Interview • Focus Groups • Think Aloud • Survey Quantitative Methods: • Psychophysiological signals (‘biometrics’) • Eye Tracking • Telemetry • Subjective information interpreted by an expert. • Answers “Why” questions. • Measureable, objective, continuous information. • Answers “What” questions Veronica Zammitto

Mixed Method Approach for Evaluating Sports Games > Triangulating: Eye Tracking • In-depth understanding of User Experience (UX). • Support design decisions. Interviews Survey UX Telemetr y Psychophysiology Veronica Zammitto

Veronica Zammitto

Veronica Zammitto

Veronica Zammitto

Case Study 1: NBA Live 10 > NBA gameplay issues to identify: > successful and unsuccessful gameplay aspects. > emotional profile of the player > engagement and emotions > attentional focus > UX throughout the game, and for certain events. Veronica Zammitto

Eye Tracking Hardware + software X, Y on screen > Tracking users’ gaze can reveal the player’s focus. > DIYS, ~US$ 4, 000 to 80, 000 > Veronica Zammitto

Veronica Zammitto

Eye Tracking > By using ET we can identify where players’ attention is. > > Fixation Saccades Gaze Movement Patterns Veronica Zammitto

Aspects Attention Play Style 2 70 % on Core Gameplay Elements (ball carrier, court assessment) Time Play Style 1 50 % on Core Gameplay Elements (ball carrier, court assessment) 50 % on non-core Gameplay Elements (audience, players on the bench, coaches) Clock is not checked often. Focus more on core gameplay elements during the last minute. Facts Focus more on core gameplay elements during the last minute (might be missed) Low or lack of interest. Medium Mainly use simple pass. Almost no use of play calling. Need to be provided with strategies Need of positive reinforcements from coach and teammates. Effective, looking mainly at icons and logos. Medium-high Fair use of direct-pass. Fair use of play calling. Applies strategies in his games. Performance In-game Adv 30 % on non-core Gameplay Elements (audience, players on the bench, coaches) Clock is checked often. Medium interest. Positive reinforcement from players at the bench and coach. Effective, looking mainly at icons and logos Veronica Zammitto

Aspects Attention Play Style 1 Play Style 2 50 % on Core Gameplay Elements 70 % on Core Gameplay Elements (ball carrier, court assessment) Time 50 % on non- Core Gameplay Elements (audience, players on the bench, coaches) Clock is not checked often. 30 % on non- Core Gameplay Elements (audience, players on the bench, coaches) Clock is checked often. Focus more on core gameplay elements during the last minute. Facts Focus more on core gameplay elements during the last minute (might be missed) Low or lack of interest. Medium Mainly use simple pass. Almost no use of play calling. Need to be provided with strategies Need of positive reinforcements from coach and teammates. Effective, looking mainly at icons and logos. Medium-high Fair use of direct-pass. Fair use of play calling. Applies strategies in his games. Performance In-game Adv Medium interest. Positive reinforcement from players at the bench and coach. Effective, looking mainly at icons and logos Veronica Zammitto

Aspects Attention Play Style 2 70 % on Core Gameplay Elements (ball carrier, court assessment) Time Play Style 1 50 % on Core Gameplay Elements (ball carrier, court assessment) 50 % on non- Core Gameplay Elements (audience, players on the bench, coaches) Clock is not checked often. Focus more on core gameplay elements during the last minute. Facts Focus more on core gameplay elements during the last minute (might be missed) Low or lack of interest. Medium Mainly use simple pass. Almost no use of play calling. Need to be provided with strategies Need of positive reinforcements from coach and teammates. Effective, looking mainly at icons and logos. Medium-high Fair use of direct-pass. Fair use of play calling. Applies strategies in his games. Performance In-game Adv 30 % on non- Core Gameplay Elements (audience, players on the bench, coaches) Clock is checked often. Medium interest. Positive reinforcement from players at the bench and coach. Effective, looking mainly at icons and logos Veronica Zammitto

Aspects Attention Play Style 2 70 % on Core Gameplay Elements (ball carrier, court assessment) Time Play Style 1 50 % on Core Gameplay Elements (ball carrier, court assessment) 50 % on non- Core Gameplay Elements (audience, players on the bench, coaches) Clock is not checked often. Focus more on core gameplay elements during the last minute. Facts Focus more on core gameplay elements during the last minute (might be missed) Low or lack of interest. Medium Mainly use simple pass. Almost no use of play calling. Need to be provided with strategies Need of positive reinforcements from coach and teammates. Effective, looking mainly at icons and logos. Medium-high Fair use of direct-pass. Fair use of play calling. Applies strategies in his games. Performance In-game Adv 30 % on non- Core Gameplay Elements (audience, players on the bench, coaches) Clock is checked often. Medium interest. Positive reinforcement from players at the bench and coach. Effective, looking mainly at icons and logos Veronica Zammitto

Play Styles video Veronica Zammitto

Triangulating Eye Tracking & Survey Knowledge of the rules of basketball Knowledge of the NBA Live title Gamer type Play Style 1 Play Style 2 0% - expert 80 % - Intermediate 20 % - The Basics 80% - very familiar. 60 % - Expert 40 % - Intermediate 0% - The Basics 80% - very familiar. 20% - familiar. 40 % - Hardcore 60 % - Casual Main platform 80 % - Console use 20 % - PC Games bought 12. 2 per year An intermediate knowledge of the rules of basketball but very familiar with the NBA Profile Live franchise. Summary Leaning towards casual player who plays games a couple times per week, and prefers FPS. 60 % - Hardcore 40 % - Casual 80 % - Console 20 % - PC 7. 4 Expert knowledge of the rule of basketball and very familiar with the NBA Live franchise. Leaning towards hardcore players who play video games everyday but across many genres. Veronica Zammitto

Telemetry Lay Up Dunk Pass Call Time Out Switch Players Steal > Hooks in the game engine that flag and time stamp pre-defined events. > Players’ in-game behavior > > > Statistical analysis Visualizations Machine learning algorithms Veronica Zammitto

Events Performed by Players Through Time in NBA 10 Veronica Zammitto

Events Performed by Players Through Time in NBA 10 Veronica Zammitto

Passes sent by players Veronica Zammitto

Players’ Scoring Location 58. 6 % 91. 4 % of the shots Veronica Zammitto

AI Scoring Location 75 % Veronica Zammitto

What Went Right – NBA study > Better understanding of : > > The new techniques were proven to provide useful data to development. > > > players styles and demographics. In-game behavior Identification of emotions (next slides) Rethink the role of game elements. I. e. , coach. Create tutorials for court observation based on eye tracking data Worthy of further investment to continue with studies Veronica Zammitto

What Went Wrong > Large scope of the NBA study > > Impacted synchronization of the usability study with production’s delivery schedule. The study should have been subdivided into mini assessments to achieve a quicker turn around. Low involvement of production in the project. Manual coding: > Time consuming. Veronica Zammitto

Case Study 2: NHL 11 > Same techniques used for NBA > Adjustments from lessons learnt: > Narrower focus: “Game Presentation” > Front-End Visualizations (Overlays). > > Cut Scenes (NIS): > > I. e. : Do players look at information provided in the UI? Are they watched or skipped? High involvement with the development team > > Meetings with development for a ‘statement of work’. Iterative process with development > Helps to define the root of usability questions. Veronica Zammitto

Front End: Overlays Scoreboard Faceoff Line Change-Strategy - Line Change – Strategy Player - AI Penalty Offside Warning Fight Stamina AI Fight Stamina Player Fight Controls Icing Warning Pulling Home Goalie Player's Name -Overlay Scouting Report End of Period Stats Veronica Zammitto

Scoreboard Faceoff Overlay Offside Warning Icing Warning Penalty Overlay Line Change. Strategy Player Overlay Line Change. Strategy AI Overlay Fight-Controls Fight-Stamina Player Fight-Stamina AI Player’s Name Overlay Scouting Report End of Period Stats Veronica Zammitto

Scoreboard Faceoff Overlay Offside Warning Icing Warning Penalty Overlay Line Change. Strategy Player Overlay Line Change. Strategy AI Overlay Fight-Controls Fight-Stamina Player Fight-Stamina AI Player’s Name Overlay Scouting Report End of Period Stats Veronica Zammitto

Scoreboard Faceoff Overlay Offside Warning Icing Warning Penalty Overlay Line Change. Strategy Player Overlay Line Change. Strategy AI Overlay Fight-Controls Fight-Stamina Player Fight-Stamina AI Player’s Name Overlay Scouting Report End of Period Stats Veronica Zammitto

Scoreboard Faceoff Overlay Offside Warning Icing Warning Penalty Overlay Line Change. Strategy Player Overlay Line Change. Strategy AI Overlay Fight-Controls Fight-Stamina Player Fight-Stamina AI Player’s Name Overlay Scouting Report End of Period Stats Veronica Zammitto

Overlays Distribution 1 st Period Frequency 725 Percent 30. 2 2 nd Period 796 33. 2 3 rd Period 842 35. 1 OT 38 1. 6 2401 100. 0 Total Frequency Fight Percent 14 . 6 Gameplay 2038 84. 9 NIS 349 14. 5 Total 2401 100. 0 Veronica Zammitto

Percent Ignored 75. 7 Observed 24. 3 Total 100. 0 Only ¼ of the overlays are actually observed, the other 75% are ignored. Veronica Zammitto

Percentage of Observed and Ignored Overlays during different game sections. Quality and sensitive information that helps the player has more chances to be looked during NISes. Veronica Zammitto

Overlay “Map” in NHL 11 with Observed and Ignored proportions Veronica Zammitto

Veronica Zammitto

Overlay End of Period Stats Observation Count Observation Length (avg. visits per (avg. in seconds per appearance) 6. 36 2. 43 Scouting report 4. 09 1. 71 Faceoff-Overlay 2. 69 0. 2 Icing Warning 1. 67 0. 06 Offside Warning 2. 54 0. 19 Penalty Overlay 2. 59 0. 21 Pulling Home Goalie 5. 5 0. 28 Fight Controls 3. 25 0. 95 Fight Stamina AI 1. 5 0. 28 Fight Stamina Player 3. 25 0. 23 Line Change-Strat - AI 1. 86 0. 26 Line Change-Strat - Player 3. 27 0. 36 Player's Name - Overlay 3. 53 0. 78 How people observed the overlays. Veronica Zammitto

Non-Interactive Sequence (NIS) Event NISes Scripts - Subsets Veronica Zammitto

NIS Percentage Average per match Watched 37. 15 11. 03 Skipped 62. 85 18. 67 100 29. 7 Total NIS Canned Context Sensitive Replay Percentage 16. 29 37. 64 46. 07 % watched 31. 72 47. 16 30. 98 NIS’ Map % skipped 68. 28 52. 84 69. 02 Veronica Zammitto

NISes by event and type Veronica Zammitto

The effectiveness of NIS is a combination of its type (Canned, Context Sensitive, and Replay), the event that triggers the sequence, and its frequency. Veronica Zammitto

Psychophysiology (Biometrics) > Infer emotions from physiological data. > > Arousal: engagement, excitement, magnitude of emotions. Valence: positive (fun) or negative (frustration) arousal Frustrating Exciting Emotional Valence Boring Fun Veronica Zammitto

Galvanic Skin Response (GSR) > Psychological arousal. > Skin’s conductance increases when a person becomes excited, stressed, or anxious. Veronica Zammitto

Veronica Zammitto

Electromyography (EMG) > Emotional valence. > Sensors capture and amplify muscles’ contractions. > Facial muscles: > > Smiling (zygomatic) = Positive emotions Frowning (corrugator) = Negative emotions (or) Cognitive load Veronica Zammitto

Facial EMG Veronica Zammitto

Veronica Zammitto

Player’s positive emotional reaction when scoring in NBA Live 10 Veronica Zammitto

Isomorphism between telemetry and GSR Veronica Zammitto

Emotional Profiling of NHL 11 Arousal = exciting, engagement. Veronica Zammitto

Levels of Arousal in NHL 11: Tendency of increasing arousal throughout the game. Veronica Zammitto

Player 2 vs AI 0 Player 6 vs AI 2 “I had a good game, but I should have readjusted the AI level. To me it’s also a challenge to score a lot of goals when I know I’m going to win, then it’s how many. So, you established your own achievement. Yes, I wanted to get 6 at least by the end of the game. ” Veronica Zammitto

NHL 11: Arousal profile NHL’s Arousal Profile Veronica Zammitto

Arousal-Valence Space Arousal = excitement Valence = > > positive emotions Negative emotions Veronica Zammitto

Veronica Zammitto

Veronica Zammitto

Veronica Zammitto

Veronica Zammitto

Take Away > New possibilities in user experience, mixed method with: ü Eye-tracking ü Biometrics ü Telemetry > Identifying the concrete information, cross-references. ü Emotions ü Engagement > ü Interview ü Survey ü Attention ü In-game behavior Information that supports design decisions. Veronica Zammitto

Thanks! Questions? Veronica Zammitto