The Turing Test for Game AI Bing Gordon

The Turing Test for Game AI Bing Gordon Electronic Arts June 2005

“I wish someone would…” …develop a 'metric' for computer npc ai. Kind of an eliza scorecard. ”

1. Context Powers of 10… 1981 Atari VCS 1986 NES 1989 Sega Genesis 1994 Sony Playstation 2001 Playstation 2 2006 PSX 3/Xbox 2 2012? PS 4/Xbox 3? 128 bytes 2 K + 2 K 64 K+64 K 2 MB + 512 K 32 MB +4 MB 512 MB 5 G?

Context Powers of polish … 75% of 90+ rated games are hits. 25% of 90+ rated movies are hits. Source: Metacritic. com

Powered by Feature Innovation Aspirations Tiger swing Gameface Franchise mode Dynamic sports news Assymmetrical sides Badges Anti-grav gun Auction House Living city Crash mode Freestyle Console online voice and custom maps

Quality bars of the near-future Procedurally generated content “Emergent” behaviors, collisions Believable characters 100 x physics Portable avatars, persistent assets Communities Economies and money Camera POV and LOD drives gameplay Collaborative and dynamic intelligences

AI could be a “killer app” feature of next gen Characters: – – Awareness Memory Complex motives, simple commands 100 x RAM allocation Procedural responses – Parsers – Stitching – Pathfinding and cover Must be co-developed with animators! – – Game AI must be acted out and seen Expressions & gestures> pathing

2. EA Learning We are Influenced by the success of our “Killer Apps” Pinball Construction Set -- floppy disk Deluxe Paint -- graphics co-processor EA Sports – 16 -bit processor Interactive movies – cd-rom MMO’s – servers Sims – object-oriented programming Spore – procedural animation Etc.

2. EA Learning EA Sports is a tech story 16 -bit (1990) Sports simulations – Individual ratings – Dynamic simulations Playstation (1994) 3 D Sports – Z-axis passing, tackling Playstation 2 (2000) Articulation – Skeletal modeling and animation

What EA learned from John Madden The Oakland Raiders playbook – Matchup strategy – 5 zones of field-position One Knee Equals Two Feet – Player ratings – All-Madden team

The Madden Test (of game AI) 1985: “That’s not football!” 1990: “I’d fire the coach!” 1995: “What are those guys doing? ” 2000: “Rookie, you’re cut!” 2005: “That’s the way I designed it!”

Madden Football Genesis 23%

Madden 97 Playstation 54%

Madden 2001 Playstation 2 63%

Madden 2005 PS 2 83%

3. Reviewers What EA learned from www. metacritic. com 1) 2) 3) 4) 5) 6) 7) 8) 9) 10) 11) 94 -- Madden NFL 2002 12) 94 -- Madden NFL 2004 13) 94 -- Burnout 3: 97 -- Grand Theft Auto III Takedown 97 -- Tony Hawk's Pro Skater 3 96 -- Metal Gear Solid 2: Sons of Liberty 14) 94 -- NCAA Football 2004 95 -- Grand Theft Auto: Vice City 15) 94 -- Virtua Fighter 4 95 -- Grand Theft Auto: San Andreas 16) 93 -- Pro Evolution 95 -- Gran Turismo 3 A-Spec Soccer 95 -- Madden NFL 2003 17) 93 -- SSX 95 -- Tony Hawk's Pro Skater 4 18) 93 -- SSX 3 95 -- Devil May Cry 19) 93 -- World Soccer 94 -- God of War Winning Eleven 6 International 20) 93 -- NFL 2 K

3. Reviewers AI is not critical…yet AI cited for 6/20 top PS 2 games. – Metal Gear, NFL x 4, Soccer AI cited for 3/10 top PC games. – Half Life x 2, Civilization

When AI is applauded… Appropriate npc behaviors – Dynamic adjustments – Satisfying mistakes – Coordinated attacks/retreats Challenging opponent

When AI is punished… Too easy – No cover – “Money plays” Too dumb – Low awareness – “Deer in headlights”

"Do NOT duck inside those doors, men! Honorable warriors need no cover!" Pointless Waste of Time. com

4. An experiment A Turing Test for Game AI (version 1. 0) Clips from 5 games – Madden, FIFA, Fight Night, Mortal Kombat – Many others could not be tested “Player or AI controlled? ” App by Karim Mahrous, UCD phd candidate

Format of test Watch a clip, multiple replays Player or AI control for specific character. Rationale: – – – Pathing Reaction Timing Effectiveness Orientation Awareness 80 respondents

App

Madden 2001

Madden 2001 Results

Madden 2005 – Clip 1

Madden 2005 – Clip 1 Results

Madden 2005 – Clip 2

Madden 2005 – Clip 2 Results

Results are discouraging (so far) 60% accurate Clip > game or technology 0 correlation between rationale, correctness Possible improvement in “orientation” AI between Madden 01 and 05.

5. For example Feature metacritic planning for

Competitive quality bar Enemy Combat AI Metal Gear Solid 2 – – – 95 metacritic overall Cited by 4/5 top reviews #2 -4 feature mentioned “communication chain” “perceptive reinforcements” “teams of soldiers will find you” – “frighteningly intelligent” – “Outclassing every game on PS 2” Half Life 2 – – – 95 metacritic overall Cited by 2/3 top reviews #5 feature mentioned “use cover effectively” “run away from grenades” “actively hunt players down and flank you ” – “if you hurt them, they set up traps” – “Nothing less than incredible”

Hall of fame feature Opponent command AI Civilization 2 – – – – 94 overall Cited by 3/3 top reviews #2 -5 feature mentioned “you pay for your mistakes” “clever and resourceful in negotiation” “respects your power in battles” “may act like a bonehead at times” Madden Football – – – – 94 overall Cited by 3/6 top reviews #2 -6 feature mentioned “Geek out the AI to the viciousness you like” “Players make plays all over the field” “Tough half-time adjustments” “Gone are the massive scores…” “AI is advanced to another level”

Hall of Fame Feature Pathfinding and formations Rome: Total War – 92 overall – Cited by 2/4 top reviews – #5 feature mentioned – “enjoyable and convincing” – “centered around troop morale” – “moments of divine inspiration” – “on occasion, cretinous”

Recommended Spore AI goals Half Life 2: enemy hunting AI Metal Gear Solid 2: communication chains Civilization 2: city/star combat and diplomacy Madden Football: “half time adjustments” Rome Total War: pathfinding and formations Plus: – Procedural combat – Collaborative filtering

10 predictions Areas of discovery for next-gen features

My bookshelf Confessions of an Advertising Man, Ogilvy Positioning, Ries and Trout The Fifth Discipline, Senge How Hits Happen, Farrell Net Gain, Hagel & Armstrong The Innovator’s Dilemma, Christenson The Medici Effect, Frans Johansson

Thanks bgordon@ea. com