Hierarchical Plan Representations for Encoding Strategic Game AI

Hierarchical Plan Representations for Encoding Strategic Game AI Hai Hoang Stephen Lee-Urban Héctor Muñoz-Avila Lehigh University http: //www. cse. lehigh. edu/~munoz/projects/AIGames/

Outline n Motivation Related Work n Background n Unreal Tournament© (UT) q Hierarchical task Network (HTN) Planning Encoding Strategic Game AI in HTNs q Coordinating teams of UT Bots using HTNs q Video Using Processes-Based Languages to Encode HTNs Final Remarks q Future work q n n n

Planning Operators Motivation: FSM vs. Planning FSM: Monster In Sight Patrol Fight No Monster • Patrol ØPreconditions: No Monster ØEffects: patrolled • Fight ØPreconditions: Monster in sight ØEffects: No Monster A resulting plan: Monster in sight Fight No Monster Patrol patrolled Neither is more powerful than the other one

But Planning Gives More Flexibility n “Separates implementation from data” --- Orkin knowledge Planning Operators inference • Patrol ØPreconditions: No Monster ØEffects: patrolled • Fight ØPreconditions: Monster in sight ØEffects: No Monster … Many potential plans: Fight Fight … Patrol Patrol

Motivation: HTNs vs. GOAP n n n Goal-Oriented Action Planning (Orkin, 2004, 2005) q characters can find alternate solutions to situations encountered in the game Goals & q characters can handle dependencies that may not have operators been thought of at development time HTNs provide distinct advantages over “STRIPS” planning, on which GOAP is based Key questions: q What kinds of Game AI knowledge to represent in Game developers should HTNs? Learn LISP? q How to reason in dynamic game environments? q How to encode the HTNs

Related Work: FPS and AI n Games have the potential to become testbed for different AI algorithms (Laird & van Lent, 1999) n FPS-based system developed and used for actual training of military personnel (Laird & Duchim, 2000) n Soar Bots for UT using the Soar Architecture (Laird & Duchim, 2000) n Game. Bots: A group at USC-ISI and CMU jointly developed the Gamebot project to create an API for other clients to connect to the UT server.

Related Work: HTNs and Game AI n Hierarchical planning can be useful in games (Wallace, 2003) n Bridge Baron game uses HTN ideas (Smith et al. , 1998). n Hierarchical FSMs are an extension of FSMs in which states can expand into their own sub-FSMs (Houlette & Fu, 2003) n HTNs used for dialog generation (Cavazza & Charles, 2005)

Unreal Tournament© n Online FPS developed by Epic Games Inc. n Various gameplay modes including team deathmatch and capture the flag n Provides a client-server architecture for controlling bots n High-level script language: Unreal. Script (Sweeney, 2004)

Client-Server Architecture for UT Bots n The UT server provides sensory information about events in the UT world and controls all the gameplay events. n Event-driven: The client program uses this information to decide commands controlling the behavior of a Bot and passes them to the server. n Two types of messages: Ø Asynchronous: special events Ø Synchronous: regular intervals n Variants: Enhanced server (USC/ISI), Java Bots (CMU), Soar Bots (U. of Michigan).

Hierarchical Task Network (HTN) Planning n Complex tasks are decomposed into simpler tasks. task t n n task t Seeks to decompose compound tasks into primitive tasks, which define actions changing the world The knowledge artifacts indicating how to decompose task into subtasks are called methods

Why HTN Planning (1)? n HTN planning is provable more expressive than STRIPS planning (Erol et al, 1994) n Even for those domains in which HTN representations can be translated into STRIPS representations, a much larger number of STRIPS operators is required to represent the HTN methods (Lotem & Nau, 2000) n Fast planning through domain-configurable HTN planners (SHOP system)

Why HTN Planning (2)? n Hierarchical planning is natural in many domains q Military planning Strategic National Strategic Theater Operational Tactical JCS / NCA CINC JTF

Application: UT Domination Games n n A number of fixed domination locations. When a team member steps into one of these locations, the status of the location changes to be under the control of his/her team. The team gets a point for every five seconds that each domination location remains under the control of that team. The game is won by the first team that gets a pre-specified amount of points.

Idea n n Use of hierarchical task network (HTN) planning techniques to laid out a grand strategy to accomplish gaming tasks Use standard event-driven programming allowing the bots to react in highly dynamic environments while contributing to the grand task UT task: Domination Strategy: secure most locations UT action: move Bot 1 to location B

Technical Difficulties of using an HTN Planner (1) UT task: Domination Strategy: secure most locations UT action: move Bot 1 to location B 2. Dealing with contingencies while Bots are executing actions 1. Declarative language for expressing conditions Solution to 1: use builtin predicates to represent conditions Solution to 2: use Java Bots to represent actions (event-driven)

Data Flow to generate HTNs (i. e. , HTN Planning) Selected method’s subtasks task compound? Yes Select applicable method Evaluate method’s preconditions No Execute action Method library UT Bot … UT Server

Video

Empirical Results It is possible to encode strategies that coordinate teams of bots in FPS games using HTNs

Using Processes to Encode Game AI Think-Act Cycle Shoot Something Go Somewhere Else Call Shoot Operator Ask Decision System: Where Do I Go? Pick up a Healthpack Call Pickup Operator

TIELT Project: Middleware between Games and AI Decision Systems TIELT’s User Interface Game Engine Library Evaluation Interface Selected Game Engine Action TIELT’s Internal Communication Modules Processed State Decision Learned Knowledge (inspectable) Game Player(s) TIELT’s KB Editors Game Model Game Interface Model Decision System Interface Model Selected Decision System Agent Description Experiment Methodology Selected/Developed Knowledge Bases Knowledge Base Libraries GM GM GM GIM GIM DSIM AD AD AD Decision Reasoning System Learning Module . . TIELT User Coordination Interface Raw State . . . Full Spectrum Command Prediction Interface TIELT User Stratagus EE 2 Advice Interface Decision System Library EM EM EM Learning Module

Result: Processes can Be Translated into HTNs TIELT HTN Iterations (while) Recursion Iterations (for) Convert to while, recursion Tasks: preconditions and effects Methods + operators If-else Method Functions Parameters of task (or call to external code, or Horn clause) Set-statement = command Priorities If-then-else

Final Remarks n n n HTNs can be used to model effective team strategies for Unreal Tournament (UT) bots. A grand strategy is laid out by the HTNs and eventdriven programming allows the bots to react in this highly dynamic environment while contributing to the grand task. Processes languages (e. g. , TMKLs) can be translated into HTNs

Future Work n n n Reinforcement learning to learn adequate task decomposition Case-based reasoning techniques to capture and reuse HTNs Experiment to measure performance of translated processes