Using Hierarchical Reinforcement Learning to Solve a Problem

Using Hierarchical Reinforcement Learning to Solve a Problem with Multiple Conflicting Sub-problems

Reinforcement Learning • • Involves an agent interacting with an environment The agent can be in one of various states in the environment The agent is not told which action is correct, but is given a measure of an action for a given state After a while the agent develops a policy

The curse • • • As complexity of the environment grows, state space increases exponentially We can try to cleverly reduce state space Hierarchical reinforcement learning

Hierarchical Reinforcement Learning • • A complex problem can often be broken up into multiple conflicting sub-problems Hierarchical reinforcement learning can handle this Deals with each sub-problem separately using reinforcement learning Decides which sub-problem to attempt next using reinforcement learning

A Practical Example: The Mars Lander Perform Various Conflicting Tasks: • Explore the terrain • Collect soil samples • Return to base for refuelling

My Project • • • Apply hierarchical reinforcement learning to a complex problem Consist of an agent existing in an environment where it will have to achieve an overall goal Agent will be a primitive creature trying to survive in the wilderness

My Project • • • The overall goal will be for the creature to remain happy or comfortable in the wilderness Overall goal can be divided into sub-goals These sub-goals will be: – – – Eating food Drinking water Resting under a Shelter Repairing Shelter Avoiding hazards

The Gridworld

Motivation for this approach • X pos • 13 x 10 x 10 = 1690000 Possible states Sub-goals separated out: • • • Y pos Hunger Thirst Fatigue Shelter Condition (Xpos, Ypos, hunger) , (Xpos, Ypos, Thirst) (Xpos, Ypos, Fatigue), (Xpos, Ypos, Shelter Condition) (13 x 10) x 4 =1690 x 4 = 6760 Possible states