Reinforcement Learning II Ata Kaban A Kabancs bham

Reinforcement Learning (II. ) Ata Kaban A. Kaban@cs. bham. ac. uk School of Computer Science University of Birmingham

Recall • • • Policy: what to do Reward: what is good Value: what is good because it predicts reward Model: what follows what Reinforcement Learning: learning what to do from interactions with the environment

Recall • Markov Decision Process • rt and st+1 depend only on the current (st) state and action (at) • Goal: get as much eventual reward as possible no matter from which state you start off.

Today’s lecture • We recall the formalism for the deterministic case • We reformulate the formalism for nondeterministic case • We learn about: – – Bellman equations Optimal policy Policy iteration Q-learning in nondeterministic environment

“What to learn” • Learn an action policy so that it maximises the eventual reward from each state • Learn this from interaction examples, i. e. data of the form ((s, a), r) • Learn an action policy so that it maximises the expected eventual reward from each state • Learn this from interaction examples, i. e. data of the form ((s, a), r)

Notations used: We assume we are at a time t, so Recall & summarize other notations as well: • Policy: . • Remember in the deterministic case, (s) is an action • In the non-deterministic case, (s) is a random variable, ie we can only talk about (a|s), which is the probability of doing action a in state s • State values in a policy: V (s) • Values of state-action pairs (i. e. Q-values): Q(s, a) • State transitions: the next state depends on the current state and current action. • ‘the state that deterministically follows s if action a is taken’: (s, a) • Now the state transitions may also be probabilistic: then, ‘the probability that s’ follows s if action a is taken’ is p(s’|s, a)

State value function • How much reward can I accumulate from state s if I follow policy π • How much reward can I expect to accumulate from state s if I follow policy π This is called ‘Bellman equation’ It is a linear system which has a unique solution!

Example • Compute V* for π*: • Compute Vα for random pol πα V*(s 5) = 0 + 0. 9*100 = 90 V α(s 6) = 0. 5*(100+0. 9*0) +0. 5*(0+0. 9*0) = 50 V α(s 5) = 0. 33*(0 + 0. 9*50) +0. 66*(0 + 0. 9*0) = 30 V α(s 4) = 0. 5*(0 + 0. 9*30) +0. 5*(0+0. 9*0) = 13. 5 Etc… If computed for all states, then start again and keep iterating till the values converge V*(s 4) = 0 + 0. 9*90 = 81 Btw, has p(s’|s, a) disappeared? V*(s 6) = 100 + 0. 9*0 = 100

What is good about it? • For any policy, Bellman eq has unique solution it has unique solution for an optimal policy as well. Denote this by V*. • The optimal policy is what we want to learn. Denote it by * • If we could learn V*, then with one look-ahead we could compute * : • How to learn V*? It depends on *, which is unknown as well: • Iterate and improve on both in each iteration until converging to V* and an associated *. This is called (generalised) policy iteration.

Generalised Policy Iteration Geometric illustration

Before going on – what does it mean ‘optimal policy’ more exactly? • π is said to be ‘better’ then another policy π’, • In an MDP there always exists at least one policy which is better than all others. This is called ‘optimal policy’. • Any policy which is greedy with respect to V* is an optimal policy.

What is bad about it? • We need to know the model of the environment for doing policy iteration. – i. e. we need to know the state transitions (‘what follows what’) • We need to be able to look one step ahead – – i. e. to ‘try out’ all actions in order to choose the best one In some applications this is not feasible Is some others it is Can you think of any examples? • a fierce battle? • an Internet crawler?

Looking ahead… Backup diagram

‘The other route: ’ Action value function • How much eventual reward can I get if making action a from state s. • How much eventual reward can I expect to get if making action a from state s. This is also a ‘Bellman equation’ It has a unique solution!

What is good about it? • If we know Q, look-ahead is not needed for following an optimal policy! i. e. if we know all action values then just do the best action from each state. • Different implementations exist in the literature to improve efficiency. We will stick with turning the Bellman equation for action-value functions into an iterative update.

Simple example of updating Q Grid world, the rewards are given on the arrows s 1 s 2 s 4 s 5 s 3 s 6 Here, the Q values from a previous iteration are given on the arrows Recall: ‘Simple’, i. e. observe this is a deterministic world.

Non-deterministic example: exercise MDP model of a fierce battle. See the full text on your worksheet Non-deterministic actions: the same action from the same state might have different consequences.

• First iteration: Q(L 1, A)=0+0. 9*(0. 7*50+0. 3*0)=… Q(L 1, S)=0+0. 9*(0. 5*(-50)+0. 5*0)=… Q(L 1, M)=… … • What is your optimal action plan?

Key points • • • Learning by reinforcement Markov Decision Processes Value functions Optimal policy Bellman equations Methods and implementations for computing value functions – Policy iteration – Q-learning