Introduction to Alpha Go Bosh Shih 1 Outline

Introduction to Alpha. Go Bosh Shih 1

Outline • Introduction • Value Network • Mini. Max • Alpha–beta pruning • Monte Carlo Tree Search • UCB 1 Algorithm • Background • • • Neural Network Convolution Neural Network Supervised Learning Training Neural Networks Reinforcement Learning Gradient Descent Policy Value Policy Gradient Method Grid World Example • Alpha. Go's Methods 2

Introduction • 3

Introduction • Value Network 1 1 2 2 3 4 4 16 5 1000 1 ? 2 ? 3 ? 4 ? 5 ? 4

Introduction • Searching • Mini. Max • minimize the possible loss for the worst case 5

Introduction • Alpha–beta pruning 4 4 5 6 3 4 7 8 9 6

Outline • Introduction • Value Network • Mini. Max • Alpha–beta pruning • Monte Carlo Tree Search • UCB 1 Algorithm • Background • • • Neural Network Convolution Neural Network Supervised Learning Training Neural Networks Reinforcement Learning Gradient Descent Policy Value Policy Gradient Method Grid World Example • Alpha. Go's Methods 7

Monte Carlo Tree Search MCTC 8

Monte Carlo Tree Search 1. Selection 2. Expansion 3. Simulation 4. Backpropagation Wins / Total 9

UCB 1 Algorithm • 7 7 7 mean 7 2/3 trade off 3 10

Outline • Introduction • Value Network • Mini. Max • Alpha–beta pruning • Monte Carlo Tree Search • UCB 1 Algorithm • Background • • • Neural Network Convolution Neural Network Supervised Learning Training Neural Networks Reinforcement Learning Gradient Descent Policy Value Policy Gradient Method Grid World Example • Alpha. Go's Methods 11

Neural Network Multi-class Softmax Backpropagation Ref : https: //mattmazur. com/2015/03/17/a-step-by-step-backpropagation-example/ 12

Convolution Neural Network 0 2 0 1 0 Image 1 0 0 1 0 4 2 0 Feature Map Response Random Artificial Fine Tune 2 2 1 2 Pooling Subsampling 13

Supervised Learning Training Machine Learning Model Testing Machine Learning Model 14

Supervised Learning Classification Regression 15

Training Neural Networks • One-Hot Encoding Input 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 Machine Learning Model 0 0 0 0 1 1 1 1 0 1 1 1 0 0 1 1 1 1 1 1 Output 0 0 0 0 0 0 1 0 0 0 0 0 Ground Truth 0 0 0 0 16

Training Neural Networks Input Machine Learning Model 0 0 0 0 0 7 0 0 0 2 0 0 0 1 0 0 0 0 0 Output 0 0 0 0 0 0 1 0 0 0 0 0 Ground Truth 0 0 0 0 17

Gradient Descent 18

Reinforcement Learning Lose or Win ? Reward (Feedback) Policy Delay Unknown Rules 19

Policy • Deterministic State s Action a • Stochastic State s 20

Value • State s 21

Policy Gradient Method • Reinforce Weights in policy function Baseline Learning rate Reward 22

Grid World Example End 0 0 0 1 0 0 0 0 0 Neural Network Policy Networks Start fail 23

Grid World Example Neural Network 24

Grid World Example Neural Network 25

Grid World Example Neural Network 26

Outline • Introduction • Value Network • Mini. Max • Alpha–beta pruning • Monte Carlo Tree Search • UCB 1 Algorithm • Background • • • Neural Network Convolution Neural Network Supervised Learning Training Neural Networks Reinforcement Learning Gradient Descent Policy Value Policy Gradient Method Grid World Example • Alpha. Go's Methods 27

Alpha. Go's Methods • Training: • Supervised Learning Classification • Reinforcement Learning • Reinforcement / Supervised Learning Regression Policy Network Value Network • Searching: • Monte Carlo Tree Search with Policy & Value network 28

Alpha. Go's Methods (Overview) Initial weight 29

Alpha. Go's Methods (Rollout , SL) Faster but less accurate , for Searching 13 -layer Convolutional Neural Network 50 GPU 3 weeks Accuracy : 57% KGS GO Dataset 160, 000 games 30

Alpha. Go's Methods (Rollout , SL) (input 1 2) 31

Alpha. Go's Methods (Rollout , SL) (input 1 2) • Symmetry Rotation Reflection 32

Alpha. Go's Methods (Rollout , SL) 13 th : Softmax 33

Alpha. Go's Methods (Overview) Initial weight 34

Alpha. Go's Methods (RL) 50 GPU , 1 day 80% won SL Policy Network 100, 000 games 35

Alpha. Go's Methods (RL) • Policy Gradient Method 36

Alpha. Go's Methods (Overview) 37

Alpha. Go's Methods (Value Network) 13 th : Softmax 15 th : Fully-connected + tanh 14 th : Fully-connected +Re. LU 38

Alpha. Go's Methods (Value Network) Value Network 39

Alpha. Go's Methods (Searching) • Searching 40

Alpha. Go's Methods (Searching) • 41

Alpha. Go's Methods (Searching) • Selection Exploitation Exploration PUCT Algorithm 42

Alpha. Go's Methods (Searching) • 43

Alpha. Go's Methods (Searching) • 44

Alpha. Go's Methods (Searching) • 45

Reference • MLDM – Alpha. Go • Alpha–beta pruning (Wiki) • Introduction to Monte Carlo Tree Search (Jeff Bradberry) • Mastering the game of Go with deep neural networks and tree search • Minimax (Wiki) • New Heuristics for Monte Carlo Tree Search Applied to the Game of Go • Teaching Deep Convolutional Neural Networks to Play Go • Alpha Go (Allan's blog) 46

47