CBOW Continuous Bag Of Words Model 2017 9

CBOW : Continuous Bag Of Words Model 2017. 9. 20 숭실대학교

Represent the meaning of word – word 2 vec 2 basic neural network models: • Skip-gram (SG) : predicting surrounding context words given a center word • Continuous Bag of Word (CBOW) : predicting a center word from the surrounding context words. CBOW Skip-gram 2 / 17

Continuous Bag of Words model (CBOW) The cat jumped over puddle CBOW 3 / 17

Continuous Bag of Words model (CBOW) |v| Black box 4 / 17

CBOW Example. “ The cat jumped over puddle “ 5 / 17

Step 1: generate one-hot vector the 1 cat jumped center word over puddle 0 0 0 1 0 0 0 1 0 0 6 / 17

Step 2: get embedded word vector … hidden layer 7 / 17

Step 2: get embedded word vector 8 / 17

Step 2: get embedded word vector 9 / 17

Step 2: get embedded word vector 10 / 17

Step 3: average vector average 11 / 17

Step 4: generate score vector … 12 / 17

Step 5: softmax Function 1 0 0 3 2 1 5 2 13 / 17

Continuous Bags of Word Model 14 / 17

Cross entropy -log(x) 그래프 15 / 17

Cross Entropy Cross entropy example -log(x) 그래프 16 / 17

Cross entropy 17 / 17