# Haitham Elmarakeby Sequence to Sequence Learning Sequence to

- Slides: 63

Haitham Elmarakeby Sequence to Sequence Learning

Sequence to Sequence �Speech recognition http: //nlp. stanford. edu/courses/lsa 352/

Sequence to Sequence �Machine translation ﻣﺮﺣﺒﺎ ﺑﻜﻢ ﻓﻲ ﺩﺭﺱ ﺍﻟﺘﻌﻠﻢ ﺍﻟﻌﻤﻴﻖ Welcome to the deep learning class

Sequence to Sequence �Question answering

Statistical Machine Translation Knight and Koehn 2003

Statistical Machine Translation Knight and Koehn 2003

Statistical Machine Translation Components �Translation model �Language Model �Decoding

Statistical Machine Translation �Translation model Learn the P(f | e) Knight and Koehn 2003

Statistical Machine Translation �Translation model � � � Input is Segmented in Phrases Each Phrase is Translated into English Phrases are Reordered Koehn 2004

Statistical Machine Translation �Language Model Goal of the Language Model: Detect good English P(e) Standard Technique: Trigram Model Knight and Koehn 2003

Statistical Machine Translation �Decoding Goal of the decoding algorithm: Put models to work, perform the actual translation Koehn 2004

Statistical Machine Translation �Decoding Goal of the decoding algorithm: Put models to work, perform the actual translation Koehn 2004

Statistical Machine Translation �Decoding Goal of the decoding algorithm: Put models to work, perform the actual translation Koehn 2004

Statistical Machine Translation �Decoding Goal of the decoding algorithm: Put models to work, perform the actual translation � Prune out Weakest Hypotheses by absolute threshold (keep 100 best) by relative cutoff � Future Cost Estimation compute expected cost of untranslated words

Sutskever et al. , 2014 Sequence to Sequence Learning with Neural Networks

Neural Machine Translation �Model W X Y Z A B C

Neural Machine Translation �Model Sutskever et al. 2014

Neural Machine Translation �Model- encoder Cho: From Sequence Modeling to Translation

Neural Machine Translation �Model- encoder Cho: From Sequence Modeling to Translation

Neural Machine Translation �Model- encoder Cho: From Sequence Modeling to Translation

Neural Machine Translation �Model- encoder Cho: From Sequence Modeling to Translation

Neural Machine Translation �Model- encoder Cho: From Sequence Modeling to Translation

Neural Machine Translation �Model- decoder Cho: From Sequence Modeling to Translation

Neural Machine Translation �Model- decoder Cho: From Sequence Modeling to Translation

Neural Machine Translation �Model- decoder Cho: From Sequence Modeling to Translation

Neural Machine Translation �RNN

Neural Machine Translation �RNN Vanishing gradient Cho: From Sequence Modeling to Translation

Neural Machine Translation �LSTM Graves 2013

Neural Machine Translation �LSTM Problem: Exploding gradient

Neural Machine Translation �LSTM Problem: Exploding gradient Solution: Scaling gradient

Sequence to Sequence � Reversing the Source Sentences Welcome to the deep learning class

Sequence to Sequence � Reversing the Source Sentences Welcome to the deep learning class

Sequence to Sequence � Results BLEU score (Bilingual Evaluation Understudy) Candidate the the Reference 1 Reference 2 there cat is is a on cat the on mat the mat P = m/w= 7/7 = 1 Papineni et al. 2002

Sequence to Sequence � Results BLEU score (Bilingual Evaluation Understudy) Candidate the the Reference 1 Reference 2 there cat is is a on cat the on mat the mat P = 2/7 Papineni et al. 2002

Sequence to Sequence � Results Sutskever et al. 2014

Sequence to Sequence � Results Sutskever et al. 2014

Sequence to Sequence � Model Analysis Sutskever et al. 2014

Sequence to Sequence � Long sentences Sutskever et al. 2014

Sequence to Sequence � Long sentences Cho et al. 2014

Bahdanau et al. , 2014 Neural Machine Translation by Jointly Learning to Align and Translate

Sequence to Sequence � Long sentences Fixed length representation maybe the cause

Jointly Learning to Align and Translate �Attention mechanism

Jointly Learning to Align and Translate �Attention mechanism

Jointly Learning to Align and Translate �Attention mechanism

Jointly Learning to Align and Translate �Attention mechanism

Jointly Learning to Align and Translate �Attention mechanism

Jointly Learning to Align and Translate �Attention mechanism

Jointly Learning to Align and Translate �Attention mechanism

Jointly Learning to Align and Translate � Long sentences Cho et al. 2014

Vinyals et al. , 2015 Grammar as a Foreign Language

Grammar as a Foreign Language Parsing tree

Grammar as a Foreign Language Parsing tree

Grammar as a Foreign Language Parsing tree

Grammar as a Foreign Language Parsing tree

Grammar as a Foreign Language Parsing tree John has a dog .

Grammar as a Foreign Language Converting tree to sequence

Grammar as a Foreign Language Converting tree to sequence

Grammar as a Foreign Language Model

Grammar as a Foreign Language Results

- Haitham hassanieh
- Cuadro comparativo entre e-learning b-learning y m-learning
- Convolutional sequence to sequence learning.
- Lda supervised or unsupervised
- Concept learning task in machine learning
- Analytical learning in machine learning
- Difference between associative and non associative learning
- Eager learning algorithm example
- Deductive learning vs inductive learning
- Difference between inductive and analytical learning
- Apprenticeship learning via inverse reinforcement learning
- Apprenticeship learning via inverse reinforcement learning
- Inductive vs deductive reasoning
- Pac learning model in machine learning
- Data mining unsupervised learning
- Machine learning t mitchell
- Inductive vs analytical learning
- Instance based learning in machine learning
- Inductive learning machine learning
- First order rule learning in machine learning
- Collaborative learning vs cooperative learning
- Alternative learning system learning strands
- Active and passive learners
- Multiagent learning using a variable learning rate
- Cmu machine learning
- Tony wagner's seven survival skills
- Self-taught learning: transfer learning from unlabeled data
- Active learning reinforcement learning
- Nucleotide to amino acid
- Get sequence get another sequence pseudocode
- Difference between infinite and finite sequence
- Zoll elearning
- Learning task 2 im an architect
- Learning task 2 look for a health product at home
- Learning task 1 draw a face
- Learning without burden introduction
- Braided learning theory
- Numeracy learning progressions
- Objective of email writing
- Lifelong learning essay conclusion
- Enchanted learning whales
- Work based learning license mn
- Learning wireshark
- Wipo distance learning
- Wipo academy courses
- What is computer assisted language learning
- Aalas learning library login
- Disadvantages of hands-on learning
- In the afylin framework learning outcomes are arranged
- Dr. krishna's learning centre
- Lippincot direct
- Learning thursdays
- Global learning collaborative high school
- Principles of instructed language learning
- Informative learning
- Aquinas learning cycle 1
- Water cycle learning outcomes
- Main vital signs
- Virginia kindergarten standards
- Verb viper
- Action learning approach in values education
- Learning platform for ipas
- Pros and cons of kinesthetic learning
- Unsupervised learning in data mining