Sequence StudentTeacher Training of Deep Neural Networks Jeremy

Sequence Student-Teacher Training of Deep Neural Networks Jeremy H. M. Wong and Mark J. F. Gales Department of Engineering, University of Cambridge Trumpington Street, CB 2 1 PZ Cambridge, England

Outline • Ensemble methods • Student-teacher training • Integrating sequence training • Experimental results • Conclusions

Ensemble methods • Ensemble combinations can give significant gains over single systems – training data is limited • Computational demand of decoding scales linearly with ensemble size

Diversity • More diverse ensembles tend to give better combination gains – Correct each other’s errors – Wider space of possible models – bagging, random decision trees and Adaboost • Diversity introduced as different random DNN initialisations

Combinations •

Student-teacher training • General framework used to compress large model – Here, use ensemble of teachers – Decode only single student model – Consider nature of information propagated to student Information

Frame-level student-teacher training •

Integrating Sequence Training • Sequence training outperforms cross-entropy training. • Want to integrate sequence training into student-teacher training. • Possible ways: – Ensemble training: sequence train the teachers – Student refinement: sequence train the student – Information transfer: propagate sequence information

Integrating Sequence Training

Hypothesis-level student-teacher training •

Computing the gradient •

Datasets • IARPA Babel Tok Pisin (IARPA-babel 207 b-v 1. 0 e) – 3 hour VLLP training set – 10 hour development set • WSJ – 14 hour si-84 training set – 64 K words open-vocabulary eval-92 test set

Experimental Setup • Ensemble size = 10 (Tok Pisin), 4 (WSJ) • Acoustic model = DNN-HMM hybrid – 1000 nodes × 4 layers for Tok Pisin – 2000 nodes × 6 layers for WSJ • The student and teacher models have the same architecture

Combination of sequence-trained teachers • Tok Pisin WERs (%) – Training teachers with sequence criteria improves the combined ensemble performance

Ensemble training • Tok Pisin WERs (%) – Gains from sequence training of the teachers can be transferred to the student models

Student refinement – Further sequence training can bring additional gains

Information transfer • Hypothesis level – Gains from sequence training of the teachers can be transferred to the student models

Conclusions • Summary: – Investigated integrating sequence training into student-teacher training. – Proposed a new method to propagate hypothesis posteriors. – All three methods of integrating sequence training are complementary to student-teacher training. • Future work: – Investigate other forms of information to propagate. • e. g. Expected loss.