n Adapting Hybrid ANNHMM to Speech Variations Stefano

n Adapting Hybrid ANN/HMM to Speech Variations Stefano Scanzio, Pietro Laface Politecnico di Torino Dario Albesano, Roberto Gemello, and Franco Mana ICASSP 2006

Acoustic Model Adaptation n Adaptation tasks n Linear Input Network n Linear Hidden Network n Catastrophic forgetting n Conservative Training n Results on several adaptation tasks ICASSP 2006 2

Acoustic model adaptation n n Specific speaker Speaking style (spontaneous, regional accents) Audio channel (telephone, cellular, microphone) Environment (car, office, …) Specific vocabulary Voice Application ASR Data Log Task independent models Adapted Models ANN Adaptation ICASSP 2006 3

Linear Input Network adaptation Acoustic phonetic units Emission Probabilities …. Speaker/Task Independent MLP …. …. LIN Speech parameters ICASSP 2006 …. Input layer 4

Linear Hidden Network - LHN Acoustic phonetic units Emission Probabilities …. LHN …. Hidden layer 2 Hidden layer 1 Speech parameters ICASSP 2006 …. …. Input layer 5

Catastrophic forgetting n Acquiring new information can damage previously learned information if the new data that do not adequately represent the knowledge included in the original training data n This effect is evident when adaptation data do not contain examples for a subset of the output classes. n Problem is more severe in ANN framework than in the Gaussian Mixture HMM framework ICASSP 2006 6

Catastrophic forgetting Back-propagation algorithm penalizes classes with no adaptation examples setting their target value to zero for every adaptation frame n Thus, during adaptation, the weights of the ANN will be biased n to favor the activations of the classes with samples in the adaptation set n to weaken the other classes. n ICASSP 2006 7

16 -classes training 6 7 20 x 2 hidden nodes 2 input nodes 16 output nodes 2500 patterns per class The adaptation set includes 5000 patterns belonging only to classes 6 and 7 Error Rate 6 : 1. 5% Total Error Rate: 4. 1% ICASSP 2006 Error Rate 7 : 6. 7% 8

Adaptation of 2 Classes 6 Error Rate 6 : 0% Total Error Rate: 16. 9% ICASSP 2006 7 Error Rate 7 : 2. 0% 9

Conservative Standard target Training assignment target policy assignment policy Posterior probability computed using the original network n 0. 03 0. 00 M 1 P 1 0. 95 1. 00 0. 02 0. 00 P 2 P 3 M 2 P 2 is the class corresponding to the current input frame Px: class in the adaptation set ICASSP 2006 Mx: missing class 10

Adaptation of 2 classes Standard adaptation 6 7 Error Rate 6 : 0% Error Rate 7 : 2% Total Error Rate: 16. 9% ICASSP 2006 Conservative Training adaptation 6 7 Error Rate 6 : 2. 2% Error Rate 7 : 5. 2% Total Error Rate: 10. 2% 11

Adaptation tasks Application data adaptation: Directory Assistance n 9325 Italian city names n 53713 training + 3917 test utterances n Vocabulary adaptation: Command words n 30 command words n 6189 training + 3094 test utterances n Channel-Environment adaptation: Aurora-3 n 2951 training + 654 test utterances n Speaker adaptation: WSJ 0 n 8 speakers, 16 KHz n 40 test + 40 train sentences n ICASSP 2006 12

Results on different tasks (%WER) Adaptation Method Adaptatio Application n Task Directory Assistance Vocabulary Command Words Channel. Environment Aurora-3 CH 1 No adaptation 14. 6 3. 8 24. 0 LIN 11. 2 3. 4 11. 0 LIN + CT 12. 4 3. 4 15. 3 9. 6 2. 1 9. 8 10. 1 2. 3 10. 4 LHN + CT ICASSP 2006 13

Mitigation of Catastrophic Forgetting using Conservative Training Tests using adapted models on Italian continuous speech (% WER) Application Vocabulary Directory Assistance Command Words 36. 3 42. 7 108. 6 Adaptation LIN + CT LHN Method 36. 5 35. 2 42. 1 40. 6 63. 7 152. 1 LHN + CT 40. 7 45. 3 44. 2 Models Adapted on LIN No Adaptation ICASSP 2006 Channel. Environment Aurora-3 CH 1 29. 3 14

Networks used in Speaker Adaptation Task STD (Standard) n 2 hidden layer hybrid MLP-HMM model n 273 input features (39 parameters and 7 context frames) n IMP (Improved) n Uses a wider input window spanning a time context of 25 frames n Includes an additional hidden layer n ICASSP 2006 15

Results on WSJ 0 Speaker Adaptation Task Net type STD IMP ICASSP 2006 Adaptation method Trigram LM No adaptation 8. 4 LIN 7. 9 LIN+CT 7. 1 LHN+CT 6. 6 LIN+LHN+CT 6. 3 No adaptation 6. 5 LHN+CT 5. 6 LIN+LHN+CT 5. 0 16

Conclusions LHN adaptation outperforms LIN adaptation n Linear transformations at different levels produce different positive effects n LIN+LHN performs better than LHN n In adaptation tasks with missing classes, Conservative Training n reduces the catastrophic forgetting effect, preserving the performance on another generic task n improve the performance in speaker adaptation with few available sentences n ICASSP 2006 17