Improving Mandarin Tone Recognition Based on DNN by

Improving Mandarin Tone Recognition Based on DNN by Combining Acoustic and Articulatory Features Using Extended Recognition Networks Ju Lin & Wei Li & Yingming Gao & Yanlu Xie & Nancy F. Chen Sabato Marco Siniscalchi & Jinsong Zhang &Chin-Hui Lee Presenter: Yu-Sen Cheng

Introduction (1/9) • we investigate the effectiveness of articulatory information for Mandarin tone modeling and recognition in a deep neural network – hidden Markov model (DNNHMM) framework • In conventional approaches, prosodic evidence (e. g. , F 0, duration and energy) is used to build tone classifiers 2

Introduction (2/9) • We here propose performance enhancement techniques in three areas: (i) Adding articulatory features (AFs) and acoustic features, such as MFCCs (Mel frequency cepstrum coefficients), for tone modeling (ii) Adopting phone-dependent tone modeling (iii) Using tone-based extended recognition network (ERN) to reduce the tone search space 3

Introduction (3/9) • Mandarin Chinese is a tonal and syllabic language in which each word is composed of one to several characters with each character pronounced as a basic syllable with five different tones, including four lexical tones (usually called tone 1 -4) and one neutral tone (usually called tone 5) • tone recognition has been a fundamental and nontrivial task for Mandarin speech recognition and pronunciation evaluation tasks • Previous studies have demonstrated satisfactory tone recognition performances for isolated syllables. 4

Introduction (4/9) • Tone recognition in continuous speech is still a challenging task, because the F 0 contour patterns usually deviate from the typical ones due to the complex phonetic and linguistic effects in different contexts. • there are rules in phonetics describing the pitch contour changes of tones due to tone concatenation which are often referred to as sandhi rules • the tone shape of a syllable is affected by the F 0 contours of its neighboring syllables. • the tone level will adjust itself to be consistent with the sentential intonation structure. • In addition, the tone shape and level may also be affected by other factors, such as cross-speakers, variable emphasis, and topic-shift effects 5

Introduction (5/9) • To deal with the complex variations in F 0 contours, many methods have been proposed to alleviate or model the above-mentioned variations • Per speaker or per utterance with moving window normalization (MWN), to alleviate variations caused by cross-speakers, and phrase-level intonations • To deal with variations resulting from tonal co-articulation, have proposed a tone-nucleus model to remove the transition part between adjacent tones • Bi-tone and tri-tone to model contextual information, and the final tone recognition sequence was achieved by dynamic searching 6

Introduction (6/9) • In addition to prosodic features, it was found recently that cepstral features are also useful for tone recognition within a DNN framework. In fact, the correlation between the voice source and the vocal tract has been studied in which indicates that mel-frequency cepstral coefficients (MFCC) might be related to the F 0 contour • Afore-mentioned input features, articulatory information (e. g. , manner and place of articulation) also has an effect on tone perception and realization • Regarding to tone realization, a recent study has shown that different articulatory characteristics will make the F 0 contour varying in its shape or pitch level for the same tone pattern 7

Introduction (7/9) • In this paper, we have investigated three approaches to improving the performance of Mandarin tone modeling and recognition. • Different from, which combines prosodic features with AFs at segment-level, our first approach proposes a framework to combine AFs with acoustic features at frame-level based on DNNHMM, which is capable of modeling the dynamic changes of combined features and wide-range dependencies of observations in phone-independent tone classifiers. • Directly extends to phone-dependent tone modeling so that each modeling unit (e. g. , tonal phone) not only represents tone information, but alsothe phone/articulatory information. • Uses tone-based, extended recognition network(ERN) to restrict the tone search space. 8

Introduction (8/9) • Compared with techniques, which adopted a two-step tone recognition approach (first using a forced-alignment technique to get syllable boundary information, and then extracting segmental features for tone classification), the contribution in our study is to investigate a DNN-based one-step approach for Mandarin tone recognition. 9

Introduction (9/9) • The method proposed in the present study outweighs previous ones in three aspects: • (1) Boundary information and segmental-level features are no longer required, instead tone characteristics are represented by a sequence of frame-level features • (2) DNN-HMM is used to model dynamic changes of frame-level features, in contrast to conventional two-step method, simply averaging frame-level features as the segmental-level feature. Our experimental results show that the frame-level dynamic changes could provide additional information for improving the performance of Mandarin tone recognition • (3) Modeling units are extended to phone-dependent tone units so that input features (e. g. , prosodic features and AFs) and modeling unit itself both contain phone/articulatory information 10

Adding Articulatory Features for Phone-independent Tone Modeling and Recognition(1/5) • Different articulatory characteristics of initial and finals may make the pitch contour different in the shape or register level • Averaged F 0 contours of the four tones of the syllables with unvoiced initial /b/ are higher than the syllables with voiced initial ‘m’ under the same final condition • Articulatory manner of vowels in finals also influences the F 0 values 11

Adding Articulatory Features for Phone-independent Tone Modeling and Recognition(2/5) • Both the Initials and the Finals can be further divided into a series of detailed categories based on articulatory movements such as manner of articulation, place of articulation • We added a SIL symbol to deal with regions of silence • The Initials are divided into 4 categories • Finals can be divided into a total 15 categories 12

Adding Articulatory Features for Phone-independent Tone Modeling and Recognition(3/5) • Manual AF annotations of speech signals are rather difficult and costly to produce, one reasonable way of generating training material for the articulatory classifiers is to convert phone-based training transcriptions to articulatory feature transcriptions 13

Adding Articulatory Features for Phone-independent Tone Modeling and Recognition(4/5) • There are five lexical tones in Mandarin speech, but no tone pattern for silence and unvoiced regions • We trained a tone DNN-HMM model with six targets: five tones and a no-tone target • The DNN-HMM classifier is an extended tone classifier because it not only classifies five tones, but it also accomplishes voicing detection (for the no-tone decision). 14

Adding Articulatory Features for Phone-independent Tone Modeling and Recognition(5/5) • To alleviate variations caused by factors mentioned in introduction, like crossspeakers, we used several feature processing • First, the MFCC and F 0 features were spliced in time taking the 3 preceding frames, the current frame and the 3 succeeding frames, followed by de-correlation and dimensionality reduction to 40 using linear discriminant analysis (LDA) • Second, the posterior probabilities of the articulatory categories were obtained from the trained articulatory DNN classifier 15

Phone-dependent Tone Modeling and ERN-based Tone Recognition (1/2) • Extend phone-independent tonal models to phone-dependent tonal models • Each modeling unit (e. g. , tonal phone) not only represents tone information, but also the phone/articulatory information • Free grammar tone recognition loop is replaced by tone-based extended recognition network (ERN) to constrain the tone searching space 16

Phone-dependent Tone Modeling and ERN-based Tone Recognition (2/2) • Each toneless syllable is extended into five different tonal syllables, which are also referred to as tone-based ERN. And then Viterbi decoding is carried out to find the best tonal syllable sequence. 17

Phone-dependent Tonal Models Training and Recognition with Tone-based ERN (1/2) • Chinese syllable has a well-defined initial and final structure, so previous studies have used initial and tonal final as acoustic units to recognize tonal syllable or word sequence • We focus on using toneme as acoustic units. The feature vector consists of 13 MFCCs in conjunction with F 0, the probability of voicing (POV) 18

Phone-dependent Tone Modeling and ERN-based Tone Recognition (1/2) 19

Data Set • Chinese National Hi-Tech Project 863 for Mandarin large vocabulary continuous speech recognition (LVCSR) system development was used for all the experiments • Data covering 74 female speakers were employed for training the articulatory DNN model with a portion (10%) of the training data withheld as a cross validation set 20

Results and Discussion(1/2) 21

Results and Discussion(2/2) DNN-C DNN-F 22

Conclusion (1/2) • Study the effectiveness of articulatory information for Mandarin tone modeling and recognition in a DNN-HMM framework • The system with MFCC and F 0 features achieves a higher tone recognition rate than the baseline system with only prosodic features • DNN model may be able to extract more useful information from the MFCC parameters for tone recognition 23

Conclusion (2/2) • All of the systems incorporating the articulatory information can further improve the tone recognition performance • Articulatory information may not only provides the pitch contours that are influenced by the articulatory characteristics, but also provide syllable boundary information of tones or no-tones to reduce the insertion or deletion errors in tone recognition • Feedbacks about the tonal errors are essential advantages in a tonal language learning system 24

Q&A Thanks for listening