Outline What is fluent speech prosody Tones and

The HPG framework—Speech Paragraph (context of cross-over & adjacency) (Tseng 2005) Discourse PG PG

Prosodic Units and Boundaries in the Framework B 5 Prosodic Group B 4 Breath

Fujisaki Model (Fujisaki 1984) Aug 14, 2008 Ne. GSST 2008 Zhao-yu Su

Auto-extraction based on Mixdorff’s method (2000, 2003) Original F 0 contour highpass filter (stop

The decision of commands Low-frequency contour (LFC) from Mixdorff’s filter B 3 optimization B

The results of auto-extraction based on Mixdorff’s method in Mandarin. Phonetics Lab, Academia Sinica

Model additional component of F 0 contour by HPG Tone (SYL) & above (PW)

Examples without PG-effect with PG-effect One expected cell mean can’t approach LFC well, PG-initial

Layered Contributions Linear regression 1. 2. 3. 4. Predict syllable duration by SYL category

Speech material Two types of Mandarin speech corpus Read speech of (1. ) plain

Aa prediction & Tone Model Cumulative accuracy of Aa prediction Corpus CL CNA Speaker

Boundary effect above PPh for Aa prediction Cumulative accuracy of Aa prediction Corpus CL

PW model Aug 14, 2008 Ne. GSST 2008 Zhao-yu Su

Ap & Aa Cumulative Accuracy Ap cumulative accuracy Corpus CL CNA Speaker PPh BG

PG Model for Ap & average duration An example of average Ap by PG-position

Significant test of PPh features Comparison of Ap by pairs of PG positions initial/medial,

Classification of Stylistic Variations regular semi-regular style of writing Index Name of files 300

Relationship between prosodic styles & HPG contribution distribution Higher level contribution by prosody style

Layered Contributions of Duration, Intensity and Boundary Pause (Tseng, 2004, 2005) Duration, Pause and

Layered Contributions of Duration, Intensity and Boundary Pause (Tseng, 2004, 2005) Duration Pattern Intensity

Comparison of Cumulative Predictions and Speech Data Comparision between speech data and predictions for

Previous and Revised Models -Duration Patterns at PPh Layer F 051 P M 051

Previous and Revised Models -Intensity Patterns at PPh Layer F 051 P M 051

Discourse boundary discrimination • Boundary properties and respective discourse identities – Only from pause

Experiment 1 Q: Examine if only syllable domain is helpful to discriminate discourse boundary

Results of Experiment 1 (1/2) Cross boundary discrimination by single acoustic features. Each panel-acoustic

Results of Experiment 1 (2/2) Cross boundary discrimination by single contrastive factors. Each panel-contrastive

Experiment 2 Q: Examine the scale of boundary context to discriminate discourse boundary identities

Results of Experiment 2 (1/2)-- Lengthening Patterns by Discourse Units Cross boundary comparison of

Results of Experiment 2 (2/2)—Lengthening Patterns by Discourse Units Cross-boundary duration patter by boundary

Conclusions of Fluent Narrative Speech • Prosody and prosody context ü • Tone and

Slides: 43

Download presentation

Outline • What is fluent speech prosody? Tones and Intonation? • Why the HPG framework (Tseng, 2004— 2008)? • How to decompose F 0 contours? • • • The nature of the Fujisaki model Auto-extracting of the Fujisaki parameters Calculating layered contribution by the HPG Modeling tones, intonation and additional components Investigating prosodic style variation • Further evidence of the HPG framework • Domains and units of boundaries (acoustic features) in fluent speech • Pause or No Pause for boundaries ? • Boundary effects and the relative-ness of supra-segmental signals Aug 14, 2008 Ne. GSST 2008 Zhao-yu Su

The HPG framework—Speech Paragraph (context of cross-over & adjacency) (Tseng 2005) Discourse PG PG BG PPh PW SYL DM/PF PW SYL Cross-over PPhs PW SYL PPh# PW SYL adjacency PPh# PPh DM/PF PW SYL BG BG PW SYL PPh# PPh DM/PF PW SYL Prosodic units Syllables (SYL), Prosodic Words (PW), Prosodic Phrase (PPh), Breath Groups (BG), Prosodic phrase Groups (PG) and corresponding Boundary Breaks B 1, B 2, B 3, B 4 and B 5 where SYL /B 1< PW /B 2< PPh /B 3< BG /B 4< PG /B 5 Output Prosody Is Super-Positional and Cumulative (Tseng et al, 2004, 2005, 2006) Aug 14, 2008 Ne. GSST 2008 Zhao-yu Su

Prosodic Units and Boundaries in the Framework B 5 Prosodic Group B 4 Breath Group PG-Initial PG-Medial PG-Final B 3 PW PW. . B 2 B 2 Aug 14, 2008 . . B 3. . . . B 2 B 2 Ne. GSST 2008 . . . PW B 2 B 2 Zhao-yu Su

Fujisaki Model (Fujisaki 1984) Aug 14, 2008 Ne. GSST 2008 Zhao-yu Su

Auto-extraction based on Mixdorff’s method (2000, 2003) Original F 0 contour highpass filter (stop frequency at 0. 5 Hz) Low-frequency contour (LFC) Aug 14, 2008 High-frequency contour (HFC) Ne. GSST 2008 Zhao-yu Su

The decision of commands Low-frequency contour (LFC) from Mixdorff’s filter B 3 optimization B 3 Optimization criteria: Min Aug 14, 2008 Ne. GSST 2008 Zhao-yu Su

The results of auto-extraction based on Mixdorff’s method in Mandarin. Phonetics Lab, Academia Sinica Aug 14, 2008 Ne. GSST 2008 Zhao-yu Su

Model additional component of F 0 contour by HPG Tone (SYL) & above (PW) PW information PWmodel Single tone SYL Tone 1 Tone 3 PW Final output of F 0 contour by HPG Prosodic phrase (PPh) & above (BG&PG) PG information PPh Single phrase model PG model (Multiple phrase) BG PG Aug 14, 2008 Ne. GSST 2008 Zhao-yu Su

Examples without PG-effect with PG-effect One expected cell mean can’t approach LFC well, PG-initial and PG-final especially. Aug 14, 2008 Ne. GSST 2008 Zhao-yu Su

Model additional component of F 0 contour by HPG Tone (SYL) & above (PW) Prosodic phrase (PPh) & above (BG&PG) Aug 14, 2008 Ne. GSST 2008 Zhao-yu Su

Layered Contributions Linear regression 1. 2. 3. 4. Predict syllable duration by SYL category from the bottom of HPG Error between prediction and real value is regarded as the effect of PW instead of unpredicted variation. The same predictions are repeated at each HPG layer from the SYL upward to PG Final output is the sum of prediction in each prosodic layer and prediction accuracy of each layer is regarded as the layered Contributions BG BG Residues PPh Residues PW SYL Aug 14, 2008 PPh PW SYL PW PW SYL Ne. GSST 2008 SYL Residues Zhao-yu Su

Speech material Two types of Mandarin speech corpus Read speech of (1. ) plain text of 26 discourse pieces by one male M 051 and one female F 051 (2. ) three rhyme formats of Chinese Classics by one male M 056 and one female F 054 Pre-analysis annotation 1. Automatically labeled segmental identities with HTK toolkit 2. Subsequent manual tagging with Sinica COSPRO Toolkit 3. Spot-checking for annotated segments Table 1 Summary of speech data by corpus type corpus CNA CL Aug 14, 2008 speaker SYL/B 1 PW/B 2 PPh/B 3 BG/B 4 PG/B 5 F 051 6583 3468 1092 297 151 M 051 6661 3332 1207 270 129 F 054 1444 599 290 135 58 M 056 1551 619 318 142 47 Ne. GSST 2008 Zhao-yu Su

Aa prediction & Tone Model Cumulative accuracy of Aa prediction Corpus CL CNA Speaker Syl Layer Contribution PW Layer Contribution Tone Context PW Boundary Infor PW Position Sequence F 054 46. 21% 54. 74% 60. 54% 66. 61% M 056 39. 12% 47. 86% 57. 68% 61. 45% F 051 38. 40% 45. 00% 48. 43% 51. 27% M 051 41. 61% 47. 96% 51. 33% 54. 53% Tone Model Aug 14, 2008 Ne. GSST 2008 Zhao-yu Su

Boundary effect above PPh for Aa prediction Cumulative accuracy of Aa prediction Corpus CL CNA Aug 14, 2008 Boundary effect above PPh Boundary Infor PG Bounary Infor Contribution of Boundary effect F 054 72. 98% 73. 80% 7. 19% M 056 64. 13% 66. 89% 5. 43% F 051 54. 41% 56. 25% 4. 98% M 051 57. 43% 59. 32% 4. 79% Speaker Ne. GSST 2008 Zhao-yu Su

PW model Aug 14, 2008 Ne. GSST 2008 Zhao-yu Su

Ap & Aa Cumulative Accuracy Ap cumulative accuracy Corpus CL CNA Speaker PPh BG PG f 054 58. 79% 63. 58% 76. 66% m 056 37. 89% 48. 99% 73. 66% F 051 80. 17% 81. 46% 87. 71% m 051 81. 53% 82. 72% 88. 20% Average of Aa and Ap predictions were used as the final accuracy of total F 0 contour prediction Final accuracy of total F 0 contour prediction Corpus CL CNA Aug 14, 2008 Speaker Aa Ap Total f 054 76. 66% 73. 80% 75. 23% m 056 73. 66% 66. 89% 70. 28% F 051 87. 71% 56. 25% 71. 98% m 051 88. 20% 59. 32% 73. 76% Ne. GSST 2008 Zhao-yu Su

PG Model for Ap & average duration An example of average Ap by PG-position of two adjacent PG’s by speaker and by speech data type. The horizontal axis represents the PG-position index. The vertical axis represents the average Ap values. The tempo of the same examples used in Figure 1 is plotted by speaker and by speech data type. The horizontal axis represents the PG-position index. The vertical axis represents the mean syllable duration values. Aug 14, 2008 Ne. GSST 2008 Zhao-yu Su

Significant test of PPh features Comparison of Ap by pairs of PG positions initial/medial, initial/final and medial/final and by speaker, the asterisk ＊ denotes statistically significant differences. Comparison of mean syllable duration by pairs of PG positions initial/medial, initial/final and medial/final and by speaker, the asterisk ＊ denotes statistically significant differences. Aug 14, 2008 Ne. GSST 2008 Zhao-yu Su

Classification of Stylistic Variations regular semi-regular style of writing Index Name of files 300 五言古詩） 07 a 正氣歌 300 五言古詩）慈烏夜啼 90 五言古詩） 17 a 慈烏夜啼 90 五言古詩） 18 a 古詩十九首之一 80 五言古詩） 19 a 長干行 150 整樂府） 21 a 古詩十九首之二 50 五言古詩） 22 a 石壕吏 120 整樂府） 23 a 古詩十九首之九 40 五言古詩） 24 a 新婚別 160 整樂府） 26 a 古詩十九首之十五 50 五言古詩） 27 a 春江花月夜 262 整樂府） Index Name of files # of syllables style of writing 01 a 禮運大同篇 107 古文） 02 a 典論論文 187 古文） 03 a 孟子齊人 202 古文）雜說四 150 古文） Zhao-yu Su Index Name of files 07 a 正氣歌 17 a # of syllables style of writing irregular Aug 14, 2008 06 a Ne. GSST 2008

Relationship between prosodic styles & HPG contribution distribution Higher level contribution by prosody style as shown from R, SMR, IR to WIR 1. Various prosodic styles can be explained by HPG framework systematically. 2. The more regular the prosodic style, the larger the prosodic domain, and more contribution from higher level information. Aug 14, 2008 Ne. GSST 2008 Zhao-yu Su

Layered Contributions of Duration, Intensity and Boundary Pause (Tseng, 2004, 2005) Duration, Pause and Intensity patterns of PW and PPh layer in speaker F 051 P Aug 14, 2008 Ne. GSST 2008 Zhao-yu Su

Layered Contributions of Duration, Intensity and Boundary Pause (Tseng, 2004, 2005) Duration Pattern Intensity Pattern Pause Pattern BG layer Duration, Pause and Intensity patterns of PW and PPh layer in speaker F 051 P Aug 14, 2008 Ne. GSST 2008 Zhao-yu Su

Comparison of Cumulative Predictions and Speech Data Comparision between speech data and predictions for M 051 Aug 14, 2008 Ne. GSST 2008 Zhao-yu Su

Previous and Revised Models -Duration Patterns at PPh Layer F 051 P M 051 P Previous Revised • The revised PPh patterns show the general pattern derived from the revised model is more contrastive than earlier patterns shown in the previous PPh patterns Aug 14, 2008 Ne. GSST 2008 Zhao-yu Su

Previous and Revised Models -Intensity Patterns at PPh Layer F 051 P M 051 P Previous Revised • The PPh patterns from the revised model decayed more drastically towards boundary, thus match the tendency of the intensity attenuation for PPh final weakening, especially for M 051 P Aug 14, 2008 Ne. GSST 2008 Zhao-yu Su

Discourse boundary discrimination • Boundary properties and respective discourse identities – Only from pause duration? – Pre-boundary syllable lengthening ? (in other intonation study) – Relative acoustic feature? • Boundary discrimination – Detect topic & discourse organization – Discourse prosody context – Cross-over & adjacency Aug 14, 2008 Ne. GSST 2008 Zhao-yu Su

Experiment 1 Q: Examine if only syllable domain is helpful to discriminate discourse boundary identities Goal: Whether singular/relative acoustic factor in syllable layer is sufficient to differentiate B 3, B 4 & B 5 Acoustic features: Singular acoustic factor (1. ) boundary pause (BP), (2. ) pre-boundary syllable duration (Pr. Du) and (3. ) preboundary syllable intensity (Pr. In) Relative acoustic factor (4. ) between-boundary syllable duration contrast (Du. Con) and (5. ) between-boundary syllable intensity contrast (In. Con) Aug 14, 2008 Ne. GSST 2008 Zhao-yu Su

Results of Experiment 1 (1/2) Cross boundary discrimination by single acoustic features. Each panel-acoustic feature. The horizontal axis -the prosodic boundary indexes. The vertical axis -the coefficient of normalized values of boundary pause (BP), per-boundary duration (Pr. Du) and per-boundary intensity (Pr. In), respectively. Aug 14, 2008 Ne. GSST 2008 Zhao-yu Su

Results of Experiment 1 (2/2) Cross boundary discrimination by single contrastive factors. Each panel-contrastive feature. The horizontal axis – prosodic boundary indexes. The vertical axis-the coefficient of normalized values between boundary duration contrasts (Du. Con) and between boundary intensity contrasts (In. Con). Aug 14, 2008 Ne. GSST 2008 Zhao-yu Su

Experiment 2 Q: Examine the scale of boundary context to discriminate discourse boundary identities Goal : How to account for boundary context in the acoustic signals by discourse specifications • Acoustic features Average duration of prosodic units by different scale -Syllable -Prosodic word -Prosodic phrase Aug 14, 2008 Ne. GSST 2008 Zhao-yu Su

Results of Experiment 2 (1/2)-- Lengthening Patterns by Discourse Units Cross boundary comparison of duration patterns by prosodic units the syllable (SYL), the PW and the PPh. The horizontal axis represents indexes of the speech data and speaker. The vertical axis denotes normalized average duration of prosodic units. Aug 14, 2008 Ne. GSST 2008 Zhao-yu Su

Results of Experiment 2 (2/2)—Lengthening Patterns by Discourse Units Cross-boundary duration patter by boundary breaks. The panel denotes result of specific prosodic unit. Each curve denotes one of speech data. The horizontal-axis represents prosodic boundary index. The vertical-axis denotes the normalized average duration for specific prosodic unit. Aug 14, 2008 Ne. GSST 2008 Zhao-yu Su

Conclusions of Fluent Narrative Speech • Prosody and prosody context ü • Tone and intonation variation can be explained by HPG framework systematically ü ü • Tone variation-Lower level by HPG Various prosodic styles-Higher level by HPG Prosody context boundaries across fluent speech – • More information beyond tones and intonation adjacency and cross-over associations discourse specified Possible application to ASR ü topic & discourse organization detection Aug 14, 2008 Ne. GSST 2008 Zhao-yu Su

Related presentations