Subjective Intelligibility Assessment Dr Herman J M Steeneken
Subjective Intelligibility Assessment Dr. Herman J. M. Steeneken Acousteen Herman J. M. Steeneken
Signal-to-Noise ratio !!! Acousteen Herman J. M. Steeneken 2
Research Questions • Intelligibility versus Quality assessment • Evaluation of a system or application • Ranking of the performance of a number of systems • Diagnostic assessment • Prediction of system performance during design Acousteen Herman J. M. Steeneken 3
Assessment Methods • Subjective assessment with subjects (speakers and listeners): representative, limited reproduction, non diagnostic, laborious • Objective assessment based on physical properties (measurements): reproducible, diagnostic, fast • Prediction of system performance: design tool Acousteen Herman J. M. Steeneken 4
SUBJECTIVE INTELLIGIBILITY Acousteen Herman J. M. Steeneken 5
Subjective Intelligibility methods • Phoneme level (nonsense words, rhyme words, consonants, vowels) • Word level (meaningful words, nonsense words, phonetically balanced PB, equally balanced Eqb) • Sentence level (Mean Opinion Score MOS, Speech Reception Threshold SRT) Acousteen Herman J. M. Steeneken 6
Methodology I Response categories: • Open response (e. g. , nonsense words) • Closed response (Rhyme tests, e. g. , MRT, DRT) • Scaling (MOS, five point scale: excellent - bad) • Ranking (e. g. , pair-wise comparison) Acousteen Herman J. M. Steeneken 7
Methodology II Test design: • Words embedded in carrier phrase • Reference conditions (e. g. MNRU, …) • Speakers (gender, number, non-native, …) • Listeners ( number of speaker-listener pairs) • Learning effects Acousteen Herman J. M. Steeneken 8
Listening test with four subjects Acousteen Herman J. M. Steeneken 9
Acousteen Herman J. M. Steeneken 10
Embedded CVC words: versta des over en nu fijs uit het woord zek einde noteer lal punt “Semi random”combination of: 17 initial consonants 15 vowels 11 final consonants Acousteen Herman J. M. Steeneken 11
Methodology III Scoring, data analysis: • Phone-word scores • Confusion matrices • Effective gain (e. g. effective SNR) • Statistics (Anova, scaling, multiple regression, . . . ) Acousteen Herman J. M. Steeneken 12
Relation Consonants-Vowels Acousteen Herman J. M. Steeneken 13
How to calculate average word scores Subject responses may require to use the median Acousteen Herman J. M. Steeneken 14
Example relation MOS-CVC Acousteen Herman J. M. Steeneken 15
Relation between methods and Qualification Acousteen Herman J. M. Steeneken 16
Test-retest variability Cronbach α based on split of speaker- listener pairs Acousteen Herman J. M. Steeneken 17
Common Intelligibility scale (IEC 60849) After Barnett and Knight 1994 CIS not linear with SNR = STI = 100 - ALcons x = AI = PB words (256 words) = Short Sentences = PB words (1000 words) = 1000 syllables Barnett and Knight (1995) Acousteen Herman J. M. Steeneken 18
CVC scores (%) of realistic conditions male female Wide band 90. 3 89. 3 Telephone band 89. 5 85. 3 White noise SNR 0 d. B 58. 0 44. 1 Speech noise SNR +3 d. B 71. 3 60. 7 Speech noise SNR -3 d. B 43. 0 40. 6 Acousteen Herman J. M. Steeneken 19
Example of consonant confusions p b f v m n R w p 1068 62 12 4 4 0 0 2 b 112 1002 0 0 11 7 0 50 f 44 1 915 193 0 0 v 6 4 337 739 0 0 2 43 m 1 5 0 0 1068 113 1 6 n 0 0 111 1081 0 2 R 1 2 0 2 1161 3 w 6 3 1 13 30 7 25 1065 Acousteen Herman J. M. Steeneken 20
Two dimensional display of confusions Acousteen Herman J. M. Steeneken 21
Introduction of phoneme specific frequency weighting Four groups of phonemes (SAMPA notation: • Fricatives (f, s, v, z) • Plosives (b, d, x, p, t, k) • Vowel-like consonants (m, n, l, R, j, w, …) • Vowels (aa, a, ee, e, o, oo, u, uu, au, …) Acousteen Herman J. M. Steeneken 22
Phoneme group specific spectra Acousteen Herman J. M. Steeneken 23
Phoneme group specific spectra Acousteen Herman J. M. Steeneken 24
Frequency weighting (fricatives) Acousteen Herman J. M. Steeneken 25
Frequency weighting (plosives) Acousteen Herman J. M. Steeneken 26
Frequency weighting (vowel-like cons) Acousteen Herman J. M. Steeneken 27
Frequency weightings (vowels) Acousteen Herman J. M. Steeneken 28
Frequency weightings (CVC words) Acousteen Herman J. M. Steeneken 29
Prediction of (CVC) word score by a weighted combination of phoneme group probabilities (DUTCH) Ci = 0. 294 fric + 0. 294 plo + 0. 412 Cvo V = V (no weighting) Cf = 0. 273 fric + 0. 273 plo + 0. 454 Cvo CVC score = Ci * V * Cf * 100 % Acousteen Herman J. M. Steeneken 30
CVC-word prediction (male) S. d. = 4. 11% Male speech Acousteen Herman J. M. Steeneken 31
CVC-word prediction (female) S. d. = 3. 63% Female speech Acousteen Herman J. M. Steeneken 32
ISO: Ergonomics – Assessment of speech communication (ISO 9921 DIS) Acousteen Herman J. M. Steeneken 33
Qualification table Acousteen Herman J. M. Steeneken 34
- Slides: 34