Julias Little Helper A Realtime Demo of CantoneseMandarin

Julia’s Little Helper : A Real-time Demo of Cantonese/Mandarin Emotional Speech Detection William Y. Wang Computer Science Suzanne Yuen Mechanical Engineering CS 6998 Computational Approach to Emotional Speech Instructor: Prof. Julia Hirschberg Columbia University 12/21/2009

Review 1. Target Languages: Cantonese (9 tones) , Mandarin (4 tones) 2. Target Emotions: Anger and Gladness 3. Lexical Features: ASR using a HMM acoustic model trained on Mandarin Broadcast News [1] and a simple hand-written decoding dictionary. 4. Prosodic Features: Energy and Tonal Features 5. Real-time drawing of pitch contour, waveform and energy. 6. A text-to-speech agent to greet and teach user how to use this demo. [1] Yang Shao, Lan Wang, E-Seminar: an Audio-guide e-Learning System, IEEE International Workshop on Education Technology and Training (ETT) 2008.

Lexical Scoring 1 -3 pts Energy 1 pt Tone 1 pt

Dictionary of Affects in Language by Dr. Cynthia Whissell Words Pleasantness Activeness Imagery affect 1. 7500 1. 8571 1. 6 affection 2. 7778 2. 2500 2. 0 success 2. 8571 1. 8000 1. 4 successes 3. 0000 2. 0000 1. 4 Total words: 8742 words were included. Source: It was actually developed using various sources, for example, college student essays, interviews and teenagers description of their own emotion state. So, it can have a broad coverage and avoid biased data.

Sentence Lexical Scoring “I won best paper award!” Words Pleasantness I 2. 3750 won 2. 5556 Best 2. 5455 Paper 1. 2857 Award 2. 8333 Score = (2. 375 + 2. 5556 + 2. 5455 + 1. 2857 + 2. 8333) / 5 = 2. 319

Machine Translation English Word Pleasantness Chinese Translation alcohol 1. 7143 酒精(alcohol) alcoholic 1. 0000 酒精 (alcohol) muscles 2. 3333 肌肉(muscles) muscular 2. 6250 肌肉(muscles) Multilingual Challenges: English Chinese

Encoding and Mapping Mandarin Word Pinyin (Phone Set) Corresponding Cantonese Pinyin Mapping 喜欢(like) xi 3 hua 1 n 喜欢(like) hei 2 fu 1 n 睡觉(sleep) shui 4 jiao 4 训觉(sleep) fen 4 gao 4 Tasks: 1. Mandarin Pinyin (Phone set used by Acoustic Model) 2. Mandarin Cantonese Note that not all words in Mandarin have theirs’ exact and direct mappings in Cantonese words and vice versa. 3. Cantonese Pinyin

Text-to-speech Engine 1. Implement the text-to-speech engine. 2. “Play with” a text-to-speech engine. 3. Engine: Tru. Voice Lernout & Hauspie Speech Products, or L&H Went bankrupt in 2001 technology now owned by Nuance

L&H TTS Functionality • Developed in 1997 • Advanced text pre-processing and no vocabulary restrictions • User-definable pronunciation dictionary • Accurately pronounces surnames and place names • Flexible pitch, volume and speech rate • Intonation support for punctuation

Test Overview 1. Participants – 1. gender: 6 male, 6 female 2. Native Language – 6 Mandarin, 6 Cantonese 2. Two Parts 1. JLH module and self-rating (24 lines total) 2. Perception test – Rating lines from others (72)

Sentences 1. Three types – questions, exclamations, statements 2. Randomized order of sentences for each participant 3. Examples: In Lexicon Not In Lexicon Angry & Not Angry Insane! 痴(chi 1) 线(xian 4)！ You're wrong. 你(ni 3) 错(cuo 4) 了 (le, liao 3, liao 4) 。 Glad & Not Glad When's the wedding? 什(shen 2, shi 2) 么 (ma, me) 时(shi 2) 候 (hou 4) 结(jie 1, jie 2) 婚 (hun 1) ？ It's bubbly? 有(you 3, you 4) 泡 (pao 1, pao 4) 吗 (ma 2, ma 3, ma) ?

Analysis 1. Plan to examine differences and affects of following: 1. Ratings - JLH star rating, self rating, & 3 perception ratings 2. Language – Cantonese, Mandarin 3. Gender – female, male 4. Sentence structure – exclamation, question, & statement 2. Interesting points – 1. Huge range of Chinese accents 2. Tones of words may change depending on previous words (such as English a mug vs. an umbrella) 3. Variations in colloquial speech, addressed by using Chinese script

Future Work 1. Improve the prosodic analysis. More features should be explored. 2. Improve the lexical scoring. Use POS tagger or other NLP tools to weigh different constituents of recognized sentence. 3. Finer-grain the emotion types and investigate the differences. 4. Study translational divergence in English-Chinese MT.

Demo