Automatic Content Extraction for Voicemail Using Ninja Steven

Automatic Content Extraction for Voicemail Using Ninja Steven Czerwinski and Barbara Hohlt Goal: Make voicemail more accessible • Enable faster browsing of many voicemails • Access from different devices with different capabilities Our solution: Use Ninja infrastructure • Augment existing Ninja email services to treat voicemails as MIME encoded email • Create Ninja services to process and interpret voicemails Specialized transcoding services • Extracts high level information from voicemails • Includes: audio, skimmed audio, transcript, text/audio summary, and outline

Voicemail Processing Techniques n Speech recognition/synthesis n n n Natural language processing n n n Detecting important words by emphasized pitch Pause n n Directed word spotting by “understanding” content Via. Voice SRCL Pitch n n Transcribe voicemail to text IBM Via. Voice SDK and custom audio libs Compression through pause removal Spurts n Retrieve sentence structure of voicemail

Architecture Client Folder Store Client Transcoder Service • Voicemail->Text Transcript Media Manager Interface • Voicemail->Text Summary Media Manager Service • Voicemail->Text Outline • Email ->Plain Audio Mail Access Interface • Voicemail->Audio Summary Ninja. Mail POP IMAP • Voicemail->Skimmed Audio

System Components n Media manager service n n n Transcoder service n n Ninja i. Space service which transforms data Folder store n n Ninja i. Space service which handles email, voicemail encoded in MIME, and specialized transcoding of voicemail Users can access all their mail across different mail protocols with different types of devices Stores user protocol information Mail access interface n A common interface to generalize access to different mail protocols

Pitch Detection n The idea n n n A speaker’s pitch naturally changes when introducing topics or emphasizing words [hirshberg 92] Use pitch increases as hints for “important” words Algorithm [aaron 95] n n n Determine pitch for each 20 ms frame (FFT with SHS) Set emphasis threshold to be top 1% of pitch values (by histogram) Mark 1 sec interval as emphasized if contains >=3 emphasized frames

Pause Detection n Why is pause detection useful? n Removing pauses speedups playback n n Typically, 50 -70% of original time [foulke 71] Long pauses signify groups (talk spurts) Noise and soft sounds create difficulties Algorithm: smoothed histogram [lamet 81] n n n Calculate energy per 10 ms frame Threshold based on smoothed histogram (5 db after first peak) Use heuristics to remove artifacts Percent of Frames Average energy (d. B)

Examples Original Voicemail: “Hello, This is Barbara. How are you and the cats doing? I was wondering if you would feed them a little more the first time in case they eat too much. My number is (713) 465 -5155. You can call me anytime. Have a very good holiday. Bye bye” Processed Voicemail: (Skimmed) Translated Talk spurts • • Phyllis Barbara Area in the cat staring And then if you run but feed them A little more the first time in case they eat too much On my number is (713) 465 -5155 You can call me anytime. Have every holiday Of light (Just pitch) Translated using NLP • Hello this is Barbara • My number is (713) 465 -5155 (Pitch emphasized words in green)

Results n Pause detection n Pitch detection n n Problems due to high pitch sounds and transitions Speech recognition n n Worked well for given applications Playback speedup by 50 -70% Performance decrease in conversational settings Natural language processing n Performed well with small grammar

Conclusion n Overall n n n What works well n n n Skimming using pause removal Detecting spurts for structure What needs work n n n System useful as navigational hints To achieve total comprehension, need better voice recognition Speech detection in conversational settings Pitch emphasis needs refining Future directions n n Pause detection/word boundaries using speech detection Developing voicemail grammars Using NLP feedback with pitch emphasis detection Improved speech detection in noisy environments