Good Morning How many times have you tried

Good Morning! • How many times have you tried to get ahold of someone and failed? • Need to keep trying • Need to find alternate ways • How many times have you forgotten an important appointment?

With VERA you can… • Schedule wake-ups for yourself and others • Schedule message delivery to yourself and others • Schedule recurring calls • Send “urgent” messages with return receipt • Automatically try multiple contacts for a person • Automatically try again later

Voice Enabled Reminding Agent Vera-OUT: Ulaş Bardak, Scott Judy, Vasco Pedro Vera-IN: Tal Blum, Jeongwoo Ko, Ryosuke Miyata

Overview • Overall Architecture • Vera. OUT Architecture • Skyper • Dialog Manager Internals • Web interface • Demo • Pros and Cons • Future Work

Overall Vera Architecture *Vera-IN by Tal Blum, Jeongwoo Ko, and Ryosuke Miyata PSTN Vera-IN* Skype Switching Center Web Interface DB Internet Vo. IP Contacts **Vera-OUT by Ulas Bardak, Scott Judy, and Vasco Pedro Vera-OUT** Skype

Vera Server Audio Monitor Speakers for Synthesized Speech Cepstral Speech Synthesizer Sound Card #1 (Windows Default Audio) Sphinx Speech Recongizer Internet Sound Card #2 (Skype Audio) Out Mic Out Vo. IP Skype Software Monitor Speakers for User Speech

Vera. OUT Building Blocks Vera. TM Web Interface Vera. DM Skyper Other Components Vera-IN DB Vera-OUT Backend DM System Vo. IP

Vera. OUT Building Blocks Vera. TM Web Interface Vera. DM Skyper Other Components Vera-IN DB Vera-OUT Backend DM System Vo. IP

Dialog Management Internals Recognition SPHINX Lang. Understand. PHOENIX/HELIOS Dialog Manag. Raven. Claw Synthesis THETA Vera Backend Lang. Generation ROSETTA From Dan Bohus’ talk on Ravenclaw

Dialog Management Internals Recognition SPHINX Lang. Understand. PHOENIX/HELIOS HUB Synthesis THETA Dialog Manag. Raven. Claw Vera Backend Lang. Generation ROSETTA From Dan Bohus’ talk on Ravenclaw

Dialog Management Internals Multiple, parallel decoders SPHINX Date. Time Recognition Server Lang. Understand. Parsing Confidence PHOENIX/HELIOS PHOENIX HELIOS Text I/O TTYServer HUB Synthesis THETA Lang. Generation ROSETTA Skyper Dialog Manag. Raven. Claw Task Manager Vera Backend Vera. TM DB Access PROCESS MONITOR From Dan Bohus’ talk on Ravenclaw

Vera-OUT Architecture Vera. TM Web Interface Vera. DM Skyper Other Components Vera-IN DB Vera-OUT Backend DM System Vo. IP

Database

Vera-OUT Building Blocks Vera. TM Web Interface Vera. DM Skyper Other Components Vera-IN DB Vera-OUT Backend DM System Vo. IP

Web Interface

VERA-OUT Demo We showed a demo at this point in the presentation. Here are some recorded sample conversations:

Raven. Claw Agent Tree A: Vera. Out E: Get. Context E: Check. Task. ID R: Wait. For. Speech A: Is. User R: Speak. With. User A: Wrong. User A: Get. User R: Wait. For. User I: Call. Later E: Call. Next A: Right. User A: Do. Task R: Logout E: Set. Completed R: get. User. String E: set. Suspend I: Message. Not. Delivered I: Message. Delivered I: Wake. Up I: Deliver. Message I: Wake. Up. System E: set. Right. User I: Inform. Message E: set. Wrong. User E: set. Not. Here A: message. Handling I: Play. Message E: Set. Message. Played R: get. Task. Done. String I: Repeat. Message

VERA-Out User Tests • A user pool of around 10 fully cooperating users • Used a relatively simpler dialogue • Problems: • Non-native speakers have a lot of trouble • Background noise and booming on the phone can affect the system • Speakers get very confused when the system repeats itself with no clarification.

Vera-Out User Test Outcome • Increased the dialog cooperation • Clarification on what’s being asked for • Stating what is expected • Added more coverage recognition grammar. • Added more coverage manager. for speech for dialog

VERA-OUT Advantages • • Skype is inxexpensive Skype has worldwide reach. No phone line or special equipment required (except for an extra sound card). Privacy is preserved • • • No numbers are shared with other users. No calls made to a number once the user removes it. Ability to call someone at a time when you may not be free yourself. Persistence Ability to confirm delivery of messages. Support for recurring tasks

VERA-OUT Issues • • • Skype does cost money when phones are called. What’s the best way to do accounting / charge the right user? Skype calls show up as “unknown” on caller ID. Being socially appropriate: • • Where’s the line between “trying hard enough to make contact” and “harassment”? How late/early should VERA call? (time zones? ) Can others override this in an emergency? Who decides what an emergency is?

VERA-OUT Issues (ctd. ) • • Should users be able to cancel all the calls that others have set up for them? Handling failure to make contact. • • • Calling other contacts. Calling back later (after 1 hour) More work needed. Answering machines How to handle someone who’s not expecting a call from a computer

VERA-OUT Future Work • • Better dialog to confirm user identity Blocked users / Allowed users Calls to multiple users Getting user contacts from the Skype directory • Ability to forward and reply to messages.

VERA-OUT Future Work (ctd. ) • Setting up meetings with multiple participants. 1. Call VERA-IN or use the web interface to specify available times and target meeting participants. 2. VERA-OUT calls everyone and finds common times. 3. VERA-OUT gets final approval from the initiator. 4. VERA-OUT calls Roomline to book a room? ? 5. VERA-OUT informs everyone of the result.

VERA-OUT Conclusion • With Vera, reach the people you need to reach, when you want to reach them!

Special Thanks! • Our professors: • Alan Black • Alex Rudnicky • Raven. Claw experts: • Dan Bohus • Jahanzeb Sherwani • Antoine Raux • Our longsuffering office-mates: • Yee-Man (Betty) Cheng • Yanjun Qi

VERA-OUT The End