A Flexible VXML Voice XML Interpreter Yevgen Borodin

A Flexible VXML (Voice XML) Interpreter Yevgen Borodin Computer Science Department, Stony Brook University Motivation Ø Inadequacy of screen-readers in Web browsing Ø Development of interface Manager for Hear. Say voice-browser Ø Absence of fully-implemented open-source VXML interpreters Ø Need for specialization in Web browsing applications Ø Need for a multiplatform, extensible, modular, flexible system Ø VXML is typically used in telephony applications Ø Computer games use VXML to program interactive dialogs Ø VXML dialogs disseminate information through public terminals Ø Voice. XML can be used in voice browsing! Variables | Files | Return | user=“Yevgen” session=“browse” … title=“circuit city” mode=“form. Fill” … fname=“Yevgen” lname=“Borodin” zip=“ 11790” … Cached Files main. vxml all. Links. vxml history. vxml favorites. vxml key. List. vxml commands. vxml … Ø Compliant with Voice. XML 2. 0 specifications Ø Geared to Web-browsing as opposed to telephony Ø Modular, Extensible, Multi-Platform (Java) Ø Extends Voice. XML 2. 0 to give more control over dialog flow Ø Loaded with add-ons: Spell Check, TTS, SR, etc. fname=“Yevgen” lname=“Borodin” zip=“ 11790” … … <rule id="phone_number"> <item repeat="0 -1">d</item> <tag><![CDATA[$. phone="ddd-dddd"; ]]></tag> </item> </rule> … Big Picture: Hear. Say Browser VXML Interpreter Future Work Ø Complete Voice. XML 2. 0 Specifications Ø Messaging between VXMLSurfer and calling application Ø Speech Recognition (CMU Sphinx) Ø Java Script Interpreter and Grammar Interpreter Modules Ø Multilingual TTS, etc. Mozilla Engine INS + F 1 Events <catch event="Insert+F 1 help"> <prompt> You are at Circuit City check out </prompt> </catch> SGRS Grammar Features of VXMLSurfer Ø VXMLSurfer is the interface of the Hear. Say voice-browser Ø Users interact with VXMLSurfer through keyboard and mic Ø HTTP request is forwarded to the Mozilla browser engine Ø The loaded Web page is converted into a frame tree Ø The frame tree is processed and converted to VXML dialogs Ø Voice. XML dialog files are sent to the interpreter for processing Ø Variables are returned to the calling application (Hear. Say) Ø Hear. Say invokes form-filling module to fill and submit the form <? xml version='1. 0'? > <vxml> … <form id=‘shipping'> <field name="fname"> <prompt>Enter first name</prompt> <grammar src="customer. xml#fname“ type="application/grammar+xml"/> </field> <field name="lname"> <prompt>Enter last name</prompt> <grammar src="customer. xml#lname" type="application/grammar+xml"/> </field> … </form> </vxml> Uses of Voice. XML Application of VXMLSurfer Back-End Processing VXMLSurfer in Action HELP This material is based upon work supported by the National Science Foundation - Awards: IIS-0534419, CNS-0751083, IIS-0808678 and National Institute on Disability and Rehabilitation Research (NIDRR) - Award: H 133 S 090065. <catch event=“Ctrl+S submit"> <submit namelist=“fname lname …”> </catch> …