Spoken Dialogue and Information Retrieval Antoine Raux Dialogs

(Spoken) Dialogue and Information Retrieval Antoine Raux Dialogs on Dialogs Group 10/24/2003

Outline n n n Interactive Information Retrieval Systems (Belkin et al) EUREKA: Dialogue-based IR for Low Bandwidth Devices Voice Access to IR

Cases, Scripts, and Information. Seeking Strategies n Belkin, Cool (Rutgers) Stein, Thiel (GMD-IPSI) n Long journal article (1995) n From the IR community (Expert Systems)

IR as Interaction n Traditional IR research focuses on document/query representation and comparison Need to focus on the user Represent IR as a dialogue between an information seeker and an information provider

Information-Seeking Strategies n Represent information-seeking behavior along 4 dimensions: Method of Interaction (scanning vs searching) n Goal of Interaction (learning vs selecting) n Mode of Retrieval (recognition vs specification) n Resource Considered (information vs meta-info) n n Binary values 16 strategies (ISS)

Dialogue Structures for Information Seeking n Mix of different formalisms: Recursive state-based schemas (COR) e. g. Request Promise Inform Be contented n Scripts: prototypical interaction for each ISS n Goal trees Retrieve Specified Items n Specify Characteristic Offer choice Recognize Desired Items Select and Specify

Deriving Scripts from Data n n Case-based approach: problem solving using previously stored solved instances Match a sequence of action to a statebased schema Extract goal tree Identify goal (which ISS? )

The MERIT System n n Theory vs Practice… Graphical interface (not NL dialogue) User does case selection (for eventual case -based reasoning) Example task is relational database (not free text IR): uses form filling (!)

Discussion n n Contribution to IR: user-centered view, application of many non-IR theories (discourse, CBR) BUT: too complicated for the user?

Discussion n n Contribution to Dialogue Systems: difficult task (not often dealt with in DS), CBR (can we learn dialogue structure from data? ) BUT: lacks a good, unified, practical framework (too many different paradigms applied…)

Dialogue-based IR: Why? n n Google-like interface still predominant (despite MERIT) Why? Users receives a lot of information (document titles, summaries) and use it as they want n Very simple to learn n Very flexible n BUT: works on high bandwidth devices n

Dialogue-based IR: Why? n n For low bandwidth devices (PDA, phone), information-rich interface don’t work Only small pieces of information exchanged at a time System has to select Less information, more interaction

EUREKA: Idea n Use dialogue to submit queries to a web search engine, browse through the hierarchically clustered results, perform query reformulation/refinement, etc…

EUREKA: Overview n n Backend: Vivisimo (through web scraper) Dialogue Management: Raven. Claw (successor of CMU Communicator) Language Understanding: Light Open Vocabulary Parser NLG/TTS: template-based & Festival

Backend: Vivisimo n n Available clustering meta-search engine www. vivisimo. com Hand-written Perl web scraper (hope Vivisimo doesn’t change their page design by the end of the semester…)

LOV Parser n n Problem: traditional NL parsers require a dictionary not applicable to open domain IR Solution (implemented in C++): fix a small number of one-word commands (new_query, open, list_clusters) n parse each line as “[command] [arguments]” or “[command]” or “[arguments]” n

Dialogue Management: Raven. Claw n Hierarchical agent architecture: EUREKA Greet User Submit Query Prompt Query Get Cluster List New Query Get Doc List Open Cluster Inform Results Close … Cluster

NLG/TTS n n Template-based Language Generation (e. g. “I found <n_doc> documents. ”) General purpose Festival voice for TTS NB: Browsing through lists is not efficient with speech, even for lists of clusters

Already Implemented n n Working prototype Commands: new_query n list_clusters, list_documents n open, close (cluster) n more, back (list of clusters/documents) n

Demo

Future Work n n Add more functionalities (query refinement, summarization…) Make clever use of the dialogue (not only command control + browsing) System can provide advice to user on search strategies (e. g. “you need to refine the query”) n User and system can negotiate to specify the user’s information need (cf Belkin: overview vs specific document) n

Future Work/Discussion n Advantage of dialogue: more feedback from the user How can dialogue improve the efficiency of low bandwidth IR? Do we need to tailor IR techniques (e. g. clustering) for dialogue, or even design new techniques?

Vocal Access to IR n n Problem: ASR introduces a lot of erroneous words in a spoken query (for an open domain, speaker independent system) However, in an IR system: access to many text documents to help language modeling…

Vocal Access to a Newspaper Archive (Crestani 02) n n n Presents studies for a full voice-controlled IR system No dialogue: user query list of summaries Focuses on issues of: n n TTS: can user make relevance judgments when they hear document descriptions synthesized over the phone? (answer: yes) ASR: how does IR perform with recognized queries?

Using IR Techniques to Deal with Recognition Errors n n WER does have an impact on precision, although not much variation for WER in 27%-47% Relevance feedback: use documents judged relevant by the user as query Use prosodic stress to estimate information content of query terms Include semantically/phonetically close terms in the query

Improving ASR (Fujii et al 02) n Fujii et al propose LM adaptation based on the IR corpus: Offline “adaptation”: train on the whole corpus n Online adaptation: adapt on the top retrieved documents (then reperform ASR and IR) n n n Good results with offline trained LM (WER < 20%, AP loss of 20 -30% from text IR) No evaluation of online adaptation…

Vocal Access to IR: Discussion n Seems to work ok for some tasks Clever use of IR techniques BUT queries are not spontaneous nor natural (maybe) LM for Web queries? ? n What about dialogue? n n n