How Do Users Respond to Voice Input Errors

How Do Users Respond to Voice Input Errors? Lexical and Phonetic Query Reformulation in Voice Search SIGIR '13 Proceedings of the 36 th international ACM SIGIR conference on Research and development in information retrieval Jiepu Jiang, Wei Jeng, Daqing He 2015/6/26 yusuke 1

Author �Jiepu Jiang(姜捷璞) information scientist College of Information and Computer Science, UMass Amherst information retrieval, Search query 2

Newly found �Idea by having read this article measure the difference of the result that users imagined and the result that were really output. (Jaccard similarity) when errors occurred in query, categorize how users reformulated queries. ▪ Which reformulation is effective for which error? 3

Introduction �Query reformulation As users may have limited understanding of their information needs, the retrieval system, it is difficult for them to develop one single query to complete the search. relevant documents may be scattered among different subtopics, so that it is impossible to retrieve all relevant documents with a single query. 4

Introduction �Voice search Along with the rapidly increasing usage of mobile devices and the improvement of speech processing, it becomes an alternative search mode. users can vocalize their queries and the retrieval system utilizes the voice recognition results for retrieval. 5

Introduction �Previous studies query reformulation plays an important role in conventional textual search systems. �In this paper. . . focus on explaining query reformulation behaviors in the context of voice search. 6

Introduction �Three research objectives to characterize the types of voice input errors in voice search and evaluate their impacts on voice search. to identify and characterize users’ query reformulation patterns in voice search. to shed light on how to support query reformulation in voice search. 7

Methods �Settings and Experiment System controlled laboratory experiment setting ▪ focus is on how users change their queries when voice input errors happen in voice search. adopted the Google search app on the i. Pad for experiment ▪ using the i. Pad for experiment also replicated some form of mobile search environment. 8

Methods �Settings and Experiment System 9

Methods �Search Tasks and Topics selected 50 TREC* topics *: Text REtrieval Conference (TREC) �Experiment Procedure recruited 20 participants (14 females and 6 males). all 20 participants were native English speakers. could only vocalize queries, browse and click the search results, and use Google’s query suggestions. 10

Methods �Data Google’s search history automatically records the system’s transcribed queries and the users’ clickthrough pages. Each participant went through a semi-structured interview. 11

Methods �Search Effectiveness Evaluation assume that a set of topic-level relevant results can be collected to evaluate each query in the search sessions dealing with that topic. did not ask the participants to make relevance judgments, but relied on the clicked results as relevant documents for evaluation. calculate standard evaluation metrics such as n. DCG of the queries. 12

Voice input errors and reformulation patterns �Voice Input Error Identification voice query (qv): a user speaks transcribed query (qtr): the search system generates the transcription of qv for search ü voice input error : the actual content of a voice query qv differs from its transcribed query qtr. ü voice query reformulation : The transition from qv(i) to qv(i+1) 13

Voice input errors and reformulation patterns �Voice Input Error Identification ü recognition errors Missing words: words in qv that do not appear in qtr. Incorrect words: words in qtr that do not appear in qv. System Interruption: the participant was improperly interrupted by the system and failed to speak all of the query words. 14

Voice input errors and reformulation patterns �Voice Query Reformulation Patterns ü Lexical Query Reformulation Additional(ADD): Substitution(SUB): 15

Voice input errors and reformulation patterns �Voice Query Reformulation Patterns ü Lexical Query Reformulation Removal(RMV): Re-ordering(ORD): 16

Voice input errors and reformulation patterns �Voice Query Reformulation Patterns ü Phonetic Query Reformulation Partial Emphasis(PE): ex) 17

Voice input errors and reformulation patterns �Voice Query Reformulation Patterns ü Phonetic Query Reformulation Whole Emphasis(WE): 18

Influence of voice input error �RQ 1: How do speech recognition errors affect voice queries? Speech recognition error is the major type of voice input error. occurred in 810 voice queries (89. 2% of all 908 queries with voice input errors). 19

Influence of voice input error �RQ 1: How do speech recognition errors affect voice queries? 20

Influence of voice input error �RQ 1: How do speech recognition errors affect voice queries? calculated the Jaccard similarity of Google’s first pages of results between voice query and transcribed query 21

Influence of voice input error �RQ 1: How do speech recognition errors affect voice queries? 22

Influence of voice input error �RQ 2: How do system interruptions affect voice queries? 23

Influence of voice input error �RQ 3: When do speech recognition errors happen? as recognition error may happen in any word of a voice query, the more words spoken, the more likely an error happens. explored the relationship between speech recognition errors and certain types of words. ▪ calculated error rates 24

Influence of voice input error �RQ 3: When do speech recognition errors happen? error rates ▪ defined as the number of times a word was not recognized correctly divided by the total number of times the word was used in voice queries. 25

Influence of voice input error �RQ 4: How do voice input errors influence search sessions? collected 500 search sessions (20 participants with each working on 25 topics). 26

Influence of voice input error �RQ 4: How do voice input errors influence search sessions? 27

Influence of voice input error �RQ 4: How do voice input errors influence search sessions? voice input errors can also affect the participants’ affective feelings. 90% of our participants reported frustration with their search experience when voice input error occurred. voice input errors significantly affected the performance of voice queries, and consequently made the whole search process more difficult and less effective. 28

Voice query reformulation �focus on users’ query reformulations � use qv(1) and qtr(1), qv(2) and qtr(2) for the voice query and transcribed query both before and after query reformulation, respectively. �RQ 5: Can users’ query reformulations improve search performance of voice queries? the magnitude depends on whether voice input errors occurred before or after reformulation. 29

Voice query reformulation �? If no voiceall input error occurred in qv(1) or voice input error occur counting 1, 182 cases of reformulation, search in qv(2), queryimproved reformulation resulted infrom limited improvements a performance significantly 0. 129 to it sometimes evenbecause hinderedofsearch 0. 143 (+10. 85%) queryperformance. reformulation. 30

Voice query reformulation �RQ 6: Can users’ query reformulation correct the speech recognition errors in previous queries? calculated the statistics by the different types of queries and voice input errors in qv(2). 31

Voice query reformulation �RQ 7: How do users utilize different query reformulate patterns in voice search? Do voice input errors influence the use of query reformulation patterns? 32

Voice query reformulation �RQ 7: How do users utilize different query reformulate patterns in voice search? Do voice input errors influence the use of query reformulation patterns? lexical reformulations were still the primary forms of query reformulation. lexical reformulations were consistently used much more frequently than phonetic reformulations. 33

Voice query reformulation �RQ 7: How do users utilize different query reformulate patterns in voice search? Do voice input errors influence the use of query reformulation patterns? To conclude, these results indicate that in voice search, a user’s adoption of both lexical and phonetic query reformulation patterns were greatly impacted by voice input errors. 34

Voice query reformulation �RQ 8: How do users utilize different reformulation patterns to handle speech recognition errors? Are these patterns effective in correcting speech recognition errors? Solutions to speech recognition errors should be able to effectively correct these errors. 35

Voice query reformulation �evaluate the reformulation patterns by their effectiveness of correcting the missing words in voice queries. 1. calculated the percentage 2. caluculated the success rate of each pattern in correcting the missing words. 3. The improvement in n. DCG@10 between qtr(1) and qtr(2) when each pattern was used. 36

Voice query reformulation Results indicate that, when recognition errors happened, these lexical patterns were primarily used to correct speech recognition errors 37

Voice query reformulation found that substitution, re-ordering, partial emphasis, whole emphasis, and repeating were five effective reformulation strategies in voice search to handle recognition errors. 38

Voice query reformulation substitution and re-ordering are lexical patterns, but they outperformed the other three phonetic patterns in solving speech recognition errors. 39

Conclusion � found that voice input errors greatly changed the content and results of queries, resulting significant decline of search performance for individual queries. � characterized users’ query reformulation patterns in voice search and evaluated the effectiveness of those patterns in handling voice input errors and improving search effectiveness. � found that users utilized both lexical query reformulation patterns that exist in conventional search and phonetic query reformulation patterns newly found in voice search. 40