Multimodal feedback An assessment of performance and mental

Multimodal feedback An assessment of performance and mental workload

1. Introduction v Multimodal interfaces are one way to broaden and strengthen the critical communication between humans and computers. v Multimodal input is often preferred over unimodal input(Oviatt et al. 1997) v By understanding how these forms of feedback affect users’ performance, general recommendations are provided that increase the effectiveness and efficiency of direct manipulation tasks.

2. 1 Multimodal interfaces v Bolt’s ‘Put That There’system(1980) →first multimodal interface (speech&hand gesture) v Cohen et al &Shoptalk →Speech input coupled with keyboard or mouse input was widely investigated. (natural language & direct manipulation) v Multimodal research focuses either on input or on ouput input-configurations , output- investigations

2. 2 Auditory, haptic and visual feedback research Auditory, haptic and visual feedback represent the most prominent types of feedback (GUIs 사 용) v 2. 2. 1 v 2. 2. 2 v 2. 2. 3 v 2. 2. 4 Auditory feedback Haptic feedback Visual feedback Supplementing visual feedback

2. 3 Performance measurements Effectiveness –‘doing the right things’ efficiency- ‘doing things the right way’ 2. 3. 1 mental workload measurement NASA-TLX →the most valid measure, the highest user acceptance, the smallest between-subject variability, sensitive, reliable 2. 3. 2 time measurements Trial or task completion time has frequently been used to gauge differences between unique modalities and combinations of modalities applied in multimodal interfaces.

3. 1 Participants v v 21~36세 사이의 사람 32명을 선출 Right-handedness Corrected vision near 20/20 Near normal hearing capability 3. 2 Apparatus v Auditory-Haptic-Visual(AHV) Mutimodal software within a GUI

3. 3 procedure v 3. 3. 1 Training : The goal of the training task was to familiarize the participants with the experimental task and each feedback condition so that by completion of the training the participants were equally proficient with each feedback condition v 3. 3. 2 Experimental task figure 1. 과 같이 file을 A~E까지 5개씩 랜덤으로 나열한 후 A~E 까지 써있는 folder에 넣는 실험을 행한다. 폴더에 정확히 넣으 면 file은 없어지고 원래 위치에 새로운 파일이 생기며 X라고 표 시하여 혼돈이 오지 않게 한다. 그리고 모든 폴더에 파일이 들어 가면 새로운 배열을 주어 다시 같은 시행을 한다.

Figure 1. Screen capture of the experimental task

3. 4 Experimental design v Two level : ‘Present’ and ‘absent’ auditory, haptic, visual feedback을 존재와 부재로 나누어 table 2와 같이 구분 2 X 2 X 2 factorial로 나누어 따져본다. v 3. 4. 1 Independent variable auditory, haptic, visual feedback v 3. 4. 2 Dependent variable Mental workload, trial completion time(TCT), target highlight time(THT)

Table 2. Experimental design: auditory, haptic and visual feedback

4. 1 trial completion time(TCT) v TCT revealed a main effect of haptic feedback(figure 2. ) →haptic feedback을 주었을 때 보다 주지 않았을 때, TCT가 빠르다. v Interaction TCT between haptic and auditory feedback(figure 3. ) →strong haptic main effect, bimodal effect of auditory and haptic feedback combined v Marginal means displayed in figure 3 are listed in table 3.

Figure 2. Haptic main effect for TCT

Figure 3. Haptic X auditory interaction for TCT

Table 3. Trial completion time – marginal means.

4. 2 target highlight time(THT) v THT increased when auditory feedback was present and decreased when haptic feedback was present (auditory<figure 4>, haptic<figure 5>) v Three-way interaction between visual and haptic with no auditory feedback present(figure 6) interaction between visual and haptic with auditory feedback present(figure 7) v All the marginal means are listed in table 4

Figure 4. Auditory main effect for THT

Figure 5. haptic main effect for THT

Figure 6. Visual X haptic interaction when Auditory is absent for THT

Figure 7. Visual X Haptic interaction when Auditory is present for THT

Table 4. Target highlight time-marginal means.

4. 3 Workload : mental demand v When visual feedback was provided, the level of perceived mental demand decreased. v The auditory by haptic interaction with visual absent(figure 8. ) v The auditory by haptic interaction with visual feedback present(figure 9. ) v Auditory feedback and haptic feedback were strongly affected by the presence of visual feedback. v The remainig marginal means are listed in table 5.

Figure 8. Auditory X Haptic interaction when visual is absent for mental demand

Figure 9. Auditory X Haptic interaction when Visual is present for Mental Demand

Table 5. Mental demand ranking-marginal means

Discussion v TCT represented the time it took to complete one entire trial(overall task performance) v THT represented the effect of the feedback v THT were included in the TCT measurement v The THTs were significantly higher when auditory feedback was presented v Figure 10 was constructed as a representation of the feedback conditions that are most effective and least effective on target highlight time(table 6 t-test result) v 어떠한bimodal feedback은 performance time과 mental demand를 증진시킨다→좋은 bimodal condition을 찾도록 노 력하자

Figure 10. Performance by feedback type for THT

Table 6. Comparisons of feedback conditions for target highlight time

Conclusions v Haptic and visual feedback are more beneficial when used alone as well as in combination with each other when compared with other uni-, biand trimodal conditions v Temporal nature of auditory feedback alone and in combination can hinder performance. Observations confirmed that participants waited for the auditory feedback to conclude before continuing to complete the task v Bimodal feedback, appropriate to the task, offers performance benefits while not increasing workload