CROSSMODAL EFFECTS IN TACTILE AND VISUAL SIGNALING Can

CROSSMODAL EFFECTS IN TACTILE AND VISUAL SIGNALING Can we get the message through?

Effective action depends on… Communicating Processing Sensing Presenting Understanding the situation

Picture of thermometer in HMMWV taken 31 May, ‘ 04 in central Baghdad.

Sensing

Sensing… with the senses covered…

Effective action depends on Communicating

Communicating… with the senses in extreme context

GPS L-Band Full FBCB 2 on Lower Tactical Internet TOC ABCS CO/PLT Leaders/Sgts CO/PLT Voice Nets SQD Leader Squad Voice Net TM Leaders DBCS (T) TM Leaders Battalion SA Net DBCS (P) Who else needs to know? GPS BFT (FBCB 2)

Literature Review • Gilson & Fenton (1974) A tactile display presenting aircraft control information was comparable to using a visual display and in some cases actually produced a reduction in tracking error, altitude variations decreased as well as speed variations. • Zlontik (1988) The use of tactile signaling helped pilots and air traffic controllers overcome some of their challenges. • Gilland & Schlegel (1994) reported that tactile communication applied to the head lowered performance on a concurrent task. • Sklar & Sarter (1999) Showed tactile signals are able to be perceived concurrently with auditory or visual information. • Cholewiak, Brill, & Schwab (2004) found that for the torso, a ring of eight loci of vibration is the most one can resolve with accuracy exceeding 90%. • Hoop, Smith, Clegg & Heggestad (2005) Tactile signals aided in interruption management. • Pettitt, Redden, & Carstens (2006) found that tactile presented arm and hand signals could be interpreted faster than standard visually presented signals when undergoing dynamic movement tasks.

Tactile Patterns emulating Standard Army Hand Signals* (FM 21 -60) Tactons*…

Merlo, J. L. , Terrence, P. I. Stafford S. C. , Gilson, R. & Hancock, P. A. , Redden E. S. , Krausman A. , Pettitt, R. , White, T. L. , & Carstens, C. B. (2006). The effects of dynamic environments and physiological stress on tactile communication and signaling. Proceedings of the Society for Human Performance in Extreme Environments 4 th Annual Meeting, San Francisco, CA. Merlo, J. L. , Stafford S. C. , Gilson, R. & Hancock, P. A. (2006). The effects of physiological stress on tactile communication. Proceedings of the Human Factors and Ergonomics Society 50 th Annual Meeting, San Francisco, CA. Merlo, J. L. , Terrence, P. I. Stafford S. C. , Gilson, R. & Hancock, P. A. , Redden, E. S. , Krausman, A. , Pettitt, R. , White, T. L. , & Carstens, C. B. (2006). Communicating through the use of vibrotactile displays for dismounted and mounted soldiers. Proceedings of the Twenty-fifth Army Science Conference, Orlando, FL.

Multi-Modal Benefits • Spence and Driver (2004) Attention and then perception is largely a multi-sensory construction • Stein and Meredith (1993) bi-modal and tri-modal neurons have stronger cellular response with two sensory modalities than with unimodal stimulation • Wallace, Meredith, Stein (1998) found that combinations of two different sensory stimuli have been shown to significantly enhance the response of Superior Colliculus neurons above those evoked by either uni-modal stimulus • Allman and Meredith (2007) found uni-modal visual neurons had their visual response modulated by concurrently presented auditory stimuli • Current research has explored these conflicts of congruent and incongruent cross-modal paradigms in many of the possible permutations and combinations of very basic visual, auditory, kinesthetic and tactile stimuli (Soto-Faraco, Lyons, Gazzaniga, Spence, & Kingstone, 2002; Soto-Faraco, Morein-Zamir, & Kingstone, 2005; Soto-Faraco, Spence, & Kingstone, 2004 a; Spence & Driver, 1997; Spence & Walton, 2005).

Experimental Method • visual only (video presentation of the signals) • tactile only (tactile presentation of the signals) • both visual and tactile simultaneous and congruent (i. e. the same signals were presented both visually on the video and through the tactile system) • both visual and tactile simultaneous and incongruent (i. e. the visually presented signal did not match the presented tactile signal).

Equipment • an acoustic transducers • displace 200 -300 Hz sinusoidal vibrations onto the skin • C 2’s contactor is 7 mm, with a 1 mm gap separating it from the tactor aluminum housing • 17 grams • 8 tactors mounted on an elastic belt with acoustic padding • 8 X 45 degrees = 360 degrees • Tactor Control Box (TCB) Stores tactile messages and contains bluetooth receiver • Total system under 2 lbs

Spatial layout of Tactile Display N NW 8 W 7 1 FRONT NE 2 LEFT RIGHT 6 BACK 5 4 SE SW S 3 E

Method Arm/ Hand Signal Attention Tactile Pattern Sequenced activation of three forward tactors creating a linear motion back and forth across the front of the body Halt Four tactors simultaneously actuated on the sides of the body Rally Sequenced activation of all tactors creating a circular motion around the body Move Out NBC Sequenced back-to-front activation of tactors creating movement from each side of the body which converges in the front Sequenced activation on both sides simultaneously creating three distinct impulses on the sides of the body Visual Pattern

Attention

RALLY

Equipment Hardware • Samsung Q 1 Ultra Mobile Ø Intel Celeron M ULV (900 MHz) Ø 7” WVGA (800 x 480) LCD • Sound dampening earmuffs Ø reduction rating of 11. 3 decibels at 250 Hz Software • Windows XP, tablet • Lab. VIEW 8. 2 National Instruments • Microsoft Office 2003 • SPSS 11. 5 • Windows Media Player

Experimental Procedure 20 Participants Training session • 10 Males • Learned both visual and • 10 Females • Mode age 19 years Signals • Attention • Halt • Move Out • NBC tactile signals simultaneously • Advanced only after mastery • Lasted approximately 10 minutes Experimental block • 10 trials visual only • 10 trials tactile only • 20 congruent trials • 20 incongruent trials 60 total trials per block • Rally 2 x Blocks = 120 trials

Screen Capture from Visual interface Participants chose the appropriate signal as fast a possible after signal presentation. The tactile only presentation showed a video of the soldier standing at the rigid position attention.

Initial Results Response Time x Condition t(19)=-3. 98, p≤. 01 t(19)=-2. 25, p≤. 04 t(19)=-2. 16, p≤. 04 Oneway ANOVA ( F(1, 19)=473. 445, p<. 000, partial eta squared =. 961, observed power, β= 1. 000 )

Accuracy x Condition t(19)=4. 03, p≤. 01 t(19)=1. 27, p≤. 22 t(19)=2. 00, p≤. 06

Initial Results

Bl oc k 2 Bl oc k 1 Initial Results

Discussion • Data should provide valuable information on: – – – Congruency effects Selected attention Interference effects Tactile signal efficiency Modality interactions • Future research for tactile signaling includes: – Spatial references – Temporal importance – Alternate inputs

Questions?