Introduction Eye Movements and Eye Tracking Irina A

Introduction: Eye. Movements and Eye. Tracking Irina A. Sekerina (Higher School of Economics and CUNY) Workshop on Reading in Cyrillic ● 21 September, 2015 1

Roadmap A. Introduction: History, Terminology, Background B. Eye-Tracking Applications C. Techniques and Equipment D. Eye-Tracking in Psycholinguistics: Reading E. Models of Reading 2

A. Introduction: History, Terminology, Background A 1. Some recent history A 2. Scanpaths and visual attention A 3. The eye A 4. Types of eye movements 3

Alfred Luk’yanovich Yarbus (1965)*: 1914 -1986 A 1: Some [Recent] History demonstrated sequential, but variable, viewing patterns over particular image regions Noton and Stark (1971)**: showed that participants tend to fixate identifiable regions of interest, containing “informative details”; coined term “scanpath” describing eye movement patterns * Ярбус, A. Л. (1965). Роль движений глаз в процессе зрения. Наука. Yarbus, A. L. (1967). Eye Movements and Vision. New York: Plenum Press. ** Noton, D. , & Stark, L. (1971). Scanpaths in eye movements during pattern perception. Science, 171(3968), 308 -311. 4

“Не ждали [Unexpected]” (by Ilya Repin, 1884) 5

Yarbus’ early scanpath recording: 1: examine at will A 2: Scanpaths and Visual Attention 2: estimate wealth 3: estimate ages 4: guess previous activity 5: remember clothing 6: remember position 7: time since last visit 6

A 2: Scanpaths of a portrait 7

A 3: The Eye The eye—“the world’s worst camera” suffers from numerous optical imperfections. . . endowed with several compensatory mechanisms 8

А 3: Human visual field The retina is a light sensitive structure inside of the eye responsible for transforming light into signals, which are later converted into an image by the visual cortex in the brain. The fovea is a section of the retina that contains a high density of both kinds of light receptor cells found in the eye, i. e. Cone and Rod cells. 9

A 4: Types of eye movements Eye movements are mainly used to reposition the fovea Five main classes of eye movements: Saccadic (saccades and fixations) Smooth pursuit Vergence Vestibular Physiological nystagmus Other types of movements are nonpositional (adaptation, accommodation) 10

A 4: Saccades Rapid eye movements between fixations used to reposition fovea Voluntary and reflexive Range in duration from 10 ms – 100 ms Effectively blind during transition Deemed ballistic (pre-programmed) and stereotyped (reproducible) 11

A 4: Fixations Possibly the most important type of eye movement for attentional applications 90% viewing time is devoted to fixations duration: 150 ms – 600 ms Duchowski, A. T. (2007). Eye-Tracking Methodology. 2 nd Ed. 360 p. ISBN: 978 -1 -84628 -808 -7 12

B. Applications B 1. Ergonomics and Human Factors B 2. Marketing and Advertising B 3. Websites B 4. Psychology, Psychophysics, Neuroscience 13

Wide variety of eye tracking applications exist, each class increasingly relying on advanced graphical techniques: B. Applications Advertising Human Factors Displays HCI & Collaborative Systems Virtual Reality Psychology, Psychophysics, Neuroscience 14

Applications range from usability studies to testing effectiveness of cockpit displays B 2: Ergonomics and Human Factors Examples: evaluation of tool icon groupings comparison of gaze-based and mouse interaction organization of click-down menus testing electronic layout of pilots’ visual flight rules testing simulators for training effectiveness 15

B 3: Marketing and Advertising Applications range from assessing ad effectiveness (copy testing) in various media (print, images, video, etc. ) to disclosure research (visibility of fine print) Examples: eye movements over print media (e. g. , magazines) eye movements over TV ads, web pages, etc. 16

B 3: Marketing and Advertising Scanpaths over printed magazine ads 17

Most people view websites in a “F” shaped flow. B 3: Websites First they scan the page at the top, from left to right. Then the eyes go back to the left and down the page. They again scan to the right and back along the same pattern. 18

B 4: Psychology, Psychophysics, and Neuroscience Applications range from basic research in vision science to investigation of visual exploration in aesthetics (e. g. , perception of art). Examples: psychophysics: spatial acuity, contrast, sensitivity perception: reading, natural scenery, . . . neuroscience: cognitive load, with f. MRI an ERP psycholinguistics 19

B 4: Perception of Art (Rembrandt’s Anatomy Lesson) (Duchowski, 2007) (a) Aesthetic group (b) Semantic group small but visible differences in scanpaths similar sets of fixated image features 20

C. Techniques and Equipment C 1. Electro-oculography C 2. Scleral contact lens/search coil C 3. Video-based combined pupil and corneal reflection 21

C. Techniques and Equipment Two broad applications of eye movement monitoring/recording techniques: measuring position of eye relative to the head measuring orientation of eye in space, or the “point of regard” (POR)—used to identify fixated elements in a visual scene The most widely used apparatus for measuring the POR is the video-based corneal reflection eye-tracker 22

C 1: Techniques First method for objective eye movement measurements using corneal reflection reported in 1901 Techniques using contact lenses to improve accuracy developed in 1950 s (invasive) Remote (non-invasive) trackers rely on visible features of the eye (e. g. , pupil) Fast image processing techniques have facilitated real-time video-based systems 23

C 1: Electroоculography • most widely used method some 30 years ago (still used today) • similar to electromechanical motioncapture • measures eye movements relative to head position Relies on measurement of skin’s potential differences, • not generally suitable using electrodes placed for POR measurement around the eye (unless head is also tracked) 24

C 2: Scleral Contact Lens/ Search Coil search coil embedded in contact lens and electromagnetic field frames • possibly most precise • similar to electromagnetic position/ orientation trackers used in motioncapture 25

C 2: Scleral Contact Lens/Search Coil Example of scleral • highly accurate, but limited suction ring insertion: measurement range (~5°) most intrusive method insertion of lens requires care wearing of lens causes discomfort • measures eye movements relative to head position • not generally suitable for POR measurement (unless head is also tracked) 26

C 3: Video. Based Combined Pupil/ Corneal Reflection H Head-mounted videobased eye tracker • most suitable for (graphical) interactive systems, e. g. , VR • Monocular and binocular systems 27

SMI C 3: Existing Eye-Trackers ISCAN TOBII Remote system ASL head-mounted ISCAN child headmounted 28

C 3: Existing Eye-Trackers 2013 next to a student’s poster 1997 “Kindergarten. Path” article 29

A light source is used to cause reflection patterns on the cornea and pupil of the test person. C 3: POR Method A camera will then be used to capture an image of the eye. The direction of the gaze is then calculated using the angles and distances. 30

C 3: How it Works 31

D. Eye. Tracking in Psycholinguistics: Reading D 1. Visual World Eye-Tracking Paradigm D 2. Dual-Purkinjie Eye-Tracking in Reading D 3. Examples of Eye Movements in Reading (Tobii) D 4. Eye-Movement Parameters in Reading 32

Cooper (1974)* and Tanenhaus et al. (1995)** D 1: The Visual World Eye-Tracking Paradigm The Mind-Eye hypothesis Relationship between eye fixations and the meaning of concurrently spoken sentence Using this relationship as a research tool in cognitive psychology and psycholinguistics Applications: Speech perception and memory Language processing *Cooper, R. (1974). The control of eye fixation by the meaning of spoken language. Cognitive Psychology, 6, 84 -107. **Tanenhaus, M. K. , Spivey-Knowlton, M. J. , Eberhard, K. M. , & Sedivy, J. C. (1995). Integration of visual and linguistic information in spoken language comprehension. Science, 268(5217), 1632 -34. Федорова, О. В. (2008). Методика регистрации движений глаз Визуальный мир: шанс для сближения психолингвистических традиций. Вопросы языкознания, 6, 33 98– 120.

D 2: Dual. Purkinjie Eye. Trackers for Reading DPI trackers measure rotational and translational eye movements Dual-Purkinje image (DPI) eye tracker so-called generation-V trackers measure the 1 st and 4 th Purkinje images 1 st and 4 th reflections move together through same distance upon eye translation, but separate upon eye rotation highly precise used to be expensive and difficult to set up 34

Purkinje images appear as small white dots in close proximity to the (dark) pupil D 2: Pupil and Purkinje Images tracker calibration is achieved by measuring user gazing at properly positioned grid points (usually 5 or 9) tracker interpolates POR on perpendicular screen in front of user Pupil and Purkinje images as seen by eye tracker’s camera 35

D 3: Tobii Examples of Eye Movements in Reading A gaze replay, recorded at 300 Hz using the Tobii TX 300 eye tracker, of a participant in a reading study: https: //www. youtube. com/watch? v=VBTZNyd. Uh 0 w Eye-tracking with Tobii (4 -year-old child) http: //www. youtube. com/watch? v=VFIZDZwdf-0 36

D 4. Eye- Movement Parameters 1. Studying eye movements per se to learn about reading 2. Using eye movements in reading as a means to infer cognitive processes (e. g. , language processing) 37

D 4. Eye. Movement Parameters 1. Saccade latency: 150 -175 ms; 2. Perceptual span: ~4 symbols to the left, ~15 symbols to the right; 3. Skipping words: 2 -3 -letter words are skipped 75%, 8 -letter words are never skipped; 4. Regressions: 10 -letter spaces happen because of problems in understanding text 38

D 4. Eye Movements and Word Recognition in Reading 1. 2. 3. 4. 5. 6. 7. Word frequency Word familiarity Age of acquisition Number of meanings Morphology Predictability Plausibility 39

E. Models of Reading E 1. Types of Models of Reading E 2. E-Z Model E 3. SWIFT 40

E. Models of Reading Models of reading behavior try to explain how the eye movement control system makes two fundamental decisions involved in reading: When the eyes should move Where the gaze should land 41

E 1. Types of Models of Reading 1. Oculomotor control models 2. Cognitive control models 1. E-Z Reader 2. SWIFT 42

E-Z Reader model http: //raynerlab. ucsd. edu/ Keith_Rayner. html E 1: E-Z Reader Model Rayner, K. (1978). Eye movements in reading and information processing. Psychological Bulletin, 85, 618 -660. Rayner, K. (1998). Eye movements in reading and information processing: 20 years of research. Psychological Bulletin, 124, 372 -422 Keith Rayner (UCSD) 1943 -2015 43

E 2. E-Z Reader 1. 50 ms: visual uptake phase 2. 75 -100 ms: L 1 phase of lexical access 3. Saccade planning 4. L 2 phase of lexical access Accounts for frequency, predictability, spillover effects, covert attention, importance of parafoveal preview 44

E 3: SWIFT Reinhold Kliegl (University of Potsdam, Germany): SWIFT Model http: //www. psych. uni-potsdam. de/people/kliegl/ 45

Reinhold Kliegl’s Lecture 1 11: 30 am-12: 30 pm “Distributed Processing During Fixation Durations in Reading” 46