Eye Tracking Techniques and applications eie 426 Eye

Eye Tracking Techniques and applications eie 426 -Eye. Tracking-200903. ppt 2020/11/27 EIE 426 -AICV 1

Outline n n Part I : Physiology of the EYE Part II : Mechanism of Eye Movements Part III : Eye Tracking Technology Part IV : Eye Tracking Applications 2020/11/27 EIE 426 -AICV 2

How does our eye work 2020/11/27 EIE 426 -AICV 3

Part I : Physiology of the EYE HOW DOES OUR EYE WORK 2020/11/27 EIE 426 -AICV 4

2020/11/27 EIE 426 -AICV 5

Part II : Mechanism of Eye Movements CAN YOU BELIEVE YOUR EYE 2020/11/27 EIE 426 -AICV 6

Part III : Eye Tracking Technology HOW EYE TRACKER WORKS 2020/11/27 EIE 426 -AICV 7

Outline 1. Current State and Application of Eye Gaze Tracking (EGT) Technology 2. Theory and the Classification of EGT Technology 3. The Framework of EGT system

Ⅰ. Current State and Application of Eye Gaze Tracking (EGT) Technology

Ⅰ. Current State and Application of Eye Gaze Tracking (EGT) Technology n What is EGT eye gaze——The line from the fovea through the center of the pupil is the line of sight (Lo. S). Usually, we take the optical axis as line of gaze (Lo. G). Lo. G can be approximate to Lo. S. In fact, Lo. S determines a person’s visual attention. eye gaze Tracking——By image Processing, if the Lo. G or Lo. S can be estimated, the point of regard (POR) is computed as the intersection of the Lo. G (or Lo. S) with the object of the scene or space. n History and Development of EGT Interests of Visual attention can be traced back to 1897. At that time, it was a kind of diagnostic research, i. e. the recording of eye movement. Those technologies include ophthalmometer, Mechanical Method, Electro-Oculography (EOG), Optical-based Method , electromagnetic Oculography eye movement initially applied in medical research, such as brain and physiology analysis. with the development of Electronics, Computer and Image processing technology, further research focuses on Eye Gaze Tracking Technology.

Search Coil

A picture wearing the EOG apparatus

Ⅰ. Current State and Application of Eye Gaze Tracking (EGT) Technology n eye tracking techniques： 1. Direct Observation： ophthalmometer, peep-hole method 2. Mechanic Method：use level to record eye movement 3. Optical-based Method ：Reflection Method （mirror or prism）、Pupil- Corneal reflection、Purkinje Image 4. Electro-Oculography (EOG): recorded the difference in the skin potential 5. Electromagnetic Oculography ：The users gaze is estimated from measuring the voltage induced in the search coil by an external electromagnetic field Except Optical-based Method, the above methods are more or less low accuracy or high intrusiveness. As a result, Modern Eye Gaze Tracking Techniques are mostly based on Optical-based Method.

Ⅱ. Theory and the Classification of EGT Technology

Ⅱ. Theory and the Classification of EGT Technology n 1. 2. 3. 4. Image-Based EGT Technology Infrared Light : out of visible light; Paired with filter to eliminate the light of other wavelength； Eye camera : tracking eye movement and recording eye image sequence Image Processing: detecting the visual elements Further Estimation: 。Having pre-processed, the data of eye gaze can be get from eye-movement model

Ⅱ. Theory and the Classification of EGT Technology System Framework Input Image acquisition Output Data Estimation Image Processing Eye Tracking Feature Estimation Calibration

Ⅱ. Theory and the Classification of EGT Technology Function of Each Module Image acquisition: 1 get bright pupil or dark pupil 2 Image Processing : 3 Feature Estimation: 4 Eye Tracking: 5 Data Estimation: Filtering , Noise reducing , difference , thresholding Estimating the center of Pupil and corneal reflection Estimation of eye gaze Algorithm validation , eye movement data analysis

Theory of EGT Technology n n 2 D to detect the Optical Axis (basic) 2 D to detect the Line of Sight (Advanced)

Classification of EGT Technology Øview of humanity Øview of application ØView of appearance

Ⅱ. Theory and the Classification of EGT Technology From View of Humanity Usually By goggle , helmet … Head features：high accuracy Mounted Eye Tracker defects：intrusiveness Usually on table Remote features：non-intrusive Eye Tracker defects：eye and head relative motion

Ⅱ. Theory and the Classification of EGT Technology From View of Theory Purkinje Image EOG Limbus-Scalar IR Tracking ANN The Purkinje images are reflections created at different layers of the eye structure. Eye gaze can be calculated from these relative positions of these reflections By placing electrodes around the eye, it is possible to measure small differences in the skin potential corresponding to eye movement. The Limbus is the boundary between the white sclera and the dark ir of the eye. By placing IR light emitting diodes and photo-transistors, respectively, above and below the eye. the resulting voltage differenc is proportional to the angular deviation of the eye. Training images are taken when the user is looking at a specific Calibration markers. Use ANN to decide the eye gaze Contact Lens Use a small coil embedded into a contact lens that is tightly fit over the sclera. The user’s gaze is estimated from measuring the voltage induced in the search coil by an external electro-magnetic field. Pupil-Corneal reflection The IR source can generate a glint on corneal and divide pupil from iris, the difference between can represent the eye gaze movement

Ⅱ. Theory and the Classification of EGT Technology From View of Theory Purkinje Image EOG Limbus-Scalar IR Tracking ANN The Purkinje images are reflections created at different The Purkinje images are reflections layers of the eye structure. Eye gaze can be calculated created different layers the eye from these at relative positions of theseof reflections structure. Eye gaze can be calculated 将两��化�皮肤表面�极分�置于眼睛左右、上下两 By placing electrodes around the eye, it is possible from these relative positions of these � 就能引起眼球�化方向上的微弱�信号 to, measure small differences in the skin, �放大后得 potential 到眼球运�的位置信息 reflections corresponding to eye movement. The Limbus is the boundary between the white sclera and the dark ir The is the boundary between the white of the. Limbus eye. By placing IR light emitting diodes and photo-transistors, respectively, above andiris below the eye. resulting voltage differenc sclera and the dark of the eye. the This boundary can is proportional to the angular deviation of the eye. easily be optically detected and tracked Training images are taken when the user is looking at a specific Calibration markers. Use ANN to decide the eye gaze Contact Lens Use a small coil embedded into a contact lens that is tightly fit over the sclera. The user’s gaze is estimated from measuring the voltage induced in the search coil by an external electro-magnetic field. Pupil-Corneal reflection The IR source can generate a glint on corneal and divide pupil from iris, the difference between can represent the eye gaze movement

Ⅱ. Theory and the Classification of EGT Technology From View of Theory Purkinje Image EOG Limbus-Scalar IR Tracking ANN The Purkinje images are reflections created at different layers of the eye structure. Eye gaze can be calculated from these relative positions of these reflections By placing electrodes around the eye, it is possible to measure small differences in the skin potential corresponding to eye movement. The Limbus is the boundary between the white sclera and the dark iris of the eye. This boundary can easily be optically detected and tracked Training images are taken when the user is looking at a specific Calibration markers. Use ANN to decide the eye gaze Contact Lens Use a small coil embedded into a contact lens that is tightly fit over the sclera. The user’s gaze is estimated from measuring the voltage induced in the search coil by an external electro-magnetic field. Pupil-Corneal reflection The IR source can generate a glint on corneal and divide pupil from iris, the difference between can represent the eye gaze movement

Ⅱ. Theory and the Classification of EGT Technology From View of Theory Purkinje Image EOG Limbus-Scalar IR Tracking ANN The Purkinje images are reflections created at different layers of the eye structure. Eye gaze can be calculated from these relative positions of these reflections By placing electrodes around the eye, it is possible to measure small differences in the skin potential corresponding to eye movement. The Limbusisisthe boundary between the white The Limbus boundary between the white sclera andthe thedark of the placing IR sclera and iris of the eye. This By boundary can easilyemitting be optically detected tracked light diodes andand photo-transistors, respectively, above and below the eye. the 将两��化�皮肤表面�极分�置于眼睛左右、上下两� , Training images aredifference taken whenis the user is looking resulting voltage proportional to 就能引起眼球�化方向上的微弱�信号 at a specific Calibration markers. Use, �放大后得到眼球 ANN to decide the angular deviation of the eye. 运�的位置信息 the eye gaze Contact Lens Use a small coil embedded into a contact lens that is tightly fit over the sclera. The user’s gaze is estimated from measuring the voltage induced in the search coil by an external electro-magnetic field. Pupil-Corneal reflection The IR source can generate a glint on corneal and divide pupil from iris, the difference between can represent the eye gaze movement

Ⅱ. Theory and the Classification of EGT Technology From View of Theory Purkinje Image EOG Limbus-Scalar IR Tracking ANN The Purkinje images are reflections created at different layers of the eye structure. Eye gaze can be calculated from these relative positions of these reflections By placing electrodes around the eye, it is possible to measure small differences in the skin potential corresponding to eye movement. The Limbus is the boundary between the white sclera and the dark iris of the eye. This boundary can easily be optically detected and tracked Training images are taken when the user is looking at a specific Calibration markers. Use ANN to decide the eye gaze Contact Lens Use a small coil embedded into a contact lens that is tightly fit over the sclera. The user’s gaze is estimated from measuring the voltage induced in the search coil by an external electro-magnetic field. Pupil-Corneal reflection The IR source can generate a glint on corneal and divide pupil from iris, the difference between can represent the eye gaze movement

Ⅱ. Theory and the Classification of EGT Technology From View of Theory Purkinje Image EOG Limbus-Scalar IR Tracking The Purkinje images are reflections created at different layers of the eye structure. Eye gaze can be calculated from these relative positions of these reflections By placing electrodes around the eye, it is possible to measure small differences in the skin potential corresponding to eye movement. The Limbus is the boundary between the white sclera and the dark iris of the eye. This boundary can easily be optically detected and tracked Contact Lens Use a small embedded Training imagescoil are taken when theinto useraiscontact looking lens that is tightly fit over the sclera. The at a specific Calibration markers. Use ANN to decide the eye gaze user’s gaze is estimated from measuring 将两��化�皮肤表面�极分�置于眼睛左右、上下两 the voltage induced in the search coil by Use a small coil embedded into a contact lens that is tightly fit � , external 就能引起眼球�化方向上的微弱�信号 , �放大后得 an electro-magnetic field. over the sclera. The user’s gaze is estimated from measuring the Pupil-Corneal reflection The IR source can generate a glint on corneal and divide pupil from iris, the difference between can represent the eye gaze movement ANN 到眼球运�的位置信息 voltage induced in the search coil by an external electro-magnetic field.

Ⅱ. Theory and the Classification of EGT Technology From View of Theory Purkinje Image EOG Limbus-Scalar IR Tracking ANN The Purkinje images are reflections created at different layers of the eye structure. Eye gaze can be calculated from these relative positions of these reflections By placing electrodes around the eye, it is possible to measure small differences in the skin potential corresponding to eye movement. The Limbus is the boundary between the white sclera and the dark iris of the eye. This boundary can easily be optically detected and tracked Training images are taken when the user is looking at a specific Calibration markers. Use ANN to decide the eye gaze Contact Lens Use a small coil embedded into a contact lens that is tightly fit over the sclera. The user’s gaze is estimated from measuring the voltage induced in the search coil by an external electro-magnetic field. Pupil-Corneal reflection The IR source can generate a glint on corneal and divide pupil from iris, the difference between can represent the eye gaze movement

Ⅱ. Theory and the Classification of EGT Technology EGC Technology Accuracy Features Purkinje Image 0. 0170 High accuracy, but the light is hard to control and can only be used in Lab EOG 1. 50 -2° low robustness, low accuracy, high intrusiveness Limbus Tracking V 10 -70 H 0. 50 -70 Horizontal accuracy is better than Vertical, but both are low ANN 1. 3 -1. 8° No need of calibration, low accuracy, Contact Lens 0. 080 high accuracy, high intrusiveness Pupil Corneal Reflection 1° The best one till now

Ⅱ. Theory and the Classification of EGT Technology Feature of Eye Movement view ØØ从系 � of humanity �用角度

Ⅱ. Theory and the Classification of EGT Technology Development n n Early stage : Direct Observation , Mechanical Method Initial Methods, not used any more for high intrusiveness and poor accuracy n Development：EOG、 Electromagnetic Oculography Although have improved a lot and widely used , these method are disappearing because of the intrusiveness n Advanced：Optical Method： Because of its’ high accuracy and low intrusiveness, Optical Method have made rapid progress in recent years. 1. 2. 3. 4. The Purkinje images are reflections created at different layers of the eye structure. and the eye gaze can be calculated from these relative positions of these reflections Photo-Oculography : it measures the eye movement during it's translation/rotation. with the IR light source, shape of the pupil , Limbus or corneal reflection is detected Limbus-Scalar IR Method: They place IR light emitting diodes and IR light sensitive phototransistors, respectively, above and below the eye. Several such IR pairs can be mounted on goggles or helmets, a photo-transistor transforms the reflected IR light into a voltage. the resulting voltage difference is proportional to the angular deviation of the eye. Pupil-Corneal Reflection: The IR source can generate a glint on corneal and divide pupil from iris, the camera can extract the pupil which represent the eye gaze, on the other hand the corneal reflection represent the head motion, as a result, the difference between them represents the real eye gaze movement. 。

Ⅱ. Theory and the Classification of EGT Technology Feature of Remote Eye Gaze Tracker ØView of appearance n n Robustness q Light Source q Eyelash, Eyelids and Camera Position q System Error q Eye station (dry or wet) q head motion Lower accuracy than Head Mounted Eye Tracker

Ⅲ A Practical Eye Gaze Tracking System

Ⅲ A Practical Eye Gaze Tracking System Framework of EGT System n Hardware：a Pair of Cameras n n n eye camera scene camera eye camera is for acquisition of pupil corneal reflection image, scene camera is for: 1 mapping from the eye image coordinate to scene image coordinate ; 2 Showing the combining result of POR and scene in same image：

Ⅲ A Practical Eye Gaze Tracking System appearance of EGT System

Ⅲ Practical Eye Gaze Tracking System Theory of a Head Mounted Eye Tracker

Ⅲ Practical Eye Gaze Tracking System Bright Pupil and Dark Pupil Bright Pupil Dark Pupil Difference Image v When IR source is placed near the optical axis of the camera bright pupil as can be seen ; while IR is placed off the optical axis, a dark pupil can be seen. By Thresholding, a robust pupil contour can be extracted.

Ⅲ Practical Eye Gaze Tracking System Bright Pupil and Dark Pupil Bright Pupil Dark Pupil Corneal Reflection The IR source can generate a glint on corneal and divide pupil from iris, the camera can extract the pupil which represent the eye gaze, on the other hand the corneal reflection represent the head motion, as a result, the difference between them represents the real eye gaze movement.

Ⅲ Practical Eye Gaze Tracking System

Ⅲ Practical Eye Gaze Tracking System

Ⅲ Practical Eye Gaze Tracking System

Part IV : Applications HOW FAR YOU CAN FLY 2020/11/27 EIE 426 -AICV 41

Applications ± Human Computer Interaction (HCI)—— efficiency, humanity ± Intelligent Control —— EGT and Weapon Control ± Human Movement Study ——Typing, physical training ± Psychology —— Antinational Neuroscience, ± Visual attention & Driving ——Aviation, navigation, driving, traffic accidents inspection ± 2020/11/27 Scene and Image Perception—— Web, AD, Designing, Scene EIE 426 -AICV 42

Most Recent Application 2020/11/27 EIE 426 -AICV 52