Senior Capstone Project Gaze Tracking System what you

  • Slides: 36
Download presentation
Senior Capstone Project Gaze Tracking System what you see is finally what you get

Senior Capstone Project Gaze Tracking System what you see is finally what you get Team IRALAR Breanna Heidenburg -- Michael Lenisa -- Daniel Wentzel Advisor: Dr. Malinowski DISCLAIMER: This presentation was used specifically for the Bradley University Student Expo on April 4 th, 2008. Other versions may contain a more comprehensive description of the project.

Presentation Outline The Project ◦ Why is it important The Goals ◦ System breakdown

Presentation Outline The Project ◦ Why is it important The Goals ◦ System breakdown Image recognition Point transformation User Interface The Results

Brief overview What is our project?

Brief overview What is our project?

Brief Overview Track a user’s eye and use the information to control a computer

Brief Overview Track a user’s eye and use the information to control a computer cursor Wherever you look on the display is where the cursor goes

Brief Overview Enhances Human Computer Interaction ◦ Speed of use ◦ Hands-free use

Brief Overview Enhances Human Computer Interaction ◦ Speed of use ◦ Hands-free use

System Development 3 Part System ◦ Image Processing Application ◦ Calibration and Mapping system

System Development 3 Part System ◦ Image Processing Application ◦ Calibration and Mapping system ◦ GUI designed for gaze-based interaction Systems developed concurrently and independently Separate Applications at run-time

System Development Hardware and Image Processing Application

System Development Hardware and Image Processing Application

System Development User Interface Hardware ◦ Camera Quick. Cam Pro for Notebooks Visible Spectrum

System Development User Interface Hardware ◦ Camera Quick. Cam Pro for Notebooks Visible Spectrum Camera ◦ Polarizer Tiffen 25 mm polarizing filter Removes glare from eye reflections ◦ Lighting Diffuse LEDs Slightly distracting to the user, but necessary to provide light for the camera

System Development User Interface Lit. Eye LE-500 ◦ ◦ High resolution (SVGA) Color Display

System Development User Interface Lit. Eye LE-500 ◦ ◦ High resolution (SVGA) Color Display Translucent or opaque operation Stationary relative to user’s eye

System Development Software – Image Processing Real time pupil tracking system ◦ Developed in

System Development Software – Image Processing Real time pupil tracking system ◦ Developed in C using Open. CV image processing libraries ◦ Traditional image processing and blob tracking Capabilities ◦ ◦ ◦ Locate and determine center of pupil in image Low light and high reflection environments All eye colors Data logging and static test modes Packaged into self contained Windows installer for easy deployment onto any computer

System Development Software – Image Processing The Process ◦ ◦ ◦ ◦ ◦ Capture

System Development Software – Image Processing The Process ◦ ◦ ◦ ◦ ◦ Capture Image Extract Red Channel Smooth image Apply binary threshold Locate blobs Reject false positives Determine center of pupil blob Adapt threshold Repeat

System Development Software – Image Processing Summary ◦ The Good Dynamically adapts to changing

System Development Software – Image Processing Summary ◦ The Good Dynamically adapts to changing lighting conditions and eye types Maintains performance in low-light and specularly noisy conditions ◦ The Bad Still relies on Logitech camera drivers Extreme reflections still cause problems

System Development Image Processing in Action Raw Pre-Processed Completed Capture Recognition Image

System Development Image Processing in Action Raw Pre-Processed Completed Capture Recognition Image

System Development Image Processing Examples of performance in poor conditions Low Light Difficult False

System Development Image Processing Examples of performance in poor conditions Low Light Difficult False Positive

System Development Calibration and point mapping

System Development Calibration and point mapping

System Development Eye Mapping System for mapping the location of the center of the

System Development Eye Mapping System for mapping the location of the center of the pupil to a pixel on a computer screen Reasons ◦ Geometry The eye is not flat but a screen is not ◦ User Customization All eyes are different Everyone wears the HMD differently ◦ User Training Calibration system also acts as a quick tutorial

System Development Point Transformation 3 dimensional best fit plane ◦ Currently using a 4

System Development Point Transformation 3 dimensional best fit plane ◦ Currently using a 4 th degree best fit Xpix = A 1 + Xeye*B 1 + Yeye*C 1 Ypix = A 2 + Xeye*B 2 + Yeye*C 2 Calibration sub-system determines these coefficients

System Development Calibration How do we solve the problem? ◦ Multiple Variable Linear Regression

System Development Calibration How do we solve the problem? ◦ Multiple Variable Linear Regression – Least Squares Y = B 0 + B 1 x 1 + … + B k x k ◦ Uses matrix algebra to obtain a coefficient matrix B[] = (X’X)-1 X’Y

System Development Calibration Results B[] = (X’X)-1 X’Y ◦ pixel. X = 224. 9

System Development Calibration Results B[] = (X’X)-1 X’Y ◦ pixel. X = 224. 9 + 4. 8*eye. X – 2. 9*eye. Y ◦ pixel. Y = 1612. 5 + 2. 0*eye. X – 5. 6*eye. Y

System Development ‘Clicking’ How do we click? ◦ Monitor eye movements ◦ Identify pauses

System Development ‘Clicking’ How do we click? ◦ Monitor eye movements ◦ Identify pauses ◦ When eye position is within a small percentage for a certain amount of time ◦ Currently set at 5 frames (~200 m. S) ◦ Generally, it takes 230 m. S for a hand to click a mouse.

System Development Communications and Custom GUI interface

System Development Communications and Custom GUI interface

System Development Communication Why is it important? ◦ Allows multiple processes to communicate ◦

System Development Communication Why is it important? ◦ Allows multiple processes to communicate ◦ Allows relay of time sensitive information UDP vs. TCP ◦ UDP -> ‘Connectionless’ ◦ TCP -> ‘Connection oriented’ Multi-threading ◦ Necessary for running multiple pieces of code in a single process

System Development Communication Multi-threading issues present

System Development Communication Multi-threading issues present

System Development Communication Multi-threading ◦ Issues with public variable usage ◦ Solutions we are

System Development Communication Multi-threading ◦ Issues with public variable usage ◦ Solutions we are using: Critical Section Raises thread priority (does not allow for interrupts) Receive request for data (over UDP) Raise Thread priority Read Variable Lower Thread Priority Reply to request

System Development User Interface Custom GUI for Gaze Tracking Applications Why? ◦ Gaze tracking

System Development User Interface Custom GUI for Gaze Tracking Applications Why? ◦ Gaze tracking accuracy limited by inherent properties of human vision system ◦ Traditional GUI too small and intrusive for use with transparent HMD ◦ Demonstrate applications of gaze tracking

System Development User Interface Achieved via a combination of Hardware and Software: WPF &

System Development User Interface Achieved via a combination of Hardware and Software: WPF & C# ◦ XAML (e. Xtensible Application Markup Language) ◦ Very similar to XML ◦ Uses ‘code-behind’ in a similar style to javascript ‘code-behind’ is in C#

System Development User Interface Multiple applications have been created within the interface Multiple screens

System Development User Interface Multiple applications have been created within the interface Multiple screens for functionality testing ◦ even games Additional ability to minimize interface and interact with external applications is present software

System Development User Interface Calibration Screen Main Screen

System Development User Interface Calibration Screen Main Screen

Results What did our test results show?

Results What did our test results show?

Results Calibration

Results Calibration

Results Calibration

Results Calibration

Results Timing

Results Timing

Results Schedule

Results Schedule

Thanks Who helped us out?

Thanks Who helped us out?

Thanks Northrop Grumman Lit. Eye Dr. Malinowski and the EE faculty Mr. Mattus &

Thanks Northrop Grumman Lit. Eye Dr. Malinowski and the EE faculty Mr. Mattus & Mr. Schmidt

Questions

Questions