Human computer interaction advancement by usage of smart

Human computer interaction advancement by usage of smart phones for motion tracking and remote operation J. ANISH DEV COLLEGE OF ENGINEERING, GUINDY, ANNA UNIVERSITY

Introduction To HCI (Human Computer Interaction) Various means of HCI - so far? Standard systems: § Standard Mouse and Keyboard Setup § Touch screen Setup Custom systems: § Gesture based interaction § Custom hardware based interaction: • • • ◦ Nintendo Wii Microsoft Kinect Infrared based interaction systems Hybrid systems involving Custom H/W and image processing

Introduction To HCI A deeper look into custom HCI systems and their components Essential components: § Custom hardware to facilitate user input or feedback

Introduction To HCI A deeper look into custom HCI systems and their components Essential components: § Computer Vision based software for Visual data processing

Analysis of current HCI systems A PRECURSOR TO DEVELOPMENT OF THE SAME

Analysis of current HCI What are the common requirements in these systems? 1) A device-linking network: § Facilitated by Wi. Fi, Bluetooth or any other mean of networking § Links the Custom hardware to the computer or target device Its implications? § Requirement of a preset registration of a user’s input device to the system § Possible hassles of setting up an ever present network § Constraints of being wired and therefore physically tethered within sometimes inconvenient operating distances § Difficulty of control of distributing control to multiple users

Analysis of current HCI What are the common requirements in these systems? 2) Custom Hardware: § Devices which are built exclusively for a singular or preplanned set of generic uses Its implications? § Construction of devices may require fabrication of basic components which may prove to be expensive § Hardware once built and deployed for a specific purpose or set of generic capabilities may require to be fundamentally modified to match that of the target computer’s Computer Vision software. § Constraints of shape, feature or functionality of devices due to usage of unmodifiable commercially available components

Analysis of current HCI What are the common requirements in these systems? 3) Inter reliance of software and hardware: § Computer vision software is written exclusive for each device Its implications? § Generally, software written for a target product cannot be used by another, even if only minutely different by design. § Software advancement depends on hardware specifications or component specific improvement.

Why work towards liberation from these requirements? EXPLORING THE POSSIBILITY OF ADVANTAGES

Custom Hardware Typical HCI Co relation of H/W and S/w Linking Network

Standard Hardware Proof of Concept HCI Unbound co relation of H/w and S/w Linking Network

IFFAT for HCI I NTEGRATED F ACULA F ILTERING A ND T RACKING FOR HUMAN COMPUTER INTERACTION

IFFAT for HCI System setup overview: Average home machine with a built in or separate webcam Smart or Feature phone Simple Computer Vision Software featuring Open. CV

IFFAT for HCI Operation Overview

Pre-Operational Procedures SYSTEM INITIALIZATION

IFFAT for HCI 1) Manual Calibration: §Brightness and Contrast §Improves accuracy for individual operating environments 2) Specification of Region of Interest: §Rectangular extraction of visual data from stream §Used to avoid light sources §Adds to convenience of operation Pre-Operational Procedures:

IFFAT for HCI 3) Specification of colour codes to client: §Range of colours Specified in RGB values §Each colour code is assigned a computer function §Colour can correspond to motion tracking or event triggers §RGB range values can be auto assigned for preset operating conditions §No restrictions on number of colours Pre-Operational Procedures: Left Click Track Mouse Open Right Click

Operational Procedures SYSTEM’S WORKING

IFFAT for HCI 1) Reception and cropping of live feed: §Application of R. O. I specifications 2) Application of manual calibration §Brightness specification §Contrast specification §cv. Set. Capture. Property() function in Open. CV when possible. §Or cv. Convert. Scale() during post processing Operational Procedures:

IFFAT for HCI 3) Visual feed processing: § Input frame is filtered in accordance with RGB ranges specified for each colour that the client expects to find. § cv. In. Range. S() is used as a filtering function § The outcome is a set of n single channelled frames, each of which are filtered with respect to the n number of colours that the client expects to find. Operational Procedures:

IFFAT for HCI 4) Gaussian blur: §Applied to each segmented frame §Reduces noise and helps improves accuracy 5) Blob Detection §Connected-component labelling is performed on these segmented frames by using the cv. Label()* cv. Blob function. * "A linear-time component-labeling algorithm using contour tracing technique“ - Fu Chang, Chun-Jen Chen and Chi-Jen Lu Operational Procedures:

IFFAT for HCI 6) Blob Removal: §Removal of blobs outside the maximum and minimum areas of expected beacon size, if specified, by usage of cv. Filter. By. Area(). §Only one of the segmented frames is expected to contain a blob. 7) Blob Tracking: §Tracking of blob movement is done by usage of cv. Blob function cv. Update. Tracks()* * “high level tracking” A. Senior, A. Hampapur, Y-L Tian, L. Brown, S. Pankanti "Appearance Models for Occlusion Handling". R. Bolle Operational Procedures:

IFFAT for HCI 8) Tracking or event detection: § If the resultant blob’s colour corresponds to the colour for motion tracking, the centroid of the blob is retrieved and its equivalent position on a computer screen is calculated by computing a new position in terms of the movement of the centroid against the bounding box of the webcam’s vision or the R. O. I • Only one of the segmented frames is expected to contain a blob. • In case of a colour corresponding to a pre-programmed event, the corresponding event is triggered by the client Operational Procedures:

Results and Performance measurements

IFFAT for HCI Test machine Specifications §The proposed approach was tested on a standard laptop machine, dubbed C 1, with specification: CPUs Intel I 3 -3210 M @ 2. 5 GHz, having a built in HP True. Vision HD webcam supporting a resolution of upto 1280 x 800. §This machine was chosen to resemble a general home machine whose capabilities do not necessarily stand out in terms of processing power of the CPU or feature or resolution of the webcam Means of Performance Measurement: 1) 2) 3) 4) 5) Stress exhibited due to addition of beacon colour messages Latency exhibited during various experimenting scenarios Latency exhibited to detect change in beacon colour Rate of success in different lighting conditions Accuracy and ease of motion tracking.

1) Stress exhibited due to addition of beacon colourmessages

IFFAT for HCI 2) Latency exhibited during various experimenting scenarios Latencies observed were highly minimal and did not cause any hindrance to the operation of the system except with the exception of minor latency spikes seen during sudden exposure or increase in general ambient light 3) Latency exhibited to detect change in beacon colour Multiple messages, namely, right-click, clickhold and click-drop were detected without a noticeable latency. Detection and firing of events corresponding to beacon messages were smooth and error free. Results and efficiency measurements

Rates of success were measured by observing the number of events in which the client failed to detect the beacon. This was then compared to the total number of attempts performed in each respective lighting condition. 4) Rate of success in different lighting conditions

IFFAT for HCI Results and efficiency measurements 5) Accuracy and ease of motion tracking All tests performed indicated an error free translation of cursor as long as the beacon signal was being tracked. Ease of use was initially restricted as maximum working efficiency was obtained only when the beacon was held in a position that caused it to maximally face the webcam during motion towards the ends of the webcam’s sight. Once this was learnt by the user, operation did not prove to be difficult

Issues and possible fixes. LIMITS AND CONSTRAINTS OF USAGE

IFFAT for HCI Operating environment lighting dependence? § Indoor lighting • Well lit • Directly facing light sources • Spot lights over operation areas §Total darkness • Paradoxically less successful. • Minimized by Anti-reflective screens Issues, limits and constraints of usage Light reflected to webcam Light From computer screen Beacon Colour appears mixed

IFFAT for HCI Issues, limits and constraints of usage Operating style dependence? § Angle towards screen in large degrees § Beacon colours diminish and fail detection in these conditions 45°

Possible Applications EXTENSIONS AND FURTHER WORK

IFFAT for HCI Issues, limits and constraints of usage Applications? § A Non-touch central system usable by multiple users without involving a network § User interaction in situations where networks cannot be suitable deployed § Alternate means of easy HCI for commercial and non commercial purposes. Extensions? § Can be extended to multi user compatibilty by beacon frequency recognition § Auto calibration § Auto colour range specification

Questions?