EEC492592 Kinect Application Development Lecture 2 Wenbing Zhao

EEC-492/592 Kinect Application Development Lecture 2 Wenbing Zhao wenbing@ieee. org 1

Outline n Reminder q n n n No class next Monday: Martin Luther King Day Components of Kinect sensor Kinect for Windows SDK Building first Kinect application 2

Components of Kinect for Windows n n n Color camera Infrared (IR) emitter IR depth sensor Tilt motor Microphone array LED 3

The Color Camera n n Red, blue, green color images (RGB) Frame rate: q q n 30 FPS at 640 x 480 12 FPS at 1280 x 960 Viewing range: q 43 degrees vertical by 57 degrees horizontal 4

IR Emitter & IR Depth Sensor n n IR emitter emits IR light in a pseudo-random dot pattern Dotted light reflects off different objects IR depth sensor reads reflected light => depth info Depth stream resolution q q q 640 x 480 320 x 240 80 x 60 5

Depth Data Processing 6

Tilt Motor n n Used to change the camera and sensor’s angles to get the correct position of the human skeleton Can tilt vertically (upwards/downwards) up to 27 degrees 7

Microphone Array n n 4 different microphones: locate the direction of the audio wave, not just capture the sound Enhanced noise suppression, echo cancellation, beam-forming 8

LED n n n Placed between camera and IR emitter Used for indicating the status of the Kinect Green indicates Kinect drivers loaded properly 9

Kinect for Xbox vs. Kinect for Windows n Kinect for Windows q q q n Near mode: as close as 40 cm away from Kinect Can be used for commercial applications Can be used in a Win 7 virtual machine: Hyper. V, VMWare, Parallels For both q q Track motion up to 12 feet (4 meters) away Identical resolution 10

Kinect for Windows SDK n Supported operating systems q n Hardware requirements q q q n Windows 7, Windows 8, Windows Embedded 7 Dual core 2. 66 GHz or faster CPU Dedicated USB 2. 0 bus 2 GB or higher RAM Software requirement q q Microsoft Visual Studio 2010 or higher Kinect for Windows SDK: current version 1. 8 11

Kinect for Windows SDK 12

Kinect for Windows SDK 13

How Applications Interact with Kinect 14

Classification of Kinect SDK APIs 15

Kinect Driver n n The Kinect driver controls the camera, depth sensor, audio microphone array, and motion Data passes between the sensor and the app in the form of data streams q q q Color data stream Depth data stream Audio data stream Infrared data stream (for low light environment) Accelerameter data 16

The Near Mode n n Track a human upper body as close as 40 cm Only Kinect for Windows supports the near mode 17

Tracking Human Skeleton and Joint Movement n n n Kinect SDK does the magic to extract the skeleton and joint positions from the data streams received Can track up to two users at the same time with full joint positions Each skeleton consists of 20 joints 18

Building First Kinect App n n Modified Kinect. Info. Box app Steps q q q Create a new project: C# WPF application Adding Kinect reference Draw the GUI Modify Main. Window. xaml Adding code 19

Creating a New Project 20

Adding Kinect Libraries 21

Create User Interface 22

Modify Main. Window. xaml 23

Adding Code Setting up name space (Main. Window. xaml. cs) n q n using Microsoft. Kinect; Setting up state variable for the Kinect sensor q q An instance of Microsoft. Kinect. Sensor class It represents the complete runtime pipeline for the Kinect sensor during life cycle of the app public partial class Main. Window : Window { Kinect. Sensor sensor; // remaining code goes here } 24

Adding Code private void Window. Loaded(object sender, Routed. Event. Args e) { if (Kinect. Sensors. Count > 0) { this. sensor = Kinect. Sensors[0]; if (this. sensor != null && !this. sensor. Is. Running) { this. sensor. Start(); display. Info(); } } else { Message. Box. Show("No device is connected with system!"); this. Close(); } } private void Window. Closing(object sender, System. Component. Model. Cancel. Event. Args e) { if (this. sensor != null && this. sensor. Is. Running) { this. sensor. Stop(); } 25

Adding Code private void display. Info() { this. text. Block 1. Text = this. sensor. Device. Connection. Id; this. text. Block 2. Text = this. sensor. Status. To. String(); this. text. Block 3. Text = this. sensor. Elevation. Angle. To. String(); } Double click “Increase” button, the template for the button 1_Click() method will be automatically created private void button 1_Click(object sender, Routed. Event. Args e) { this. sensor. Elevation. Angle = this. sensor. Elevation. Angle + 1; display. Info(); } Double click “Decrease” button, the template for the button 2_Click() method will be automatically created private void button 2_Click(object sender, Routed. Event. Args e) { this. sensor. Elevation. Angle = this. sensor. Elevation. Angle - 1; display. Info(); } 26

More In-depth Stuff n n Starting Kinect sensor: sensor. Start() In the Start() method q q n First check the status of Kinect, if no sensor connected, runtime will throw an Invalid. Operation. Exception object with the Kinect. Not. Ready message Init the sensor if connected Init options q q q None: default option Use. Depth. And. Player. Index Use. Color Use. Skeleton. Tracking Use. Depth Use. Audio 27

More In-depth Stuff n Open data streams q q q n Color. Image. Stream: this. sensor. Color. Stream. Enable(); Depth. Image. Stream: this. sensor. Color. Stream. Enable(); Skeleton. Stream: this. sensor. Skeleton. Stream. Enable(); Stopping the Kinect sensor q q q this. sensor. Stop(); Stops the color, depth, and skeleton data stream Stops audio source, if open Kills all threads spawned Shuts down Kinect sensor and sets init option to None 28