EEC492592 Kinect Application Development Lecture 14 Wenbing Zhao

EEC-492/592 Kinect Application Development Lecture 14 Wenbing Zhao wenbing@ieee. org

Outline n Unity 3 D + ZDK

Creating a New Unity Project n n Open Unity, create a new project, choose a location folder of your choice Important ZDK plugin q q q Go to Assets menu, select “Import Package”, then “Custom Package…” In the dialogue, find and select the ZDK plugin and click OK: ZDK_Unity 40_1. 1_trial. unitypack age In the “Import package” dialogue, keep default selection and click Import button

Project Layout after ZDK is imported

Scene View n Where you plan and execute your ideas q To show the grid, toggle off the game overlay button

Hierarchy View n n n Shows what’s in the currently active scene. Game. Objects that are dynamically added and removed from the scene during runtime will appear here when they are active in the scene By default, the main camera is added

Constructing Scene n Add the following items to the scene: q q Avatar: Dana (from ZDK) n In Assets from Project panel, go to Zig. Fu folder, then _Data folder n Drag and drop Dana@t-pose_3 to the Hierarchy panel n Rotation 180 degree so that it faces front Floor n Add a Cube game object, change its size to a thin layer q n n q q Change material to floor (imported from ZDK) Use Inspector, to go Mesh Renderer section, then materials => Element 0 => browse => choose Floor Directional light Empty object: to enable bootstrapping with ZDK n n n Scale: x=30, y=0. 1, z=30 From Game. Object menu, choose Create Empty Rename it to Init. ZDK (right click the object, then Rename) Adjust main camera so that Dana almost fill the entire game view

Connecting Existing Scripts n n n Change Project panel to one column Expand Zig. Fu folder, then expand Scripts folder and all subfolders Locate Zig script, drag and drop it to Init. ZDK in the Hierarchy panel Locate Zig. Depth. Viewer script under the Viewers subfoler to Init. ZDK Locate Zig. Users. Radar script under the Viewers subfoler to Init. ZDK Locate Zig. Engage. Single. User script under the User. Engagers subfoler to Init. ZDK q n n Drag and drop Dana from the Hierarchy panel to Engaged Users section under Zig Engage Singler User in the inspector Locate kinect. Specific script under the _Internal subfoler to Init. ZDK Locate Zig. Skeleton script under the User. Controls subfoler to Dana

After Connecting Existing Scripts

n n Connecting Joints for Dana the Avatar Expand the entire hierarchy of Dana mesh Drag and drop matching joints from the Dana mesh in the hierarchy panel to the appropriate joints under Zig Skeleton section in the inspector q q q q q n Head => Head Left Hip => Left. Up. Leg Neck => Neck Left Knee=> Left. Leg Torso => Spine 1 Left Ankle => Left. Foot Waist => Spine Right Hip => Right. Up. Leg Left Shoulder => Left. Arm Right Knee => Right. Leg Left Elbow => Left. Fore. Arm Right Ankle => Right. Foot Left Wrist => Left Hand Right Shoulder => Right. Arm Right Elbow => Right. Fore. Arm Right Wrist => Right. Hand Check the Mirror checkbox in the Dana inspector

Connecting Joints for Dana the Avatar

What Do the Scripts Do? n Zig. cs q q Take user settings on what to update, smoothing, and smoothing parameters Public member variables can be set via the inspector public class Zig : Mono. Behaviour { public Zig. Input. Type input. Type = Zig. Input. Type. Auto; public Zig. Input. Settings settings = new Zig. Input. Settings(); public List<Game. Object> listeners = new List<Game. Object>(); public bool Verbose = true;

Zig. Depth. Viewer. cs n n On. GUI(): called for rendering and handling GUI events static void Draw. Texture(Rect position, Texture image, Scale. Mode scale. Mode = Scale. Mode. Stretch. To. Fill, bool alpha. Blend = true, float image. Aspect = 0); q q q Position: Rectangle on the screen to draw the texture within. Image: Texture to display. scale. Mode: How to scale the image when the aspect ratio of it doesn't fit the aspect ratio to be drawn within. alpha. Blend: Whether to apply alpha blending when drawing the image (enabled by default). image. Aspect: Aspect ratio to use for the source image void On. GUI() { if (null == target) { GUI. Draw. Texture(new Rect(Screen. width - texture. width - 10, Screen. height - texture. height - 10, texture. width, texture. height), texture); } }

Zig. Depth. Viewer. cs n n Texture: the visual and especially tactile quality of a surface. Parent class for Texture 2 D reference q n http: //docs. unity 3 d. com/Documentation/Script. Reference/Tex ture 2 D. html You can also add a button void On. GUI() { if (GUI. Button(new Rect(10, 150, 100), "I am a button")) print("You clicked the button!"); }

Zig. Depth. Viewer. cs n Getting input from Kinect void Zig_Update(Zig. Input input) { if (Use. Histogram) { Update. Histogram(Zig. Input. Depth); // Zig. Input. Depth contains the depth data } else { depth. To. Color[0] = Color. black; for (int i = 1; i < Max. Depth; i++) { float intensity = 1. 0 f - (i/(float)Max. Depth); depth. To. Color[i]. r = (byte)(Base. Color. r * intensity); depth. To. Color[i]. g = (byte)(Base. Color. g * intensity); depth. To. Color[i]. b = (byte)(Base. Color. b * intensity); depth. To. Color[i]. a = 255; } } Update. Texture(Zig. Input. Depth); }

Zig. Depth. Viewer. cs n Update Depth Image void Update. Texture(Zig. Depth depth) { short[] raw. Depth. Map = depth. data; int depth. Index = 0; int factor. X = depth. xres / texture. Size. Width; int factor. Y = ((depth. yres / texture. Size. Height) - 1) * depth. xres; // invert Y axis while doing the update for (int y = texture. Size. Height - 1; y >= 0 ; --y, depth. Index += factor. Y) { int output. Index = y * texture. Size. Width; for (int x = 0; x < texture. Size. Width; ++x, depth. Index += factor. X, ++output. Index) { output. Pixels[output. Index] = depth. To. Color[raw. Depth. Map[depth. Index]]; } } texture. Set. Pixels 32(output. Pixels); texture. Apply(); }

Zig. User. Radar. cs: Track where the user is void On. GUI () { if (!Zig. Input. Instance. Reader. Inited) return; int width = (int)((float)Pixels. Per. Meter * (Radar. Real. World. Dimensions. x / 1000. 0 f)); int height = (int)((float)Pixels. Per. Meter * (Radar. Real. World. Dimensions. y / 1000. 0 f)); GUI. Begin. Group (new Rect (Screen. width - 20, width, height)); Color old. Color = GUI. color; GUI. color = box. Color; GUI. Box(new Rect(0, 0, width, height), "Users Radar", style); GUI. color = old. Color; foreach (Zig. Tracked. User current. User in Zig. Input. Instance. Tracked. Users. Values) { // normalize the center of mass to radar dimensions Vector 3 com = current. User. Position; Vector 2 radar. Position = new Vector 2(com. x / Radar. Real. World. Dimensions. x, -com. z / Radar. Real. World. Dimensions. y); // X axis: 0 in real world is actually 0. 5 in radar units (middle of field of view) radar. Position. x += 0. 5 f; radar. Position. x = Mathf. Clamp(radar. Position. x, 0. 0 f, 1. 0 f); radar. Position. y = Mathf. Clamp(radar. Position. y, 0. 0 f, 1. 0 f); Color orig = GUI. color; GUI. color = (current. User. Skeleton. Tracked) ? Color. blue : Color. red; GUI. Box(new Rect(radar. Position. x * width - 10, radar. Position. y * height - 10, 20), current. User. Id. To. String()); GUI. color = orig; } GUI. End. Group(); }

Zig. Engage. Single. User. cs n Connect Zig. Input to the avatar: that is why you must drag and drop Dana to Engaged. Users field public class Zig. Engage. Single. User : Mono. Behaviour { public bool Skeleton. Tracked = true; public bool Raise. Hand; public List<Game. Object> Engaged. Users; void Start() { // make sure we get zig events Zig. Input. Instance. Add. Listener(game. Object); } void Zig_Update(Zig. Input zig) { if (Skeleton. Tracked && null == engaged. Tracked. User) { foreach (Zig. Tracked. User tracked. User in zig. Tracked. Users. Values) { if (tracked. User. Skeleton. Tracked) { Engage. User(tracked. User); } }

Zig. Engage. Single. User. cs void Engage. User(Zig. Tracked. User user) { if (null == engaged. Tracked. User) { engaged. Tracked. User = user; foreach (Game. Object go in Engaged. Users) user. Add. Listener(go); Send. Message("User. Engaged", this, Send. Message. Options. Dont. Require. Receiver); } } Component. Send. Message: void Send. Message(string method. Name, object value = null, Send. Message. Options options = Send. Message. Options. Require. Receiver); // Calls the method named method. Name on every Mono. Behaviour in this game object

kinect. Specific. cs: Kinect specific settings void On. GUI() { long. Word = GUI. Text. Field(new Rect(10, 200, 30), reading. Angle ? get. Angle(). To. String() : long. Word, 20); if (GUI. Button(new Rect(10, 40, 200, 30), "Set. Elevation")) { angle = int. Parse(long. Word); Nui. Wrapper. Nui. Camera. Elevation. Set. Angle(angle); t = new Thread(set. Angle); //attempted a Paramaterized Thread to no avail t. Start(); Thread. Sleep(0); } reading. Angle = GUI. Toggle(new Rect(10, 80, 200, 30), reading. Angle, "Read Angle"); bool n. Near. Mode = GUI. Toggle(new Rect(10, 160, 20), Near. Mode, "Near Mode"); if (n. Near. Mode != Near. Mode) { Near. Mode = n. Near. Mode; Zig. Input. Instance. Set. Near. Mode(Near. Mode); } bool n. Seated. Mode = GUI. Toggle(new Rect(10, 190, 20), Seated. Mode, "Seated Mode"); bool n. Track. Skeleton. In. Near. Mode = GUI. Toggle(new Rect(10, 220, 20), Track. Skeleton. In. Near. Mode, "Track Skeleton In Near. Mode"); if ((n. Seated. Mode != Seated. Mode) || (Track. Skeleton. In. Near. Mode != n. Track. Skeleton. In. Near. Mode)) { Seated. Mode = n. Seated. Mode; Track. Skeleton. In. Near. Mode = n. Track. Skeleton. In. Near. Mode; Zig. Input. Instance. Set. Skeleton. Tracking. Settings(Seated. Mode, Track. Skeleton. In. Near. Mode); } }

Zig. Skeleton. cs public class Zig. Skeleton : Mono. Behaviour { public Transform Head; public Transform Neck; public Transform Torso; public Transform Waist; public Transform Left. Collar; public Transform Left. Shoulder; public Transform Left. Elbow; public Transform Left. Wrist; public Transform Left. Hand; public Transform Left. Fingertip; public Transform Right. Collar; public Transform Right. Shoulder; public Transform Right. Elbow; public Transform Right. Wrist; public Transform Right. Hand; public Transform Right. Fingertip; public Transform Left. Hip; public Transform Left. Knee; public Transform Left. Ankle; public Transform Left. Foot; public Transform Right. Hip; public Transform Right. Knee; public Transform Right. Ankle; public Transform Right. Foot; public bool mirror = false; public bool Update. Joint. Positions = false; public bool Update. Root. Position = false; public bool Update. Orientation = true; public bool Rotate. To. Psi. Pose = false; public float Rotation. Damping = 30. 0 f; public float Damping = 30. 0 f; public Vector 3 Scale = new Vector 3(0. 001 f, 0. 001 f); public Vector 3 Position. Bias = Vector 3. zero; private Transform[] transforms; private Quaternion[] initial. Rotations; private Vector 3 root. Position; Quaternions are used to represent rotations, they are based on complex numbers and are not easy to understand intuitively

Zig. Skeleton. cs Zig. Joint. Id mirror. Joint(Zig. Joint. Id joint) { switch (joint) { case Zig. Joint. Id. Left. Collar: return Zig. Joint. Id. Right. Collar; case Zig. Joint. Id. Left. Shoulder: return Zig. Joint. Id. Right. Shoulder; case Zig. Joint. Id. Left. Elbow: return Zig. Joint. Id. Right. Elbow; case Zig. Joint. Id. Left. Wrist: return Zig. Joint. Id. Right. Wrist; case Zig. Joint. Id. Left. Hand: return Zig. Joint. Id. Right. Hand; case Zig. Joint. Id. Left. Fingertip: return Zig. Joint. Id. Right. Fingertip; case Zig. Joint. Id. Left. Hip: return Zig. Joint. Id. Right. Hip; …. . // map right to left default: return joint; } }

Zig. Skeleton. cs public void Awake() { int joint. Count = Enum. Get. Names(typeof(Zig. Joint. Id)). Length; transforms = new Transform[joint. Count]; initial. Rotations = new Quaternion[joint. Count]; transforms[(int)Zig. Joint. Id. Head] = Head; transforms[(int)Zig. Joint. Id. Neck] = Neck; transforms[(int)Zig. Joint. Id. Torso] = Torso; transforms[(int)Zig. Joint. Id. Waist] = Waist; …. . // save all initial rotations // NOTE: Assumes skeleton model is in "T" pose since all rotations are relative to that pose foreach (Zig. Joint. Id j in Enum. Get. Values(typeof(Zig. Joint. Id))) { if (transforms[(int)j]) { // we will store the relative rotation of each joint from the gameobject rotation // we need this since we will be setting the joint's rotation (not local. Rotation) but we // still want the rotations to be relative to our game object initial. Rotations[(int)j] = Quaternion. Inverse(transform. rotation) * transforms[(int)j]. rotation; } } }

Zig. Skeleton. cs void Zig_Update. User(Zig. Tracked. User user) { Update. Root(user. Position); if (user. Skeleton. Tracked) { foreach (Zig. Input. Joint joint in user. Skeleton) { if (joint. Good. Position) Update. Position(joint. Id, joint. Position); if (joint. Good. Rotation) Update. Rotation(joint. Id, joint. Rotation); } } } void Update. Root(Vector 3 skel. Root) { // +Z is backwards in Open. NI coordinates, so reverse it root. Position = Vector 3. Scale(new Vector 3(skel. Root. x, skel. Root. y, skel. Root. z), do. Mirror(Scale)) + Position. Bias; if (Update. Root. Position) { transform. local. Position = (transform. rotation * root. Position); } } Vector 3. Scale(Vector 3 a, Vector 3 b): Multiplies two vectors component-wise.

Zig. Skeleton. cs void Update. Rotation(Zig. Joint. Id joint, Quaternion orientation) { joint = mirror ? mirror. Joint(joint) : joint; // make sure something is hooked up to this joint if (!transforms[(int)joint]) { return; } if (Update. Orientation) { Quaternion new. Rotation = transform. rotation * orientation * initial. Rotations[(int)joint]; if (mirror) { new. Rotation. y = -new. Rotation. y; new. Rotation. z = -new. Rotation. z; } transforms[(int)joint]. rotation = Quaternion. Slerp(transforms[(int)joint]. rotation, new. Rotation, Time. delta. Time * Rotation. Damping); } static Quaternion Slerp(Quaternion from, Quaternion to, float t); } // Spherically interpolates between from and to by t. void Update. Position(Zig. Joint. Id joint, Vector 3 position) { joint = mirror ? mirror. Joint(joint) : joint; if (!transforms[(int)joint]) { return; } if (Update. Joint. Positions) { Vector 3 dest = Vector 3. Scale(position, do. Mirror(Scale)) - root. Position; //Vector 3. Lerp: Linearly interpolates between two vectors. transforms[(int)joint]. local. Position = Vector 3. Lerp(transforms[(int)joint]. local. Position, dest, Time. delta. Time * Damping); } }