Oculus Rift DK 2 Leap Motion Unity Tutorial

Oculus Rift DK 2 + Leap Motion Unity Tutorial Chris Zaharia @chrisjz

Content • Leap Motion VR • Hand tracking using Leap SDK V 2 (Skeletal Tracking) • Object interaction • Movement using Rift DK 2 positional tracking

Leap Motion SDK V 2 • Currently in beta • Introduces skeletal tracking

Leap Motion VR

Augmented Reality • Image API • Infrared & Night Vision • Enable in Leap Motion Control Panel > Settings > General > Allow Images • Leap Oculus Passthrough Unity Package • Next Leap Motion may have colour camera

Requirements • IDE • Unity Pro • Leap Motion Driver • Leap Motion V 2 Tracking Beta (v. 2. 1. 1. 21671+) • Leap Motion V 2 Skeletal Assets (beta) • Rift DK 2 • Unity 4 Pro Integration

Scope • Map physical hand finger movements to 3 D hand model and physics • Grab physics objects (rigidbodies) by pinching • Movement and jump using DK 2’s positional tracker

Leap Motion - Unity File Structure • Leap. Motion • Materials • Blinn 2. mat • Gloves. mat • Hands_Alt_blinn 1. mat • Models • Hands. Realistic. fbx • Prefabs • Hand Graphics • Rigid. Left. Hand. prefab • Rigid. Right. Hand. prefab • Hand Physics • Thick. Rigid. Hand. prefab • Hand. Controller. prefab

• Leap. Motion • … • Scripts • Hand • Finger. Model. cs • Hand. Model. cs • Rigged. Finger. cs • Rigged. Hand. cs • Rigid. Finder. cs • Rigid. Hand. cs • Skeletal. Finger. cs • Skeletal. Hand. cs • Tool. Model. cs • Utils • Leap. Recorder. cs • Leap. Unity. Extensions. cs • Magnetic. Pinch. cs • Hand. Controller. cs

• Plugins • [Include all files]

Attach hands to player • Assumed that you’ve already integrated Rift with player • Is compatible with Sixense’s Razer Hydra integration via script (see Leap. Hand. Extend. Controller. cs in tutorial)

1. Create empty Game. Object called “Leap. Hand. Controller” and place under Ovr. Camera. Controller. 2. Either attach the Hand. Controller prefab to game object or attach Hand. Controller. cs script. 3. Fill out variables with the following:

Setup Rigid. Left. Hand + Rigid. Right. Hand prefabs • Check if both prefabs are set as follows or change accordingly. Magnetic. Pinch. cs script allows user to grab rigidbodies by pinching fingers

Each finger must have the following variables, based on type of finger (i. e. index, pinky) and if it’s left (starts with L) or right (starts with R): For right hand, choose the [. . ]Arm 3 mesh

Setup Thick. Rigid. Hand prefab • Gives the hand collisions and physics • Can therefore push objects

Magnetic Pinch Script • Allows user to grab closest object which has a rigidbody • Force Spring Constant sets the elasticity of the grip to object • Magnetic Distance determines how close an object must be from a hand for it to be grabbed Warning: There is a conflict between Magnetic Pinch and enabling the Left/Right Physics Models. You’d need to fix this conflict by making the magnetic script ignore the hand’s physics

Grab Hand Grabbable scripts • As an alternative to Magnetic Pinch script • Assign Grab. Hand. cs to hand graphic prefab • Assign Grabbable. cs to rigidbody object to be grabbed

Hand Physics • Interact with objects in a realistic way • • Grab small objects with one hand Grab larger objects with multiple hands Push/Pull other objects Other possibilities • Grabbable objects must have colliders Currently, grasping objects with hands is still quite jittery. Future SDK or Leap hardware updates should improve on this hopefully.

Handle conflicts with other hand trackers • This approach will hide/disable Sixense hands if leap motion detects leap hands in scene • Start by extending Leap Motion’s Hand. Controller. cs

Leap. Hand. Extend. Controller. cs (1) using Unity. Engine; using System. Collections; using Leap; public class Leap. Hand. Extend. Controller : Hand. Controller { protected Controller leap_controller_; protected void Awake () { leap_controller_ = new Controller(); }

Leap. Hand. Extend. Controller. cs (2) protected void Late. Update () { if (leap_controller_ == null) return; Check. If. Hands. Enabled (); }

Leap. Hand. Extend. Controller. cs (3) protected void Check. If. Hands. Enabled () { Frame frame = leap_controller_. Frame(); Hand. List hands = frame. Hands; int num_hands = hands. Count; if (num_hands > 0) { Trigger. Sixense. Hands (false); } else { Trigger. Sixense. Hands (true); } }

Movement - DK 2 Positional Tracking • Move/Jump by using DK 2’s positional tracking • • Move forward or backward by moving head in those directions Either rotate or strafe sideways by moving head left/right Jump by popping head upwards Could crouch too by popping head downwards • User should normally be positioned directly in front of DK 2’s tracker

Logic - Positional Tracking Movement • Create/modify your player input controller script, attached to player • Head position is calculated by subtracting the initial local position of the main camera, of when scene is loaded, from its current position • Create configurable Vector 3 variables for: • Sensitivity – strength multiplier of movement or jump actions • Minimum – The minimum position that the Rift must be away from the centre position, for the action to actually be triggered • Movement will use the X (-left/+right) and Z axis (+forward/-backward) in the direction of the camera as a variable sent to the Character Motor’s input. Move. Direction variable • Jump will use the Y axis and will be mapped to the Character Motor’s input. Jump variable

Code – Positional Track Movement [Require. Component(typeof(Character. Motor))] public class FPSInput. Controller : Mono. Behaviour { … public bool ovr. Movement = false; // Enable move player by moving head on X and Z axis public bool ovr. Jump = false; // Enable player jumps by moving head on Y axis upwards public Vector 3 ovr. Control. Sensitivity = new Vector 3(1, 1, 1); // Multiplier of positiona tracking move/jump actions public Vector 3 ovr. Control. Minimum = new Vector 3(0, 0, 0); // Min distance of head from centre to move/jump public enum Ovr. XAxis. Action { Strafe = 0, Rotate = 1 } public Ovr. XAxis. Action ovr. XAxis. Action = Ovr. XAxis. Action. Rotate; // Whether x axis positional tracking performs strafing or rotation private Game. Object main. Camera; private Character. Motor motor; // OVR positional tracking, currently works via tilting head private Vector 3 init. Pos. Track. Dir; private Vector 3 cur. Pos. Track. Dir; private Vector 3 diff. Pos. Track. Dir; // Camera where movement orientation is done and audio listener enabled

Start + Update void Start() { … init. Pos. Track. Dir = main. Camera. transform. local. Position; } void Update() { // Get the input vector from OVR positional tracking if (ovr. Movement || ovr. Jump) { cur. Pos. Track. Dir = main. Camera. transform. local. Position; diff. Pos. Track. Dir = cur. Pos. Track. Dir - init. Pos. Track. Dir; }

Update (continued) … { if (ovr. Movement) { if (diff. Pos. Track. Dir. x <= -ovr. Control. Minimum. x || diff. Pos. Track. Dir. x >= ovr. Control. Minimum. x) if (ovr. XAxis. Action == Ovr. XAxis. Action. Strafe) { diff. Pos. Track. Dir. x *= ovr. Control. Sensitivity. x; } else { transform. Rotate(0, diff. Pos. Track. Dir. x * ovr. Control. Sensitivity. x, 0); diff. Pos. Track. Dir. x = 0; } } else { diff. Pos. Track. Dir. x = 0; }

Update (continued) … if (diff. Pos. Track. Dir. z <= -ovr. Control. Minimum. z || diff. Pos. Track. Dir. z >= ovr. Control. Minimum. z) { diff. Pos. Track. Dir. z *= ovr. Control. Sensitivity. z; } else { diff. Pos. Track. Dir. z = 0; } direction. Vector = new Vector 3(diff. Pos. Track. Dir. x, 0, diff. Pos. Track. Dir. z); }

Update (continued) if (ovr. Jump) { if (diff. Pos. Track. Dir. y > ovr. Control. Minimum. y) { motor. input. Jump = true; } else { motor. input. Jump = false; } } else { motor. input. Jump = Input. Get. Button ("Jump"); } … motor. input. Move. Direction = main. Camera. transform. rotation * direction. Vector; // performs actual movement