DIGITAL DANCING ETHAN BROWN KATE RAFTER What I

DIGITAL DANCING ETHAN BROWN / KATE RAFTER

What I learned…

SEEING THE WORLD IN THREE DIMENSION Humans are good at understanding depth from a multitude of cues: DEPTH PARALLAX STEREOPSIS PERSPECTIVE RELATIVE SIZE FAMILIAR SIZE OCCLUSION CONVERGENCE MANY MORE…. Computers: not so much.

INFRARED DEPTH MEASUREMENT: A CLEVER CHEAT § Infrared laser projects a pattern of infrared dots § Depth inferred from multiple cues § Kinect uses proprietary technology developed by Prime. Sense

HARDWARE Occipital Microsoft KINECT 1 KINECT 2 STRUCTURE SENSOR

SKELETON TRACKING § Complicated problem § Uses techniques from artificial intelligence, machine learning, and computer vision § Requires massive amounts of training data

SOFTWARE TOOLKITS § Open. Frameworks (C++) § Cinder (C++) § Kinect. NET SDK (C#) § Point Cloud Library (C++) § Open. NI (C++) § Open. CV (C++)

NODE / JAVASCRIPT § Edge. js makes Kinect SDK available § THREE. js / Canvas 3 D / Cubic. VR § Distinct lack of libraries (Open. CV, PCL)

DISPLAYING POINT DATA § Open. Frameworks / Open. GL § Kinect provides world coordinates; store as vectors (of. Vec 3 f) § Convert point cloud vectors to mesh (of. Mesh) § Some effects “accidental” artifacts of mesh rendering

FLOOR REMOVAL § Plane detection with RANSAC algorithm, courtesy PCL § Hooray math! Point/plane distance § Expensive filtering: optimization pays

“EXPLODING” SHADOWS § Easy centroid calculation: § Subtract centroid from each point and normalize: unit vector away from centroid § Scale unit movement vector and add to point to move away from centroid

KATE RAFTER ON TECHNOLOGY AND PERFORMANCE

LEARNING MORE § Trigonometry (Kahn Academy) § Linear Algebra (Kahn Academy) th § The C++ Programming Language, 4 Edition § Making Things See by Greg Borenstein § Books on Open. Frameworks and Cinder from Pact Publishing

ONWARD…. § Projector/real-world correspondence § Identify real-world features (with depth and/or visual data)

THANK YOU www. popart. com § ethan. brown@popart. com § @Ethan. RBrown § k. e. rafter@gmail. com § www. katerafter. com zepln. com/wdne