Chapter 3 Tracking Augmented Reality Principles and Practice

Chapter 3: Tracking Augmented Reality – Principles and Practice http: //www. augmentedrealitybook. org

Agenda • Tracking, Calibration, and Registration • Coordinate Systems • Characteristics • Stationary Tracking Systems • Mobile Sensors • Optical Tracking • Sensor Fusion www. augmentedrealitybook. org Tracking 2

Tracking, Calibration, and Registration • Registration = alignment of spatial properties • Calibration = offline adjustment of measurements • Spatial calibration yields static registration • Offline: once in lifetime or once at startup • Alternative: autocalibration • Tracking = dynamic sensing and measuring of spatial properties Calibration no calib nra pa spat tion tra ram ial of ck ete sp ing rs ati de of al vic es Tracking Dynamic registration tic tion a St tra gis re Registration • Tracking yields dynamic registration • Tracking in AR/VR always means “in 3 D”! www. augmentedrealitybook. org Tracking 3

Coordinate Systems Local object coordinates Eye coordinates Perspective transformation • Calibrate offline • For both camera and display Model transformation • Track for moving objects, if there are static objects as well View transformation • Track for moving objects, if there are no static objects • Track for moving observer Global world coordinates www. augmentedrealitybook. org Tracking 4

Measurement Coordinates • Global vs. local measurements • Global larger (or unlimited) workspace • Local better accuracy • Absolute vs. relative measurements • Absolute coordinate system defined in advance • Relative incremental sensing www. augmentedrealitybook. org Tracking 5

Physical Phenomena • Electromagnetic radiation • • • Visible light Infrared light Laser light Radio signals Magnetic flux • Sound • Physical linkage • Gravity • Inertia www. augmentedrealitybook. org Tracking 6

Measurement Principle • Signal strength • Signal direction • Time of flight • Absolute time • Signal phase • Requires synchronized clocks www. augmentedrealitybook. org Tracking 7

Degrees of Freedom (DOF) • DOF = independent dimension of measurement • Full tracking requires 6 DOF • 3 DOF position (x, y, z) • 3 DOF orientation (roll, pitch, yaw) • Some sensor deliver only a subset • E. g. , gyroscope 3 DOF orientation only • E. g. , tracked LED 3 DOF position only • E. g. , mouse 2 DOF position only www. augmentedrealitybook. org Tracking 8

Sensor arrangement • Multiple sensors in rigid geometric configuration • E. g. , stereo camera rig • Sparse or dense sensors • E. g. , digital camera is dense 2 D array of intensity sensor with know angles • Advanced technical issues • Sensor synchronization • Sensor fusion www. augmentedrealitybook. org Tracking 9

Sensor Group Arrangement Outside-in • Stationary mounted sensors • Good position, poor orientation www. augmentedrealitybook. org Inside-out • Mobile sensor(s) • Good orientation, poor position Tracking 10

Signal Sources • Passive sources • Natural signals • E. g. , natural light, earth magnetic field • Active sources • • Electronic components producing physical signal Can be direct or indirect (reflected) E. g. , acoustic, optical, radiowaves Most forms require open line of sight • No sources • Most important example: inertia www. augmentedrealitybook. org Tracking 11

Measurement error • Accuracy • How close is measurement to true value • Affected by systematic errors • Can be improved with better calibration • Precision • • How closely do multiple measurements agree (random error, noise) Varies per type of sensor Varies per degree of freedom Can be improved with filtering (more computation, more latency) • Resolution • Minimum difference that can be discriminated between two measurements • Cannot be reached in practice because of noise www. augmentedrealitybook. org Tracking 12

Temporal Characteristics • Update rate: • Number of measurements per time interval • Measurement latency • Time it takes from occurrence of physical event to data becoming available • End-to-end latency • Time it takes from occurrence of physical event to presentation of a stimulus www. augmentedrealitybook. org Tracking 13

Stationary Tracking Systems • Mechanical Tracking • Electromagnetic Tracking • Ultrasonic Tracking www. augmentedrealitybook. org Tracking 14

Mechanical Tracking • Track end-effector of articulated arm • Joints with 2 or 3 DOF • Rotary encoders or potentiometers • High precision • Fast • Freedom of operation limited www. augmentedrealitybook. org Tracking Cyber. Grask Fakespace BOOM 15

Electromagnetic Tracking • Stationary source produces three orthogonal magnetic fields • Current induced in sensor coils • Measurement of strength and phase of signal • Signal strength falls off quadratically with distance • Working range: half-sphere with 1 -3 m radius • Prolems with electromagnetic interference www. augmentedrealitybook. org Tracking Razer Hydra 16

Ultrasonic Tracking • Measures time of flight of sound pulse • Trilateration of 3 measurements • Requires synchronized time (cables) or more than 3 measurements • Low update rate (10 -50 Hz) due to slow speed of sound • Possible fusion with fast inertial sensors (e. g. , Inter. Sense IS-600) • Requires open line of sight • Suffers from noise or change of temperature • Wide-area configuration, e. g. , AT&T BAT system • Microphones mounted in ceiling www. augmentedrealitybook. org Tracking 17 Image: Joseph Newman

Mobile Sensors • Global Positioning System • Wireless Networks • Magnetometer • Gyroscope • Linear Accelerometer • Odometer www. augmentedrealitybook. org Tracking 18

Global Positioning System • Planet-scale outside-in radiowave time-of-flight • Requires clock synchronization • Must receive signals from at least 4 satellites GPS receiver www. augmentedrealitybook. org Tracking 19

Differential GPS • Compensate for atmospheric distortion • Receive correction signal from base station via network • Real-Time Kinematics (RTK) Differential GPS also uses signal phase satellites GPS receiver www. augmentedrealitybook. org correction signal Tracking base station 20

Wireless Networks • Measure signal strength from Wi. Fi, Bluetooth, mobile phone towers • Potential trilateration/triangulation • Mostly only good for coarse location (e. g. , based on Wi. Fi SSID) • Fingerprinting: carefully map the signal reception in a given area • Recent use: Bluetooth i. Beacon in department stores • Assisted GPS: accelerate GPS initialization using Wi. Fi or GSM id • Skyhood, Google, Broadcomm etc. www. augmentedrealitybook. org Tracking 21

Magnetometer • Electronic compass • Measure direction of Earth magnetic field in 3 D • Principle: magnetoresistance (Hall effect) • Often very distorted measurements www. augmentedrealitybook. org watch Tracking 22

Gyroscopes • Determines rotational velocity • Electronic gyro • • • Radial movement Coriolis movement Image: Hideyuki Tamura Measures Coriolis force of small vibrating object Micro-electromechanical system (MEMS) High update rate (1 KHz) Only relative measurements Must integrate once to determine orientation drift • Laser gyro (fiber-optic gyro) • Measures angular acceleration based on light interference • Large, expensive, used in aviation www. augmentedrealitybook. org Tracking rotation 23

Linear accelerometer • MEMS device • Displacement of small mass • Measures spring mass • Change of electric capacity, or • Piezoresistive effect of bending • Subtract gravity (the difficult part!) • Integrate twice numerically to get position mass • Drift problems • Combine lin. acc. , gyro + compass into inertial measurement unit (IMU) www. augmentedrealitybook. org Tracking 24

Odometer • Mechanical or opto-electrical wheel encoder • E. g. , traditional ball mouse www. augmentedrealitybook. org Tracking 25

Optical Tracking • Optical sensors • Model-Based versus Model-Free Tracking • Illumination • Markers versus Natural Features • Target Identification www. augmentedrealitybook. org Tracking 26

Optical Sensors • Digital cameras are cheap and powerful • CCD (charge coupled devices) – professional photography • CMOS (complementary metal oxide semiconductor) – fast and cheap • Computer vision techniques improve with Moore’s law • Lenses are becoming the most limiting part Bayer pattern www. augmentedrealitybook. org Tracking 27

Model-Based versus Model-Free Tracking Model-based • A tracking model representing the 3 D world is available • Compare the model to observations in the images www. augmentedrealitybook. org Model-free • At start-up, no tracking model is available • Most build a temporary tracking model while tracking • Measurements only relative to starting point Tracking 28

ARTTrack Illumination • Passive illumination • Natural (or existing) light sources • Visible spectrum 380 -780 nm • Cannot track when it is too dark (mostly indoors) • Active illumination • • Often infrared spectrum LED beacons Camera with infrared filter delivers high contrast Not suitable with sunlight • Structured light • Project a known pattern into the scene • Projector with regular light or laser • Laser ranging Microsoft Kinect V 1 • Measure time of flight taken by laser pulse • Steerable MEMS mirror for scanning laser • LIDAR (light radar): long range laser used in surveying www. augmentedrealitybook. org Tracking 29

Leap Motion • 2 cameras, 3 infrared LEDs • Short-distance reflection of the hands www. augmentedrealitybook. org Tracking 30

Valve/HTC Vive • “Lighthouses” = two scanning infrared lasers • Photodiodes on head pick up lasers www. augmentedrealitybook. org Tracking 31

Markers vs Natural Features • Fiducials markers • • Artificial tracking targets Square shapes yield 4 points (enough for pose ) Circular shapes yield only 1 point Digital marker model exists first, marker manufactured second (e. g. , printing) Image: Daniel Wagner • Natural feature tracking • Existing visual features in the environment • Physical features exist first, tracking model reconstructed second www. augmentedrealitybook. org Tracking 32 Image: Andrei State, UNC Chapel Hill

Flat Marker Designs Image: Daniel Wagner www. augmentedrealitybook. org Tracking 33

Retro-Reflective Ball Markers • Light reflected towards light-source • Illuminate with infrared LED flash • Infrared camera observes bright blobs • 4 or more spheres in known configuration to recover 6 DOF pose • Multiple targets distinguished by their geometric configuration www. augmentedrealitybook. org Tracking 34 Advanced Realtime Tracking Gmb. H

Image: Martin Hirzer Natural Features • Detect salient interest points in image • Must be easily found • Location in image should remain stable when viewpoint changes • Requires textured surfaces • Alternative: can use edge features (less discriminative) • Match interest points to tracking model database • Database filled with results of 3 D reconstruction • Matching entire (sub-)images is too costly • Typically interest points are compiled into “descriptors” www. augmentedrealitybook. org Tracking Image: Gerhard Reitmayr 35

Marker Target Identification • More targets or features more easily confused • Must be as unique as possible Image: Mark Fiala • Square markers • 2 D barcodes with error correction • E. g. , 6 x 6=36 bits (2 orientation, 6 -12 payload, rest for error correction) • Marker tapestries • Spherical targets • 5 spheres in different geometric configurations • Can distinguish 10 -20 targets • Pulsed LEDs www. augmentedrealitybook. org Image: Greg Welch, UNC Chapel Hill Tracking 36

Natural Feature Target Identification • Individual natural interest points too easily confused • Rely on co-occurency of interest points for detection • Probabilistic search methods used to deal with errors • Vocabulary trees • Random sampling consensus www. augmentedrealitybook. org Image: Martin Hirzer Tracking 37

Complementary Sensor Fusion • Combining sensors with different degrees of freedom • Sensors must be synchronized (or requires inter-/extrapolation) • E. g. , combine position-only and orientation-only sensor • E. g. , orthogonal 1 D sensors in gyro or magnetometer are complementary www. augmentedrealitybook. org Tracking 38

Competitive Sensor Fusion • Different sensor types measure the same degree of freedom • Redundant sensor fusion • Use worse sensor only if better sensor is unavailable • E. g. , GPS + pedometer • Statistical sensor fusion (see next slide) www. augmentedrealitybook. org Tracking 39

Statistical Sensor Fusion • Important form of competitive fusion for higher quality • Combine measurement to improve quality • Establish statistical estimate of the true system state • Predict future system state Correct from observation (measurement) • Extended Kalman filter for Gaussian error distribution • Unscented Kalman filter for highly non-linear systems • Particle filter for systems with multiple state hypothesis • E. g. , maintain estimate with fast IMU + update when slow computer vision results come in www. augmentedrealitybook. org Tracking 40

Cooperative Sensor Fusion • Primary sensor relies on information from secondary sensor to obtain its measurements • E. g. , A-GPS combines celltower + GPS • Combination of inside-out + outside-in • Stereo cameras with known epipolar geometry • Non-overlapping cameras (e. g. , 360°) • Indirect sensing (cont’d) www. augmentedrealitybook. org Image: Georg Klein Tracking Point. Grey Lady. Bug 41

Cooperative Sensor Fusion for Indirect Sensing “Track around the corner” C 1 C 2 A B www. augmentedrealitybook. org Tracking 42

Tracking-based AR applications A. Pattern-recognition based approaches 1. AR toolkit, demo (Magic Book) 2. 3 D model retrieval, demo (NTU light-field 3 D) 3. Eye-tracking (Tobii eye-tracker) demo B. Vision-based optical tracking 1. UNC optical tracker (1989 demo, Church’s algorithm) demo 2. Mocap devices demo Communications & Multimedia Lab 43

C. Magnetism-based tracking: 1. Polhemus tracker (demo) (Chinese: 鱟魚) (demo 2) Gyroscope + linear/angular accelerators 2. Xsens motion capture (demo) (demo 2) (demo 3) D. Kinect-based tracking: 1. Human body skeleton tracking (demo) 2. Human-head motion tracking (demo) E. Acoustic tracking system Ultrasound tech (sound speed: shoot and rebound) Communications & Multimedia Lab 44

Optical Trackers: Environment setup (“A Real-time Optical Tracker For Head-Mounted Display Systems”, I 3 D, by JF Wang, 1990) • Outside-in vs Inside-out optical trackers • Environment LED lights are placed on the ceiling of a room Communications & Multimedia Lab 45

Algorithms: Inferring 3 D position • Inferring 3 D position from 2 D image • 12 unknowns in M, each 3 D point provides 2 independent constrains => 6 points Communications & Multimedia Lab 46

Algorithms: Inferring 3 D position Church(1945) method: modified by iterative converging. Communications & Multimedia Lab 47

How to calculate partial derivatives? Numerical difference approximation {X = X + 0. 1 (cm, for example) X = X - 0. 1 Y = Y + 0. 1, Y = Y – 0. 1 Z = Z + 0. 1, Z = Z – 0. 1} Then choose the one that can best reduce the angle difference || angel (a. Ob) – angel (AOB)|| in the previous page. Adaptive method: how about X = X + 0. 01 (cm)? , for better results www. augmentedrealitybook. org Tracking 48

Algorithms: System errors • Result • In camera coordinate • long focal length • large separation between image points • high resolution of photodiode • In world coordinate • large separation between beacons Communications & Multimedia Lab 49

Algorithms: Multiple views Communications & Multimedia Lab 50

Performance Evaluation in 1990, using old machines) (when it was first invented Communications & Multimedia Lab 51

Homework 1 (see the following two slides) • Goal: To detect the 3 D position of three markers placed in a room. • Method: Using your smartphone or Notebook PC camera, together with the above iterative algorithm, deadline: April 23 www. augmentedrealitybook. org Tracking 52

Three markers (A, B, C), camera f/1. 7, 26 mm, 4: 3 ratio, 1080 x 2340 (display) pixels, 15. 73 x 7. 49 cm in dimension, Oppo Reno Z A typical camera, with f/1. 7, 20 mm, Panasonic Lumix G www. augmentedrealitybook. org Tracking 53

Long rage, short range, photos Long range www. augmentedrealitybook. org Tracking Short range 54

Camera configuration (a typical figure, homework 1) www. augmentedrealitybook. org Tracking 55

For homework 1: converted camera parameters, after calibration • Assuming the smartphone camera has a photo of 4000 x 3000 in resolution, with a scaling factor: W for display (the camera has a very small focus, such as a 26 mm built-in zoom lens, and a very small CMOS image sensor). Typically, 4 x 3 ratio for a photo displayed in a Notebook PC, with width W*14 cm, height W*10. 5 cm. The S’ distance (camera lens to image sensor distance) will be W*10. 4 cm, in our given photo. For instance, if W = 0. 048, S’ = 5 mm. (when you are looking at a photo in your notebook PC, or a smartphone, W can be 1. 0 for NB display) Now, you can work on homework 1, using the above three photos as test input. What are the values of an object (with markers) to camera lens distance S? www. augmentedrealitybook. org Tracking 56