Overview of coded light projection techniques for automatic

Overview of coded light projection techniques for automatic 3 D profiling Computer Vision and Robotics Group Institut d’Informàtica i Aplicacions Jordi Pagès Joaquim Salvi

Presentation outline Formació Introduction Classification Experiments Sub-pixel matching • Introduction • Coded pattern classification – Time-multiplexing – Spatial codification – Direct codification • Experimental results • Sub-pixel matching • Conclusions & guidelines Conclusions 1/36 2

Introduction: shape acquisition Shape acquisition techniques Introduction Shape acquisition Non-contact Contact Stereovision Encoding/ decoding Reflective Non-destructive Transmissive Destructive Non-optical Classification CMM Jointed arms Industrial CT Slicing Microwave radar Sonar Optical Active Experiments Passive Stereo Imaging radar Sub-pixel matching Shape from X Triangulation Motion Stereo Shading Silhouettes Texture Interferometry Conclusions Moire Source: Brian Curless (Coded) Structured light Holography 2/36 3

Introduction: passive stereovision Introduction Shape acquisition Stereovision Encoding/ decoding Classification • Correspondence problem • geometric constraints search along epipolar lines • 3 D reconstruction of matched pairs by triangulation Experiments Sub-pixel matching Conclusions 3/36 4

Introduction: active stereo (coded structured light) Introduction Shape acquisition Stereovision Encoding/ decoding Classification • One of the cameras is replaced by a light emitter • Correspondence problem is solved by searching the pattern in the camera image (pattern decoding) Experiments Sub-pixel matching Conclusions 4/36 5

Introduction: pattern encoding/decoding (I) Introduction • Shape acquisition A pattern is encoded when after projecting it onto a surface, a set of regions of the observed projection can be easily matched with the original pattern. Example: pattern with two-encoded-columns Stereovision Encoding/ decoding Classification Pixels in red and yellow are directly matched with the pattern columns Object scene Codification using colors Experiments Sub-pixel matching Conclusions • The process of matching an image region with its corresponding pattern region is known as pattern decoding similar to searching correspondences • Decoding a projected pattern allows a large set of correspondences to be easily found thanks to the a priori knowledge of the light pattern 5/36 6

Introduction: pattern encoding/decoding (II) Introduction Shape acquisition • Stereovision Two ways of encoding the correspondences: single and double axis codification it determines how the triangulation is calculated Single-axis encoding Encoding/ Double-axis encoding decoding Classification Triangulation by line-to-plane intersection Triangulation by line-to-line intersection Experiments Sub-pixel matching Conclusions • Decoding the pattern means locating points in the camera image whose corresponding point in the projector pattern is a priori known 6/36 7

Coded structured light patterns: a classification proposal Introduction Classification Timemultiplexing Spatial codification Direct codification Experiments Sub-pixel matching Conclusions 7/36 8

Time-multiplexing Introduction • Classification • Timemultiplexing Spatial codification Example: 3 binaryencoded patterns which allows the measuring surface to be divided in 8 sub-regions The time-multiplexing paradigm consists of projecting a series of light patterns so that every encoded point is identified with the sequence of intensities that receives The most common structure of the patterns is a sequence of stripes increasing its length by the time singleaxis encoding Projected over time Direct codification Experiments • Sub-pixel matching • Conclusions Advantages: – high resolution a lot of 3 D points – High accuracy (order of m) – Robustness against colorful objects since binary patterns can be used Pattern 3 Pattern 2 Drawbacks: – Static objects only – Large number of patterns Pattern 1 8/36 9

Time-multiplexing: binary codes (I) Introduction Formació • Coding redundancy: every edge between adjacent stripes can be decoded by the sequence at its left or at its right Classification Timemultiplexing Spatial codification Direct codification Experiments Sub-pixel matching Conclusions 9/36 10

Time-multiplexing: Binary codes (II) Introduction • Every encoded point is identified by the sequence of intensities that receives Classification • n patterns must be projected in order to encode 2 n stripes Timemultiplexing Example: 7 binary patterns proposed by Posdamer & Altschuler Projected over time Spatial codification Direct codification Experiments … Pattern 3 Pattern 2 Sub-pixel matching Conclusions Pattern 1 Codeword of this píxel: 1010010 identifies the corresponding pattern stripe 10/36 11

Time-multiplexing: n-ary codes (I) Introduction • Classification Timemultiplexing • n-ary codes reduce the number of patterns by increasing the number of projected intensities (grey levels/colours) increases the basis of the code The number of patterns, the number of grey levels or colours and the number of encoded stripes are strongly related fixing two of these parameters the reamaining one is obtained Spatial codification Direct codification Using a binary code, 6 patterns are necessary to encode 64 stripes Experiments Sub-pixel matching Conclusions 3 patterns based on a n-ary code of 4 grey levels (Horn & Kiryati) 64 encoded stripes 11/36 12

Time-multiplexing: n-ary codes (II) Introduction • n-ary codes reduce the number of patterns by increasing the number of projected intensities (grey levels/colours) Classification Timemultiplexing Spatial codification Direct codification Experiments Sub-pixel matching Conclusions 12/36 13

Time-multiplexing: Gray code + Phase shifting (I) Introduction • A sequence of binary patterns (Gray encoded) are projected in order to divide the object in regions Classification Example: three binary patterns divide the object in 8 regions Timemultiplexing Spatial codification Direct codification Experiments Sub-pixel matching Conclusions • An additional periodical pattern is projected • The periodical pattern is projected several times by shifting it in one direction in order to increase the resolution of the system similar to a laser scanner Gühring’s line-shift technique Without the binary patterns we would not be able to distinguish among all the projected slits Every slit always falls in the same region 13/36 14

Time-multiplexing: Gray code + Phase shifting (II) Introduction Classification • A periodical pattern is shifted and projected several times in order to increase the resolution of the measurements • The Gray encoded patterns permit to differentiate among all the periods of the shifted pattern Timemultiplexing Spatial codification Direct codification Experiments Sub-pixel matching Conclusions 14/36 15

Time-multiplexing: hybrid methods (I) • In order to decode an illuminated point it is necessary to observe not only the Introduction Formació Classification sequence of intensities received by such a point but also the intensities of few (normally 2) adjacent points • The number of projected patterns reduces thanks to the spatial information that is taken into account Timemultiplexing 10 Pattern 1 11 Pattern 2 01 Pattern 3 01 Pattern 4 Spatial codification Direct codification Experiments Sub-pixel matching Conclusions Hall-Holt and Rusinkiewicz technique: 4 patterns with 111 binary stripes Edges encoding: 4 x 2 bits (every adjacent stripe is a bit) • The redundancy on the binary codification is eliminated Edge codeword: 10110101 15/36 16

Time-multiplexing: hybrid methods (II) Introduction Classification Timemultiplexing Spatial codification Direct codification Experiments Sub-pixel matching Conclusions 16/36 17

Spatial Codification Introduction Classification Timemultiplexing Spatial codification • Spatial codification paradigm encodes a set of points with the information contained in a neighborhood (called window) around them • The codification is condensed in a unique pattern instead of multiplexing it along time • The size of the neighborhood (window size) is proportional to the number of encoded points and inversely proportional to the number of used colors • The aim of these techniques is to obtain a one-shot measurement system moving objects can be measured Direct codification • Drawbacks: Experiments • Advantages: ― Moving objects supported Sub-pixel matching Conclusions ― Possibility to condense the codification to a unique pattern ― Discontinuities on the object surface can produce erroneous window decoding (occlusions problem) ― The higher the number of used colours, the more difficult to correctly identify them when measuring nonneutral surfaces ― Maximum resolution cannot be reached 17/36 18

Spatial codification: non-formal codification (I) Introduction • The first group of techniques that appeared used codification schemes with no mathematical basis • Drawbacks: Classification Timemultiplexing - the codification is not optimal and often produces ambiguities since different regions of the pattern are identical Spatial codification Direct codification Durdle et al. periodic pattern Experiments - the structure of the pattern is too complex for a good image processing Sub-pixel matching Conclusions Maruyama and Abe complex structure based on slits containing randomly placed cuts 18/36 19

Spatial codification: non-formal codification (II) Introduction Classification Timemultiplexing Spatial codification Direct codification Experiments Sub-pixel matching Conclusions 19/36 20

Spatial codification: De Bruijn sequences (I) Introduction • A De Bruijn sequence (or pseudorrandom sequence) of order m over an alphabet of n symbols is a circular string of length nm that contains every substring of length m exactly once (in this case the windows are unidimensional). 1000010111101001 Classification Timemultiplexing Spatial codification Direct codification m=4 (window size) n=2 (alphabet symbols) • The De Bruijn sequences are used to define coloured slit patterns (single axis codification) or grid patterns (double axis codification) • In order to decode a certain slit it is only necessary to identify one of the windows in which it belongs to Experiments Sub-pixel matching Conclusions Zhang et al. : 125 slits encoded with a De Bruijn sequence of 8 colors and window size of 3 slits Salvi et al. : grid of 29 29 where a De Bruijn sequence of 3 colors and window size of 3 slits is used to encode the vertical and horizontal slits 20/36 21

Spatial codification: De Bruijn sequences (II) Introduction Classification Timemultiplexing Spatial codification Direct codification Experiments Sub-pixel matching Conclusions 21/36 22

Spatial codification: M-arrays (I) Introduction Classification Timemultiplexing Spatial codification • An m-array is the bidimensional extension of a De Bruijn sequence. Every window of w h units appears only once. The window size is related with the size of the m-array and the number of symbols used 0 0 1 1 0 1 0 0 1 0 Example: binary marray of size 4 6 and window size of 2 2 Direct codification Experiments Shape primitives used to represent every symbol of the alphabet Sub-pixel matching Conclusions Morano et al. M-arry represented with an array of coloured dots M-array proposed by Vuylsteke et al. Represented with shape primitives 22/36 23

Spatial codification: M-arrays (II) Introduction Classification Timemultiplexing Spatial codification Direct codification Experiments Sub-pixel matching Conclusions 23/36 24

Direct Codification Introduction • Every encoded pixel is identified by its own intensity/colour • Since the codification is usually condensed in a unique pattern, the spectrum of intensities/colours used is very large • Additional reference patterns must be projected in order to differentiate among all the projected intensities/colours: Classification Timemultiplexing • Ambient lighting (black pattern) • Full illuminated (white pattern) • … Spatial codification Direct codification • Advantages: – Reduced number of patterns – High resolution can be teorically achieved • Drawbacks: – Very noisy in front of reflective properties of the objects, non-linearities in the camera spectral response and projector spectrum non-standard light emitters are required in order to project single wave-lengths – Low accuracy (order of 1 mm) Experiments Sub-pixel matching Conclusions 24/36 25

Direct codification: grey levels (I) Introduction • Every encoded point of the pattern is identified by its intensity level Classification Timemultiplexing Spatial codification Direct codification Every slit is identified by its own intensity Experiments Sub-pixel matching Conclusions Carrihill and Hummel Intensity Ratio Sensor: fade from balck to white • Every slit must be projected using a single wave-length Requirements to obtain high resolution • Cameras with large depth-per-pixel (about 11 bits) must be used in order to differentiate all the projected intensities 25/36 26

Direct codification: grey levels (II) Introduction Classification Timemultiplexing Spatial codification Direct codification Experiments Sub-pixel matching Conclusions 26/36 27

Direct codification: Colour (I) Introduction • Every encoded point of the pattern is identified by its colour Classification Timemultiplexing Spatial codification Direct codification Experiments Sub-pixel matching Conclusions Tajima and Iwakawa rainbow pattern (the rainbow is generated with a source of white light passing through a crystal prism) T. Sato patterns capable of cancelling the object colour by projecting three shifted patterns (it can be implemented with an LCD projector if few colours are projected drawback: the pattern becomes periodic in order to maintain a good resolution) 27/36 28

Direct codification: Colour (II) Introduction Classification Timemultiplexing Spatial codification Direct codification Experiments Sub-pixel matching Conclusions 28/36 29

Experiments: quantitative results Technique St. Dev( m) 3 D Points Resolution (%)* Patterns Binary code Posdamer (128 stripes) 37. 6 13013 11. 18 9 n-ary code Horn (64 stripes) 9. 6 12988 11. 15 5 Phase Shift + Gray code Gühring (113 slits) 4. 9 27214 23. 38 14 De Bruijn (64 slits) 13. 1 13899 11. 94 1 Salvi (29 29 grid) 72. 3 372 0. 32 1 M-array Morano (45 45 array) 23. 6 926 0. 80 1 Colour Sato (64 stripes) 11. 9 10204 8. 77 3 Introduction Classification Timemultiplexing Experiments Quantitative results Qualitative results Sub-pixel matching Conclusions Spatical codification Direct codification De Bruijn sequence Results obtained by reconstructing 30 times two flat panels separated by 40 mm. The distance between both panels obtained for each technique was calculated for every reconstruction. The Standard Deviation is indicated. (*) % of pixels inside a window of 515 x 226 of the camera image that have been triangulated 29/36 30

Experiments: qualitative results Horn (64 stripes) Gühring (113 slits) De Bruijn (64 slits) Salvi (29 x 29 slits) Morano (45 x 45 dot array) De Bruijn Introduction Gühring Posdamer (128 stripes) Timemultiplexing Classification Experiments Quantitative results Spatial codification Qualitative results Sato (64 slits) Sub-pixel matching Conclusions Direct codification 30/36 31

Sub-pixel matching: a key point • Introduction A key point in the accuracy of the 3 D measurements is to locate the correspondences between camera image and projector pattern with sub-pixel coordinates The reflected rays of light fall between ajdacent pixels in the camera Classification sensor Experiments This must be taken into account when triangulating the 3 D points • Camera image Projected pattern Qualitative example: Sub-pixel matching Stripe patterns Other patterns Conclusions Horse reconstruction with pixel-accuracy triangulation (20000 points) Horse reconstruction with subpixel-accuracy triangulation (10000 points) 31/36 32

Sub-pixel matching: stripe-patterns Introduction Classification • Only points belonging to the edges between adjacent stripes are decoded and reconstructed. Two possible strategies: - Intersecting the stripe intensity profile with and adaptative binarization threshold (calculated from two additional images: full illuminated and ambient lighting) - More accurate: Projecting positive and negative patterns and intersecting the stripe profiles Experiments + Sub-pixel matching Stripe patterns Other patterns Conclusions 32/36 33

Sub-pixel matching: other patterns Introduction Classification Experiments Sub-pixel matching Stripe patterns Other patterns Conclusions Arry of dots: the subpixel position of the dots is often calculated with their mass centre not very accurate since the circles are observed like deformed ellipses due to the change of perspective and the measuring surface Slit patterns: every observed slit can be modelled with a gaussian profile and peak detectors can be applied (like Blais & Rioux detector) very accurate 33/36 34

Conclusions Introduction Types of techniques Time-multiplexing • Highest resolution • High accuracy • Easy implementation • Inapplicability to moving objects • Large number of patterns Spatial codification • Can measure moving objects • A unique pattern is required • Lower resolution than timemultiplexing • More complex decoding stage • Occlusions problem • High resolution • Few patterns • Very sensitive to image noise cameras with large depth-per-pixel required • Sensitive to limited bandwith of LCD projectors special projector devices are usually required • Inapplicability to moving objects Classification Experiments Sub-pixel matching Direct codification Conclusions Guidelines 34/36 35

Guidelines Requirements Introduction Classification • High accuracy • Highest resolution • Static objects • No matter the number of patterns Best technique Phase shift + Gray code Gühring’s line-shift technique Experiments Sub-pixel matching Conclusions Guidelines • • High accuracy High resolution Static objects Minimum number of patterns • High accuracy • Good resolution • Moving objects N-ary pattern Horn & Kiryati Caspi et al. De Bruijn pattern Zhang et al. 35/36 36

END Computer Vision and Robotics Group Institut d’Informàtica i Aplicacions Jordi Pagès Joaquim Salvi