Image Processing for Videorate Scanning Probe Microscopy Martin

Image Processing for Video-rate Scanning Probe Microscopy Martin Moene Interface Physics Leiden University The Netherlands 50 x 49 nm 300 K Au(110) graphic by Prof. Dr. Richard Berndt, Kiel University Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Scanning Probe Microscopy • 1981 Scanning Tunneling Microscope (STM) [1] • 1986 Atomic Force Microscope (AFM) • Other variants… 20 x 13 nm 300 K Si(111) graphic by Prof. Dr. Richard Berndt, Kiel University [1] G. Binnig, H. Rohrer, C. Gerber, and E. Weibel, Phys. Rev. Lett. 49, 57 (1982). Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

40 s per Image 90 x 90 nm 1024 x 1024 Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

27 IMAGES per second Pan Rotate Zoom (64 x 64 pixels 2) a re 27 Hz Au(110) l m ti e HOPG 80 Hz [2] M. J. Rost, L. Crama, P. Schakel, E. van Tol et al. ; Rev. Sci. Instrum. 76 (2005) 053710 Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Leiden. Probe. Microscopy. com STM Head Feedback Drivers Scan Generator ADCs Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Stabilizing and Comparing Images thermal drift 50 x 49 nm 300 K Au(110) Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Apply Image Stabilisation to: • Stay Focused • Enable Quantitative Analysis (comparing images) AU 1 st Solution: Normalized Cross-correlation (NCC) A tool for both • Image Stabilisation and • Quantitative Analysis e tlin h g i he pixels Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

What is Cross-correlation (CC) ? Simplified nano wire or single-atom row x Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

What is Cross-correlation (CC) ? Simplified crystal surface x Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

What is Cross-correlation (CC) ? ” r o rr “e x Cross-correlation CC(c) = x f(x) t(x − c) c CC depends on offset and amplitude where f is the image and the sum is over x under the window containing the feature t positioned at c: x = c. . c+w Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Better Correlate Signal Form Mean subtracted Cross-correlation CC(c) = x f’(x) t’(x − c) where f is the image and the sum is over x under the window. Cross-correlation containing the feature t Normalized positioned at c: x = c. . c+w [-1, +1] Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Symmetric Computation CC(c) = N-1 x=0 f(c + x − N/2) t(x) The usual notation to compute symmetrically around the column at hand Values required that are outside the signal Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

CC(c) = N-1 x=0 f(c + x − N/2) t(x) Boundary Conditions Values required that are outside the image Constant 0 0 0 1 2 3 0 0 2 3 4 Extend 0 0 3 4 5 1 1 1 2 3 4 3 2 3 4 1 1 1 2 3 2 2 2 3 4 3 3 3 4 5 Periodic Reflect 5 4 3 4 5 1 1 1 2 3 5 4 3 4 5 3 4 2 3 4 4 5 3 4 5 2 3 1 2 3 3 4 2 3 4 4 5 3 4 5 Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

NCC Application 1: determine shift vector template image dy dx Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

NCC Application 2: compare images Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

NCC Application 3: locate feature image template 1 0. 7 • Qualitative: locate ‘a’ at global peak • Quantitative: ‘a’-s can be found at 1 • Quantitative: ‘o’-s can be found at 0. 7 Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Several Ways to Normalise Cross-correlation energy zero-mean image mean under template [3] J. Martin and J. L. Crowley. Experimental comparison of correlation techniques. In Proc. International Conf. on Intelligent Autonomous Systems, 1995. Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Fast NCC Implementation [4] Numerator computed via FFT as a convolution with the template reversed • FFT requires size 2 N, pad with zeros • FFT is periodic, prevent errors by padding larger area [5] [4] J. P. Lewis. Fast normalized cross-correlation. In Vision Interface, pages 120– 123, 1995. [5] H. Huang, D. Dabiri and M. Gharib. On errors of digital particle image velocimetry. Meas. Sci. Technol. 8 (1997) 1427 -1440. Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Fast NCC Implementation image energy under template Denominator computed from table containing the integral (running sum) of the image square over the search area. Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Fast NCC Implementation: Integral Image Def: The integral image at location (x, y), is the sum of the pixel values above and to the left of (x, y), inclusive. Using the integral image representation one can compute the value of any rectangular sum in constant time. For example the integral sum inside rectangle D we can compute as: ii(4) + ii(1) — ii(2) — ii(3) [6] P. Viola and M. Jones. Robust real-time object detection. Second International Workshop on Statistical and Computational Theories of Vision, 2001. Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Results: Timing *) While Analysing, Registrate and Correlate • Spatial Domain NCC: 40 minutes • Fast NCC: 300 ms While Measuring, Registrate (Preliminary) • Decimate image to 64 x 64 pixels 2 • Apply Gaussian sub-pixel interpolation [7] • Background subtraction plus fast NCC: 14 ms *) timing for images of 512 x 512 pixels 2 on a PC with an AMD Athlon at 2. 8 GHz [7] J. Bolinder. On the accuracy of a digital particle image velocimetry system. 1999. Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Results: Stabilisation Au(110) 300 K 39 x 38 nm 26 sec/frame Au(110) 300 K 52 x 55 nm 3. 8 sec/frame Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Summary: • NCC enables finding features • NCC enables quantitatively comparing features & images • NCC enables tracking to compensate for drift, there is room for improvement Future improvement: Lucas-Kanade [8] • • Spatial intensity gradient Taylor series expansion, iteration Gaussian Filter ( resolution) Pyramid of images at different resolution [8] B. Lucas and T. Kanade, An iterative image registration technique with an application to stereo vision, in Proc. Imaging Understanding Workshop, 1981, pp. 121— 130. Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Recognizing Features Coalescence of Vacancy Islands on Cu(100) Paul Ruijgrok 200 x 200 nm 300 K Cu(100) Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Finding the Vacancy Islands Paul Ruijgrok Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Leveling the Image Accuracy: • Data based number of bins • Fit (part of) Gaussian curve Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion Paul Ruijgrok

Finding the Vacancy Islands: threshold Paul Ruijgrok hthreshold = h 0 + sa 0 , s: 0. 1… 0. 9 Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Detecting the Island Edges Paul Ruijgrok Island A Erosion E(A, N 4) ∂A = A—E(A, N 4) erosion Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Finding the Vacancy Lines a = 1, b = 1 y = x + 1 1 1 2 2 1 1 1 1 2 4 1 1 2 2 1 1 1 1 1 1 1 1 1 Paul Ruijgrok Hough Transform yi = axi + b or b = -xia+ yi Transform points to curves in parameter space [9] Duda, R. O. and P. E. Hart, "Use of the Hough Transformation to Detect Lines and Curves in Pictures, " Comm. ACM, Vol. 15, pp. 11– 15 (January, 1972). Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Finding the Vacancy Lines Paul Ruijgrok Hough Transform • Slope-intercept representation: unbounded parameters • Want grid of limited size: • ρ = x cos(θ) + y sin(θ) , or • ρ = C cos(θ + δ) Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Summary Paul Ruijgrok 11 2 1 1 1 111212411111 11221 2 11 11 1 1 1 Thanks to DIPimage team, Delft University of Technology. DIPimage: a scientific image processing toolbox for MATLAB. Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Thanks To: Staff prof. dr. J. W. M. Frenken (Joost) dr. ir. T. H. Oosterkamp (Tjerk) dr. M. J. Rost (Marcel) Ph. D. Students Technicians drs. K. Schoots (Koen) L. Crama (Bert) E. van Tol-Homan (Els) Undergraduate Students R. Koehler (Raymond) P. V. Ruijgrok (Paul) P. Schakel (Peter) www. Leiden. Probe. Microscopy. com Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Summary template 1 0. 7 Hough Transform Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion image

The Future: Superresolution ? template 1 0. 7 Hough Transform Introduction | Stabilizing and Comparing Images | Recognizing Features | Future ? image

References [1] G. Binnig, H. Rohrer, C. Gerber, and E. Weibel, Phys. Rev. Lett. 49, 57 (1982). [2] M. J. Rost, L. Crama, P. Schakel, E. van Tol et al. ; Rev. Sci. Instrum. 76 (2005) 053710 [3] J. Martin and J. L. Crowley. Experimental comparison of correlation techniques. In Proc. International Conf. on Intelligent Autonomous Systems, 1995. [4] J. P. Lewis. Fast normalized cross-correlation. In Vision Interface, pages 120– 123, 1995. [5] H. Huang, D. Dabiri and M. Gharib. On errors of digital particle image velocimetry. Meas. Sci. Technol. 8 (1997) 1427 -1440. [6] P. Viola and M. Jones. Robust real-time object detection. Second International Workshop on Statistical and Computational Theories of Vision, 2001. [7] J. Bolinder. On the accuracy of a digital particle image velocimetry system. 1999. [8] B. Lucas and T. Kanade, An iterative image registration technique with an application to stereo vision, in Proc. Imaging Understanding Workshop, 1981, pp. 121 --130. [9] R. Duda and P. Hart. Use of the Hough transformation to detect lines and curves in pictures. Comm. ACM, Vol. 15, pp. 11– 15 (January, 1972). Introduction | Stabilizing and Comparing Images | Recognizing Features | References

Other Information Du-Ming Tsai , Chien-Ta Lin, Fast normalized cross correlation for defect detection, Pattern Recognition Letters, v. 24 n. 15, p. 2625 -2631, November 2003 Ian T. Young, Jan. J. Gerbrands and Lucas J. van Vliet. Fundamentals of Image Processing. 1998. W. H. Press, S. A. Teukolsky, W. T. Vetterling, B. P. Flannery. Numerical Recipes in C: The Art of Scientific Computing, 2 nd edition. Cambridge University Press. New York, NY, USA. Ullrich Köthe. STL-Style Generic Programming with Images. C++ Report Magazine 12(1), pp. 24 -30, January 2000. Leiden Probe Microscopy Interface Physics at Leiden University This presentation from author’s web-site Introduction | Stabilizing and Comparing Images | Recognizing Features | References

Software Stan Birchfield. Dept. of Electrical and Computer Engineering. Clemson University. KLT: An Implementation of the Kanade-Lucas-Tomasi Feature Tracker. Quantitative Imaging Group at the Faculty of Applied Sciences, Delft University of Technology. The Delft Image Processing library. 1999 -2004. Quantitative Imaging Group at the Faculty of Applied Sciences, Delft University of Technology. DIPimage, A Scientific Image Processing Toolbox for MATLAB. 1999 -2004. Insight Software Consortium. National Library of Medicine Insight Segmentation and Registration Toolkit (ITK). 1999 -2003. Cognitive Systems Group, University of Hamburg, Germany. The VIGRA Computer Vision Library. 1999 -2005. Chair of Technical Computer Science, RWTH Aachen University. LTI-Lib library for image processing and computer vision. 1999 -2003. Introduction | Stabilizing and Comparing Images | Recognizing Features | Software