Getting Started with Open CV Vadim Pisarevsky vadim

Getting Started with Open. CV Vadim Pisarevsky (vadim. pisarevsky@intel. com)

Agenda Open. CV facts and overall structure First steps Almighty High. GUI Using Open. CV within your program Now turn on IPP Dynamic structures Save your data Some useful Open. CV tips & tricks Where to get more information *Third party marks and brands are the property of their respective owners

What is Open. CV? Open. CV stands for Open Source Computer Vision Library Being developed at Intel since 1999 Written in C/C++; Contains over 500 functions. Available on Windows, Linux and Mac. OSX. So far is extensively used in many companies and research centers Home page: www. intel. com/technology/computing/opencv/ *Third party marks and brands are the property of their respective owners

Can you use Open. CV? Extract from license: [Copyright Clause] Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistribution's of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistribution's in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * The name of Intel Corporation may not be used to endorse or promote products derived from this software without specific prior written permission. [Disclaimer] In brief: Sure! *Third party marks and brands are the property of their respective owners

Now the question is: Should you use it?

Open. CV structure CV Image processing and vision algorithms High. GUI, Image and Video I/O CXCORE basic structures and algoritms, XML support, drawing functions *Third party marks and brands are the property of their respective owners We will mostly focus on these two in this presentation

First Open. CV Program 1. #include <cxcore. h> 2. #include <highgui. h> 3. #include <math. h> 4. int main( int argc, char** argv ) { 5. Cv. Point center; 6. double scale=-3; 7. Ipl. Image* image = argc==2 ? cv. Load. Image(argv[1]) : 0; 8. if(!image) return -1; 9. center = cv. Point(image->width/2, image->height/2); 10. for(int i=0; i<image->height; i++) 11. for(int j=0; j<image->width; j++) { 12. double dx=(double)(j-center. x)/center. x; 13. double dy=(double)(i-center. y)/center. y; 14. double weight=exp((dx*dx+dy*dy)*scale); 15. uchar* ptr = &CV_IMAGE_ELEM(image, uchar, i, j*3); 16. ptr[0] = cv. Round(ptr[0]*weight); 17. ptr[1] = cv. Round(ptr[1]*weight); 18. ptr[2] = cv. Round(ptr[2]*weight); } 19. cv. Save. Image( “copy. png”, image ); 20. cv. Named. Window( "test", 1 ); 21. cv. Show. Image( "test", image ); 22. cv. Wait. Key(); 23. return 0; } *Third party marks and brands are the property of their respective owners Radial gradient

How to build and run the program? 1. Obtain the latest version of Open. CV: a) b) 2. : pserver: anonymous@cvs. sourceforge. net: /cvsroot/opencvlibrary Install it: a) b) 3. On Windows run installer, on Linux use “. /configure + make install” (see INSTALL document) On Windows: opencv. dsw, do “build all”, add opencv/bin to the system path; on Linux - see a) Build the sample: 4. Get the official release (currently, Open. CV b 5 for Windows = Open. CV 0. 9. 7 for Linux) at http: //www. sourceforge. net/projects/opencvlibrary Get the latest snapshot from CVS: Windows: o cl /I<opencv_inc> test. cpp /link /libpath: <opencv_lib_path> cxcore. lib cv. lib highgui. lib o Create project for VS (see opencv/docs/faq. htm) or use opencv/samples/c/cvsample. dsp as starting point Linux: g++ -o test `pkg-config –cflags` test. cpp `pkg-config –libs opencv` Run it: test lena. jpg or. /test lena. jpg *Third party marks and brands are the property of their respective owners

The Program Quick Review 1. #include <cxcore. h> 2. #include <highgui. h> 3. #include <math. h> 4. int main( int argc, char** argv ) { 5. Cv. Point center; 6. double scale=-3; 7. Ipl. Image* image = argc==2 ? cv. Load. Image(argv[1]) : 0; cv. Load. Image 8. if(!image) return -1; 9. center = cv. Point(image->width/2, image->height/2); cv. Point 10. for(int i=0; i<image->height; i++) 11. for(int j=0; j<image->width; j++) { 12. double dx=(double)(j-center. x)/center. x; 13. double dy=(double)(i-center. y)/center. y; 14. double weight=exp((dx*dx+dy*dy)*scale); 15. uchar* ptr = &CV_IMAGE_ELEM(image, uchar, i, j*3); CV_IMAGE_ELEM 16. ptr[0] = cv. Round(ptr[0]*weight); cv. Round 17. ptr[1] = cv. Round(ptr[1]*weight); cv. Round 18. ptr[2] = cv. Round(ptr[2]*weight); } cv. Round 19. cv. Save. Image( cv. Save. Image “copy. png”, image ); 20. cv. Named. Window( cv. Named. Window "test", 1 ); 21. cv. Show. Image( cv. Show. Image "test", image ); 22. cv. Wait. Key(); cv. Wait. Key 23. return 0; } *Third party marks and brands are the property of their respective owners short and clear program, no need to mess with MFC/GTK/QT/…, cross-platform cv. Load. Image() and cv. Save. Image() provide the easiest way to save/load images of various formats. cv. Point and other “constructor” functions make the code shorter and allow 1 -line functions call. CV_IMAGE_ELEM() – pretty fast way to access image pixels cv. Round() is very fast and convenient way to cast float/double to int cv. Named. Window() creates “smart” window for viewing an image cv. Show. Image() shows image in the window cv. Wait. Key() delays the execution until key pressed or until the specified timeout is over

What else High. GUI can do? “Smart” windows Image I/O, rendering Processing keyboard and other events, timeouts Trackbars Mouse callbacks Video I/O *Third party marks and brands are the property of their respective owners

Windows cv. Named. Window(window_name, fixed_size_flag); creates window accessed by its name. Window handles repaint, resize events. Its position is remembered in registry: cv. Named. Window(“View. A”, 1); cv. Move. Window(“View. A”, 300, 100); cv. Destroy. Window(“View. A”); … cv. Show. Image(window_name, image); copies the image to window buffer, then repaints it when necessary. {8 u|16 s|32 f}{C 1|3|4} are supported. only the whole window contents can be modified. Dynamic updates of parts of the window are done using operations on images, drawing functions etc. On Windows native Win 32 UI API is used Linux – GTK+ 2 Mac. OSX – X 11 & GTK+ 2; Native Aqua support is planned. *Third party marks and brands are the property of their respective owners

Image I/O Ipl. Image* cv. Load. Image(image_path, colorness_flag); loads image from file, converts to color or grayscle, if need, and returns it (or returns NULL). image format is determined by the file contents. cv. Save. Image(image_path, image); saves image to file, image format is determined from extension. BMP, JPEG (via libjpeg), PNG (via libpng), TIFF (via libtiff), PPM/PGM formats are supported. Ipl. Image* img = cv. Load. Image(“picture. jpeg”, -1); if( img ) cv. Save. Image( “picture. png”, img ); *Third party marks and brands are the property of their respective owners

Waiting for keys… cv. Wait. Key(delay=0); waits for key press event for <delay> ms or infinitely, if delay=0. The only function in highgui that process message queue => should be called periodically. Thanks to the function many highgui programs have clear sequential structure, rather than event-oriented structure, which is typical for others toolkits To make program continue execution if no user actions is taken, use cv. Wait. Key(<delay_ms!=0>); and check the return value // opencv/samples/c/delaunay. c … for(…) { … int c = cv. Wait. Key(100); if( c >= 0 ) // key_pressed break; } *Third party marks and brands are the property of their respective owners delaunay. c, 240 lines

Trackbars cv. Create. Trackbar(trackbar_name, window_name, position_ptr, max_value, callback=0); creates trackbar and attaches it to the window. Value range 0. . max_value. When the position is changed, the global variable updated and callback is called, if specified. // opencv/samples/c/morphology. c int dilate_pos=0; // initial position value void Dilate(int pos) { … cv. Show. Image( “E&D”, erode_result ); } int main(…){ … cv. Create. Trackbar(“Dilate”, ”E&D”, &dilate_pos, 10, Dilate); … cv. Wait. Key(0); // check for events & process them. . . } *Third party marks and brands are the property of their respective owners morphology. c, 126 lines

Mouse Events cv. Set. Mouse. Callback(window_name, callback, userdata=0); sets callback on mouse events (button clicks, cursor moves) for the specified window // opencv/samples/c/lkdemo. c void on_mouse(int event, int x, int y, int flags, void* param) { … } int main(…){ … cv. Set. Mouse. Callback(“Lk. Demo”, on_mouse, 0); … cv. Wait. Key(0); // check for events & process them … } *Third party marks and brands are the property of their respective owners Scroll forward for example

Video I/O API Cv. Capture* cv. Capture. From. CAM(camera_id=0); initializes capturing from the specified camera Cv. Capture* cv. Capture. From. File(videofile_path); initializes capturing from the video file. Ipl. Image* cv. Query. Frame(capture); retrieves the next video frame (do not alter the result!), or NULL if there is no more frames or an error occured. cv. Get. Capture. Property(capture, property_id); cv. Set. Capture. Property(capture, property_id, value); retrieves/sets some capturing properties (camera resolution, position within video file etc. ) cv. Release. Capture(&capture); do not forget to release the resouces at the end! Used interfaces: – Windows: VFW, IEEE 1394, MIL, DShow (in progress), Quicktime (in progress) – Linux: V 4 L 2, IEEE 1394, FFMPEG – Mac. OSX: FFMPEG, Quicktime (in progress) *Third party marks and brands are the property of their respective owners

Video I/O Example // opencv/samples/c/lkdemo. c int main(…){ … Cv. Capture* capture = <…> ? cv. Capture. From. CAM(camera_id) : cv. Capture. From. File(path); if( !capture ) return -1; for(; ; ) { Ipl. Image* frame=cv. Query. Frame(capture); if(!frame) break; // … copy and process image cv. Show. Image( “Lk. Demo”, result ); c=cv. Wait. Key(30); // run at ~20 -30 fps speed if(c >= 0) { // process key }} cv. Release. Capture(&capture); } *Third party marks and brands are the property of their respective owners lkdemo. c, 190 lines (needs camera to run)

Using Open. CV in User Apps Case 1. Least squares (real example from Open. CV itself). // was (A – Nx. N matrix, b – Nx 1 right-side vector, x – solution): some_old_least_sq_func_32 f( /* float* */ A, /* int */ N, /* float* */ b, /* float* */ x); // has been converted to: { Cv. Mat _A=cv. Mat(N, N, CV_32 F, A), _b=cv. Mat(N, 1, CV_32 F, b), _x=cv. Mat(N, 1, CV_32 F, x); cv. Solve( &_A, &_b, &_x, CV_SVD ); } Advantages: – error handling – size and type checking – easier 32 f<->64 f conversion *Third party marks and brands are the property of their respective owners

Using Open. CV in User Apps (II) Case 2. Film Grain Direct. Show filter prototype. // inside Direct. Show filter Transform method … p. Media. Sample->Get. Pointer(&p. Data); AM_MEDIA_TYPE* p. Type = &m_p. Input->Current. Media. Type(); BITMAPINFOHEADER *bh = &((VIDEOINFOHEADER *)p. Type->pb. Format)->bmi. Header; Ipl. Image* img = cv. Create. Image. Header(cv. Size(bh. bi. Width, bh. bi. Height), 8, 3); cv. Set. Data( img, p. Data, (bh. bi. Wdith*3 + 3) & -4 ); cv. Cvt. Color( img, CV_BGR 2 YCr. Cb ); Ipl. Image* Y=cv. Create. Image(cv. Get. Size(img), 8, 1); cv. Split(img, Y, 0, 0, 0); Ipl. Image* Yf=cv. Create. Image(cv. Get. Size(img), 32, 1); Cv. RNG rng=cv. RNG(<seed>); cv. Rand. Arr(&rng, Yf, cv. Scalar. All(0), cv. Scalar. All(GRAIN_MAG), CV_RAND_NORMAL); cv. Smooth(Yf, CV_GAUSSIAN, GRAIN_SIZE, 0, GRAIN_SIZE*0. 5); cv. Acc(Y, Yf); cv. Convert(Yf, Y); cv. Merge(Y, 0, 0, 0, img); cv. Cvt. Color( img, CV_YCr. Cb 2 BGR ); cv. Release. Image(&Yf); cv. Release. Image(&Y); cv. Release. Image. Header(&img); … *Third party marks and brands are the property of their respective owners

Using Open. CV in User Apps (III) Case 3. Visualization inside some test program (hypothetical example, yet quite real too). // was … /* some image processing code containing bug */ … // has been converted to … … /* at this point we have the results */ #if VISUALIZE #include <highgui. h> #pragma comment(“lib”, ”cxcore. lib”) #pragma comment(“lib”, ”highgui. lib”) Cv. Mat hdr; cv. Init. Mat. Header(&hdr, 200/*height*/, 320/*width*/, CV_8 UC 3, data_ptr); Cv. Mat* vis_copy=cv. Clone. Mat(&hdr); cv. Named. Window(“test”, 1); cv. Circle(vis_copy, cv. Center(bug_x, bug_y), 30, CV_RGB(255, 0, 0), 3, CV_AA ); // mark the suspicious area cv. Show. Image(“test”, vis_copy); cv. Wait. Key(0 /* or use some delay, e. g. 1000 */); // cv. Save. Image( “buggy. png”, vis_copy ); // if need, save for post-mortem analysis // cv. Destroy. Window(“test”); // if need #endif *Third party marks and brands are the property of their respective owners

What can be drawn in image (besides circles)? cxcore provides numerous drawing functions: – – Lines, Circles, Ellipses, Elliptic Arcs Filled polygons or polygonal contours Text (using one of embedded fonts) Everything can be drawn with different colors, different line width, antialiasing on/off – Arbitrary images types are supported (for depth!=8 u antializing is off) *Third party marks and brands are the property of their respective owners

Using Open. CV with IPP Triple 3 x 3 median filter #include <cv. h> #include <highgui. h> #include <ipp. h> #include <stdio. h> int main(int, char**){ const int M=3; Ippi. Size msz={M, M}; Ippi. Point ma={M/2, M/2}; Ipl. Image* img=cv. Load. Image(“lena. jpg”, 1); Ipl. Image* med 1=cv. Create. Image(cv. Get. Size(img), 8, 3); Ipl. Image* med 2=cv. Clone. Image(med 1); int 64 t 0 = cv. Get. Tick. Count(), t 1, t 2; Ippi. Size sz = {img->width-M+1, img->height-M+1}; double isz=1. /(img->width*img->height); cv. Smooth(img, med 1, CV_MEDIAN, M); // use IPP via Open. CV t 0=cv. Get. Tick. Count()-t 0; cv. Use. Optimized(0); // unload IPP t 1 = cv. Get. Tick. Count(); cv. Smooth(img, med 1, CV_MEDIAN, M); // use C code t 1=cv. Get. Tick. Count()-t 1; t 2=cv. Get. Tick. Count(); ippi. Median. Filter_8 u_C 3 R( // use IPP directly &CV_IMAGE_ELEM(img, uchar, M/2*3), img->width. Step, &CV_IMAGE_ELEM(med 1, uchar, M/2*3), med 1 ->width. Step, sz, ma ); t 2=cv. Get. Tick. Count()-t 2; printf(“t 0=%. 2 f, t 1=%. 2 f, t 2=%. 2 fn”, (double)t 0*isz, (double)t 1*isz, (double)t 2*isz); return 0; } *Third party marks and brands are the property of their respective owners

How does it works? (Quite Similar to IPP dispatcher mechanism) // cv/src/_cvipp. h … IPCVAPI_EX(Cv. Status, icv. Filter. Median_8 u_C 3 R, “ippi. Filter. Median_8 u_C 3 R”, CV_PLUGINS 1(CV_PLUGIN_IPPI), (const void* src, int srcstep, void* dst, int dststep, Cv. Size roi, Cv. Size ksize, Cv. Point anchor )) … // cv/src/cvswitcher. cpp … #undef IPCVAPI_EX #define IPCVAPI_EX() … static Cv. Plugin. Func. Info cv_ipp_tab[] = { #undef _CV_IPP_H_ /* with redefined IPCVAPI_EX every function declaration turns to the table entry */ #include "_cvipp. h“ #undef _CV_IPP_H_ {0, 0, 0} }; static Cv. Module. Info cv_module = { 0, "cv", "beta 5 (0. 9. 7)", cv_ipp_tab }; static int loaded_functions = cv. Register. Module( &cv_module ); … // cv/src/cvsmooth. cpp icv. Filter. Median_8 u_C 3 R_t icv. Filter. Median_8 u_C 3 R_p = 0; void cv. Smooth() { if( icv. Filter. Median_8 u_C 3 R_p ) { /* use IPP */ } else { /* use C code */… } *Third party marks and brands are the property of their respective owners

Dynamic Structures (when 2 d array is not enough) The sample task: collect the locations of all non-zero pixels in the image. Where to store the locations? Possible solutions: – Allocate array of maximum size (sizeof(Cv. Point)*width*height – a huge value) – Use two-pass algorithm – Use list or similar data structure – … (Any other ideas? ) // construct sequence of non-zero pixel locations Cv. Seq* get_non_zeros( const Ipl. Image* img, Cv. Mem. Storage* storage ) { Cv. Seq* seq = cv. Create. Seq( CV_32 SC 2, sizeof(Cv. Seq), sizeof(Cv. Point), storage ); for( int i = 0; i < img->height; i++ ) for( int j = 0; j < img->width; j++ ) if( CV_IMAGE_ELEM(img, uchar, i, j) ) { Cv. Point pt={j, i}; cv. Seq. Push( seq, &pt ); } return seq; } *Third party marks and brands are the property of their respective owners

Memory Storages … Memory storage is a linked list of memory blocks. Data is allocated in the storage by moving “free space” position, new blocks are allocated and added to the list if necessary It’s only possible to free the continous block of data by moving position back, thus both allocation and deallocation are virtually instant(!) Functions to remember: cv. Create. Mem. Storage(block_size), cv. Release. Mem. Storage(storage); cv. Clear. Mem. Storage(storage); cv. Mem. Storage. Alloc(storage, size); All Open. CV “dynamic structures” reside in memory storages. The model is very efficient for repetitive processing, where cv. Clear. Mem. Storage is called on top level between iterations. *Third party marks and brands are the property of their respective owners

Sequences Sequence is a linked list of memory blocks (sounds familiar, eh? ). Large sequences may be split into multiple storage blocks, while several small sequences may fit into the single storage block. Sequence is deque: adding/removing elements to/from either end is O(1) operation, inserting/removing elements from the middle is O(N). Accessing arbitrary element is also ~O(1) (when storage block size >> sequence block) Sequences can be sequentially read/written using readers/writers (similar to STL’s iterators) Sequences can not be released, use cv{Clear|Release}Mem. Storage() Basic Functions to remember: cv. Create. Seq(flags, header_size, element_size, storage), cv. Seq. Push[Front](seq, element), cv. Seq. Pop[Front](seq, element), cv. Clear. Seq(seq), cv. Get. Seq. Elem(seq, index), cv. Start. Read. Seq(seq, reader), cv. Start. Append. To. Seq(seq, writer). *Third party marks and brands are the property of their respective owners

Sets and Sparse Matrices Set is variant of sequence with “free node list” and it’s active elements do not follow each other – that is, it is convenient way to organize a “heap” of same-size memory blocks. Sparse matrix in Open. CV uses Cv. Set for storing elements. // Collecting the image colors Cv. Sparse. Mat* get_color_map( const Ipl. Image* img ) { int dims[] = { 256, 256 }; Cv. Sparse. Mat* cmap = cv. Create. Sparse. Mat(3, dims, CV_32 SC 1); for( int i = 0; i < img->height; i++ ) for( int j = 0; j < img->width; j++ ) { uchar* ptr=&CV_IMAGE_ELEM(img, uchar, i, j*3); int idx[] = {ptr[0], ptr[1], ptr[2]}; ((int*)cv. Ptr. ND(cmap, idx))[0]++; } // print the map Cv. Sparse. Mat. Iterator it; for(Cv. Sparse. Node *node = cv. Init. Sparse. Mat. Iterator( mat, &iterator ); node != 0; node = cv. Get. Next. Sparse. Node( &iterator )) { int* idx = CV_NODE_IDX(cmap, node); int count=*(int*)CV_NODE_VAL(cmap, idx); printf( “(b=%d, g=%d, r=%d): %dn”, idx[0], idx[1], idx[2], count ); } return cmap; } *Third party marks and brands are the property of their respective owners

Save your data (to load it then) Need a config file? Want to save results of your program to pass it to another one, or look at it another day? It all can be easily done with Open. CV persistence-related functions. my_matrix. xml // somewhere deep in your code… you get 5 x 5 // matrix that you’d want to save { Cv. Mat A = cv. Mat( 5, 5, CV_32 F, the_matrix_data ); cv. Save( “my_matrix. xml”, &A ); } // to load it then in some other program use … Cv. Mat* A 1 = (Cv. Mat*)cv. Load( “my_matrix. xml” ); <? xml version="1. 0"? > <opencv_storage> <my_matrix type_id="opencv-matrix"> <rows>5</rows> <cols>5</cols> <dt>f</dt> <data> 1. 0. 0. 0. 1. </data></my_matrix> </opencv_storage> *Third party marks and brands are the property of their respective owners

So what about config file? Writing config file Cv. File. Storage* fs = cv. Open. File. Storage(“cfg. xml”, 0, CV_STORAGE_WRITE); cv. Write. Int( fs, “frame_count”, 10 ); cv. Write. Start. Write. Struct( fs, “frame_size”, CV_NODE_SEQ); cv. Write. Int( fs, 0, 320 ); cv. Writeint( fs, 0, 200 ); cv. End. Write. Struct(fs); cv. Write( fs, “color_cvt_matrix”, cmatrix ); cv. Release. File. Storage( &fs ); Reading config file cfg. xml <? xml version="1. 0"? > <opencv_storage> <frame_count>10</frame_count> <frame_size>320 200</frame_size> <color_cvt_matrix type_id="opencvmatrix"> <rows>3</rows> <cols>3</cols> <dt>f</dt> <data>…</data></color_cvt_matrix> </opencv_storage> Cv. File. Storage* fs = cv. Open. File. Storage(“cfg. xml”, 0, CV_STORAGE_READ); int frame_count = cv. Read. Int. By. Name( fs, 0, “frame_count”, 10 /* default value */ ); Cv. Seq* s = cv. Get. File. Node. By. Name(fs, 0, ”frame_size”)->data. seq; int frame_width = cv. Read. Int( (Cv. File. Node*)cv. Get. Seq. Elem(s, 0) ); int frame_height = cv. Read. Int( (Cv. File. Node*)cv. Get. Seq. Elem(s, 1) ); Cv. Mat* color_cvt_matrix = (Cv. Mat*)cv. Read(fs, 0, ”color_cvt_matrix”); cv. Release. File. Storage( &fs ); *Third party marks and brands are the property of their respective owners

Some Open. CV Tips and Tricks Use short inline “constructor” functions: cv. Scalar, cv. Point, cv. Size, cv. Mat etc. Use cv. Round and vector math functions (cv. Exp, cv. Pow …) for faster processing Use cv. Stack. Alloc for small buffers Consider CV_IMPLEMENT_QSORT_EX() as possible alternative to qsort() & STL sort(). To debug Open. CV program when runtime error dialog error pops up, press “Retry” (works with debug version of libraries) … *Third party marks and brands are the property of their respective owners

Where to get more information? Open. CV Wiki-pages: http: //opencvlibrary. sourceforge. net Supplied documentation: Open. CV/docs/index. htm, faq. htm The forum: opencv@yahoogroups. com *Third party marks and brands are the property of their respective owners

Thank you! Questions?