Towards Performance Evaluation of Symbol Recognition Spotting Systems

Towards Performance Evaluation of Symbol Recognition & Spotting Systems in a Localization Context Mathieu Delalandre CVC, Barcelona, Spain Euro. Med Meeting LORIA, Nancy city, France Monday 18 th of May 2009

Introduction Electrical diagram scanne d Mechanical drawing CAD file Utility map Web image Symbol recognition: ““a particular application of the general problem of pattern recognition, in which an unknown input pattern (i. e. input image) is classified as belonging to one of the relevant classes (i. e. predefined symbols) in the application domain” [Chhabra 1998][Cordella 1999] [Llados 2002] labels learnin tub [Tombre 2005] g doo ski datab docum doo r sofn ent ase. Recognition r a databa se Spotting r 1 r 2 r 3 rank symbol backgrou text nd Query By Exampl e (QBE) “a way to efficiently localize Symbol spotting: possible symbols and limit the computational complexity, without using full recognition methods” [Tombre 2003] [Dosch 2004] [Tabbone 2004] [Zuwala 2006] [Locteau 2007] [Qureshi 2007] [Rusinol 2007]

Introduction Performance evaluation: Information Retrieval [Salton 1992], Computer Vision [Thacker 2005], CBIR [Muller 2001], DIA [Haralick 2000] d. ATA Traini System Data Case of symbol recognition & spotting: [Ezra 2008][Delalandre 2008] Labels tub door ng data Groundtruthin g Results Groundtruth Characterisati on Performance evaluation QBE Learning door Spotting/Recognition System Mapping Ranks sofa r 1 r 3 Region Of Interest Groundtruth skin truth results r 2 Characterization Performance evaluatio

Plan 1. Groundtruth and test documents 2. Performance characterization 3. Conclusions and perspectives

1. Overview of approaches 2. Existing datasets Groundtruth and test documents Overview of approaches Noise Conne Groundtruth Document Groundtruthing Document real appro ach - - ++ - manyes no y - ++ -- manyes no y synthetic approach Rusinol’ 09 cted Symb ol Reliabi Yan’ 04 lity Realis m Speed -weak ++ good [Dosch’ 06 Real approach Aksoy’ 00 ++2006 - - ++ man no ye Dosch and al groundtruthed s groundtruth y Zhai’ 03 ++ - - ++ one. GT no ye drawings groun s validation d. Valveny’ 07 ++ - - ++ one no ye drawing truthin svalidatio s and g Delalandre’ + ++ manyes no n alerts + recogniti and 08 y on evaluation alerts test results imag Rusinol and al 2009 Yan and al 2004 4 5 1 3 0 4 1 2 5 0 2 3 connected parallel and Groundtruth

1. Overview of approaches 2. Existing datasets Groundtruth and test documents Overview of approaches Synthetic approach Groundtruth Setting real appro ach - - ++ - manyes no y - ++ -- manyes no y ++ - - ++ man no ye Aksoyy 2000 s Zhai’ 03 ++ - - ++ one no ye s Valveny’ 07 ++ - - ++ one no ye s Delalandre’ + + ++ manyes no 08 y Groundtruthin g Groundtruth Document synthetic approach Aksoy’ 00 Noise Conne Rusinol’ 09 cted Symb ol Reliabi Yan’ 04 lity Realis m Speed -weak ++ good [Dosch’ 06 Valveny and al 2007 Zhai and al 2003 binary noise vectorial

Groundtruth and test documents 1. Overview of approaches 2. Existing datasets Overview of approaches Delalandre 2008 Graphical documents are composed of two layers symbol backgroun d real appro ach - - ++ - manyes no y - ++ -- manyes no y ++ - - ++ man no ye y s Zhai’ 03 ++ - - ++ one no ye s Valveny’ 07 ++ - - ++ one no ye s Delalandre’ + + ++ manyes no 08 y synthetic approach Aksoy’ 00 Noise Conne Rusinol’ 09 cted Symb ol Reliabi Yan’ 04 lity Realis m Speed -weak ++ good [Dosch’ 06 To use a same background layer with different symbol layers

1. Overview of approaches 2. Existing datasets Groundtruth and test documents Overview of approaches Delalandre 2008 ++ - - ++ man no ye y s Zhai’ 03 ++ - - ++ one no ye s Valveny’ 07 ++ - - ++ one no ye s Delalandre’ + + ++ manyes no 08 y synthetic approach - - ++ - manyes no y - ++ -- manyes no y C 1 C 2 M 1 M 2 M 3 M 4 real appro ach Aksoy’ 00 Noise Conne Rusinol’ 09 cted Symb ol Reliabi Yan’ 04 lity Realis m Speed -weak ++ good [Dosch’ 06 C 3 C 4 c 1 c 2 L symbol model loaded symbol bounding box and control point alignm ent p p 1 L 1 θ 1 L 2 p 2 θ 2

1. Overview of approaches 2. Existing datasets Groundtruth and test documents Overview of approaches Delalandre 2008 Noise Conne Symbol Models real appro ach - - ++ - manyes no y - ++ -- manyes no y ++ - - ++ man no ye y s Zhai’ 03 ++ - - ++ one no ye s Valveny’ 07 ++ - - ++ one no ye s Delalandre’ + + ++ manyes no 08 y synthetic approach Rusinol’ 09 cted Symb ol Reliabi Yan’ 04 lity Realis m Speed -weak ++ good [Dosch’ 06 Positioning Constraints Document Generation Symbol Positioning (2) run GT GT GT Symbol Models Aksoy’ 00 (1) edit (3) display Building Engine Backgroun d Image GT

Groundtruth and test documents Existing datasets datase image symbol degradat mode ts s s ions ls #30 3000 10 5 -50 25150 50150 GREC’ 05 #16 1000 6 GREC’ 07 #6 2100 6 ICPR’ 00 #9 450 11250 9 bags #16 1600 15046 none floorplans diagrams queries #10 #6 1000 6000 26830 14100 6000 none 25150 16 21 16 -21 Rusinol’ 0 9 #1 42 344 none 38 GRE ICPR C GREC’ 03 SESYD 25 1. Overview of approaches 2. Existing datasets Others

Groundtruth and test documents Existing datasets datase image symbol degradat mode ts s s ions ls #30 3000 10 5 -50 25150 50150 GREC’ 05 #16 1000 6 GREC’ 07 #6 2100 6 ICPR’ 00 #9 450 11250 9 bags #16 1600 15046 none floorplans diagrams queries #10 #6 1000 6000 26830 14100 6000 none 25150 16 21 16 -21 Rusinol’ 0 9 #1 42 344 none 38 GRE ICPR C GREC’ 03 SESYD 25 1. Overview of approaches 2. Existing datasets Others

Groundtruth and test documents Existing datasets datase image symbol degradat mode ts s s ions ls #30 3000 10 5 -50 25150 50150 GREC’ 05 #16 1000 6 GREC’ 07 #6 2100 6 ICPR’ 00 #9 450 11250 9 bags #16 1600 15046 none floorplans diagrams queries #10 #6 1000 6000 26830 14100 6000 none 25150 16 21 16 -21 Rusinol’ 0 9 #1 42 344 none 38 GRE ICPR C GREC’ 03 SESYD 25 1. Overview of approaches 2. Existing datasets Others

Groundtruth and test documents Existing datasets datase image symbol degradat mode ts s s ions ls #30 3000 10 5 -50 25150 50150 GREC’ 05 #16 1000 6 GREC’ 07 #6 2100 6 ICPR’ 00 #9 450 11250 9 bags #16 1600 15046 none floorplans diagrams queries #10 #6 1000 6000 26830 14100 6000 none 25150 16 21 16 -21 Rusinol’ 0 9 #1 42 344 none 38 GRE ICPR C GREC’ 03 SESYD 25 1. Overview of approaches 2. Existing datasets Others

Groundtruth and test documents Existing datasets datase image symbol degradat mode ts s s ions ls #30 3000 10 5 -50 25150 50150 GREC’ 05 #16 1000 6 GREC’ 07 #6 2100 6 ICPR’ 00 #9 450 11250 9 bags #16 1600 15046 none floorplans diagrams queries #10 #6 1000 6000 26830 14100 6000 none 25150 16 21 16 -21 Rusinol’ 0 9 #1 42 344 none 38 GRE ICPR C GREC’ 03 SESYD 25 1. Overview of approaches 2. Existing datasets Others

Groundtruth and test documents Existing datasets datase image symbol degradat mode ts s s ions ls #30 3000 10 5 -50 25150 50150 GREC’ 05 #16 1000 6 GREC’ 07 #6 2100 6 ICPR’ 00 #9 450 11250 9 bags #16 1600 15046 none floorplans diagrams queries #10 #6 1000 6000 26830 14100 6000 none 25150 16 21 16 -21 Rusinol’ 0 9 #1 42 344 none 38 GRE ICPR C GREC’ 03 SESYD 25 1. Overview of approaches 2. Existing datasets Others

Groundtruth and test documents Existing datasets datase image symbol degradat mode ts s s ions ls #30 3000 10 5 -50 25150 50150 GREC’ 05 #16 1000 6 GREC’ 07 #6 2100 6 ICPR’ 00 #9 450 11250 9 bags #16 1600 15046 none floorplans diagrams queries #10 #6 1000 6000 26830 14100 6000 none 25150 16 21 16 -21 Rusinol’ 0 9 #1 42 344 none 38 GRE ICPR C GREC’ 03 SESYD 25 1. Overview of approaches 2. Existing datasets Others

Groundtruth and test documents 1. Overview of approaches 2. Existing datasets datase image symbol degradat mode ts s s ions ls #30 3000 10 5 -50 25150 50150 GREC’ 05 #16 1000 6 GREC’ 07 #6 2100 6 ICPR’ 00 #9 450 11250 9 bags #16 1600 15046 none floorplans diagrams queries #10 #6 1000 6000 26830 14100 6000 none 25150 16 21 16 -21 Rusinol’ 0 9 #1 42 344 none 38 GRE ICPR C GREC’ 03 of queries 3. Random crop SESYD 25 1. Random selection of a document Groundtruth 2. Radom selection of a symbol Generator y Others s [0, 1] 0 v x vmax

Groundtruth and test documents Existing datasets datase image symbol degradat mode ts s s ions ls #30 3000 10 5 -50 25150 50150 GREC’ 05 #16 1000 6 GREC’ 07 #6 2100 6 ICPR’ 00 #9 450 11250 9 bags #16 1600 15046 none floorplans diagrams queries #10 #6 1000 6000 26830 14100 6000 none 25150 16 21 16 -21 Rusinol’ 0 9 #1 42 344 none 38 GRE ICPR C GREC’ 03 SESYD 25 1. Overview of approaches 2. Existing datasets Others

Plan 1. Groundtruth and test documents 2. Performance characterization 3. Conclusions and perspectives

Performance characterization Introduction Performance characterisation (segmented symbols) [Valveny 2004] [Dosch 2006] [Valveny 2007, 2008 a, 2008 b] Performance characterisation (real context) Labels tub door üRecognition rate üPrecision/Recall üHomogeneity üSeparability QBE Learning door Spotting/Recognition System Mapping Ranks sofa r 1 r 3 Region Of Interest Groundtruth skin truth results r 2 Characterization Performance evaluation

Performance characterization About mapping Mapping cases False alarm : a Single : a model line detected line matches only with one doesn't match with any detected line. model lines. truth results Split : two model lines match with one detected. Miss : a model line doesn't line. match with any Merge : a model line detected lines. matches with two Symbol spotting Text/graphics detected lines. [Rusinol 2009] [Wenyin 1997] g 1 g 2 Groundtruth r c 1 Results c 2 Layout analysis [Antonacopoulos 1999] segmentation segmen tation groundt ruth separation groundtr uth Mapping segmentat ion

Performance characterization Mapping, application to symbol Which representation ? point How to define the regions ? How to define local thresholds the polarized pat of the capacitor belong to the symbol ? wrapper box, ellipsis the precision will depend of the model Same for the moving area of the door ? convex polygon concave could be of weak precision precise but comparison is time consuming Compatibility with recognition systems ? Lot of systems use sliding windows to detect symbols providing only points [Adam 2001] [Dosh 2004] [Rusinol 2007] Systems providing region of interest can “tune” their results, how to limit the over segmentation cases ? groundtruth segmentation

Performance characterization Work in progress Comparison of some criteria System of [Qureshi’ 08] , 100 floorplans (2521 symbols) Signature based characterization Domain definition of the ROI Orientation sampling [02π] Rate s% 12, 00 10, 00 8, 00 resu lts Reporting [02π] groundt ruth 6, 00 4, 00 0, 00 1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96 101 106 2, 00 Region size dx×dy

Plan 1. Groundtruth and test documents 2. Performance characterization 3. Conclusions and perspectives

Conclusions and perspectives • Conclusions – – Large databases of segmented symbol images exist “GREC” Synthetic databases in real context exist “SESYD” True-life documents and groundtruth are at the corner “EPEIRES” Characterization tools have been proposed “Symbol. Rec” • Perspectives – Continue to produce other databases, using existing platforms – Mapping is the key problem today, to achieve a performance evaluation in real context

Thanks All the referenced papers can be found in [1] M. Delalandre, E. Valveny and J. Lladós Performance Evaluation of Symbol Recognition and Spotting Systems: A Overview. Workshop on Document Analysis Systems (DAS), pp 497 -505, 2008.