Workpackage 4 Image Analysis Algorithms Kirk Martinez Paul

Workpackage 4 Image Analysis Algorithms Kirk Martinez, Paul Lewis and Stephen Chan Intelligence, Agents and Multimedia Department of Electronics and Computer Science University of Southampton UK Project IST_1999_11978 - ARTISTE – An Integrated Art Analysis and Navigation Environment Review Meeting N. 1: Paris, C 2 RMF, November 28, 2000

Task 4. 1 User requirements Analysis • • • First step was to identify the requirements of the users (note overlap with other workpackages) Required Output: Collation of scenarios and functionality Output achieved in collaboration with other participants and delivered in the form of some initial scenarios and a set of 16 goals. These are published as part of the System design document. Project IST_1999_11978 - ARTISTE – An Integrated Art Analysis and Navigation Environment Review Meeting N. 1: Paris, C 2 RMF, November 28, 2000

Task 4. 3 Recognition Algorithm Development • • PM 8 -24 Aim is to develop image analysis algorithms to meet user requirements Required outputs: Image content analysis software and report Consider 4. 1 and 4. 3 together in this presentation Project IST_1999_11978 - ARTISTE – An Integrated Art Analysis and Navigation Environment Review Meeting N. 1: Paris, C 2 RMF, November 28, 2000

Goals • • • G 1 Matching of similar images (includes “have you got this picture”) G 2 Automatic search using synonyms G 3 Search based on features oriented to the restoration framework - uv spotmeasures - x-ray and reverse pic views of frames - craquelure classifier -search based on “butterfly” supports in the frame Project IST_1999_11978 - ARTISTE – An Integrated Art Analysis and Navigation Environment Review Meeting N. 1: Paris, C 2 RMF, November 28, 2000

Goals Cont. • • G 5 Access information quickly and easily G 6 Search based on colour G 7 Query by low quality images (especially faxes) G 8 Query by sketch G 9 Query refinement G 10 Joint retrieval by content and by text G 11 Use of publishing products built on the Artiste system G 12 Detail finding Project IST_1999_11978 - ARTISTE – An Integrated Art Analysis and Navigation Environment Review Meeting N. 1: Paris, C 2 RMF, November 28, 2000

Goals Cont. • • G 13 Search using multilingual vocabulary G 14 Respect installation site privacy and security policy G 15 Produce a sustainable system after the end of the project G 16 Be consistent with partners’ predefined technical constraints Project IST_1999_11978 - ARTISTE – An Integrated Art Analysis and Navigation Environment Review Meeting N. 1: Paris, C 2 RMF, November 28, 2000

Analysis • • • The goals were analysed in terms of the implications for image analysis Possible image processing (IP) approaches were identified for goals requiring IP Six distinct groups of algorithms were identified together with the goals to which they could contribute. Project IST_1999_11978 - ARTISTE – An Integrated Art Analysis and Navigation Environment Review Meeting N. 1: Paris, C 2 RMF, November 28, 2000

1. Algorithms which will find similarity matches based on global histograms (colour or grey scale) between a query image or sub-image and images in the database collections. • • • Could contribute to goals 1, 3, 7 Useful for basic image matching May contribute to style search and classification Potentially faster than spatial-colour matching methods Status: Implemented at IAM Project IST_1999_11978 - ARTISTE – An Integrated Art Analysis and Navigation Environment Review Meeting N. 1: Paris, C 2 RMF, November 28, 2000

Example of Global Colour Histogram Search Query image Best Matches Project IST_1999_11978 - ARTISTE – An Integrated Art Analysis and Navigation Environment Review Meeting N. 1: Paris, C 2 RMF, November 28, 2000

2. Algorithms which will find similarity matches between a query image or sub-image and images or sub-images in the database using spatial colour distributions. • • • Goals: 1, 3, 6, 10, 12 Takes into account the spatial arrangement of colours Finds similar colour patterns at similar locations Or similar colour patterns at any location Status: Implemented a hierarchical colour coherence vector based matcher in IAM Project IST_1999_11978 - ARTISTE – An Integrated Art Analysis and Navigation Environment Review Meeting N. 1: Paris, C 2 RMF, November 28, 2000

Example of H-CCV Matching Query sub-image Best Matches Project IST_1999_11978 - ARTISTE – An Integrated Art Analysis and Navigation Environment Review Meeting N. 1: Paris, C 2 RMF, November 28, 2000

3. Algorithms which will segment image into regions of similar texture and record feature vectors representing texture for each main regions. User can then indicate a query texture either by indicating a region in a particular image or selecting a texture from a texture palette. Images in the database containing texture regions matching the query are then retrieved. • • • Goals: 1, 3, 12 Status: Previously implemented texture extraction algorithms Not yet implemented automatic texture segmentation Project IST_1999_11978 - ARTISTE – An Integrated Art Analysis and Navigation Environment Review Meeting N. 1: Paris, C 2 RMF, November 28, 2000

4. Algorithms which will match an outline query shape with similar shapes within database images. • • • Goals: 1, 3, 4, 7. 8, 10, 12 Pre extracting all shapes of all objects in all images is impossible i. e. can not pre-index shapes! Techniques like the Generalised Hough Transform (GHT) use evidence accumulation to find a shape in an image and are related to template matching. They are computationally intensive Status: Not yet implemented for this project Project IST_1999_11978 - ARTISTE – An Integrated Art Analysis and Navigation Environment Review Meeting N. 1: Paris, C 2 RMF, November 28, 2000

5. Algorithms which will detect and in some cases analyse specific image features • • Goals: 4 May be able to use e. g. the GHT Will also require specially tailored algorithms Status: Not yet implemented Project IST_1999_11978 - ARTISTE – An Integrated Art Analysis and Navigation Environment Review Meeting N. 1: Paris, C 2 RMF, November 28, 2000

6. Algorithms to provide basic image manipulation • • Goals: All involving image handling Include operations like image conversion, compression, scaling, rotation etc Most are widely available but may need re-implementing or tailoring in context of Artiste Status: Partial implementation Project IST_1999_11978 - ARTISTE – An Integrated Art Analysis and Navigation Environment Review Meeting N. 1: Paris, C 2 RMF, November 28, 2000

Environment for Algorithm Testing • • • Test-beds in the IAM lab include VIPS and MAVIS 2 Algorithms developed as “stand alone” modules which deliver feature vectors (FVs) and modules for calculating similarity between FVs MAVIS 2 is a multimedia retrieval and navigation environment It associates media content and the concepts they represent Concept layer equivalent to a thesaurus Allows integrated content and concept based searching with query scope expansion Project IST_1999_11978 - ARTISTE – An Integrated Art Analysis and Navigation Environment Review Meeting N. 1: Paris, C 2 RMF, November 28, 2000