Scene recognition by inexact graph matching Celso C
Scene recognition by inexact graph matching Celso C. Ribeiro * Claudia Boeres Isabelle Bloch IV ALIO/EURO Workshop on Applied Combinatorial Optimization Pucón, Chile, November 4 -6, 2002 November 2002 1 Scene recognition by inexact graph matching
Summary • • Motivation and applications Optimal graph matching formulation Model: objective function and constraints Randomized construction algorithm Local search GRASP heuristic Numerical results Current and future work November 2002 2 Scene recognition by inexact graph matching
Motivation and applications • Recognition and understanding of complex scenes require a detailed description of … – … the objects involved – … the spatial relationships between the objects • The diversity of the forms of the same object in different instantiations of a scene, and also the similarities of different objects in the same scene, make the relationships between objects of prime importance to help the recognition of objects with similar appearance. November 2002 3 Scene recognition by inexact graph matching
Motivation and applications • Graph-based representations often used for scene representation: – Nodes represent the objects in the scene. – Edges represent the relationships between the objects. • Relevant information is extracted from the scene and represented by relational attributed graphs. • Model-based recognition: both the model and the scene are represented graphs. by Graph matching November 2002 4 Scene recognition by inexact graph matching problem
Motivation and applications • Matching: graph or subgraph isomorphism – The assumption of a bijection between the elements in two instantiations of the same scene is too strong for some problems. – Usually, the model has a schematic aspect. – Construction of the image graph often relies on segmentation techniques that mail fail in accurately representing the image in meaningful entities, therefore no isomorphism can be expected. – Recognition can be better expressed as an inexact (non-bijective) graph matching problem. November 2002 5 Scene recognition by inexact graph matching
Motivation and applications • Application: – Recognition of brain structures from 3 D magnetic resonance images previously processed by a segmentation model (ENST and hospital Saint Vincent de Paul, Paris) – Model is an anatomical atlas: nodes represent the anatomical structures, while edges represent the spatial relationships between them. – Inaccuracies: oversegmented image, unexpected objects found (e. g. pathologies), expected objects not found, deformations, imprecise attributes November 2002 6 Scene recognition by inexact graph matching
Motivation and applications • Axial slices of a brain (each grey level corresponds to a unique connected structure): middle dark structures (lateral ventricles) are much bigger in (b) than in (a), the white hypersignal structure (tumor) in (b) does not appear in (a) or (c). (a) Normal November 2002 brain 7 (b) Pathological (c) Brain atlas Scene recognition by inexact graph matching brain
Motivation and applications • Other applications: – Aerial or satellite image interpretation using a map – Face recognition – Character recognition November 2002 8 Scene recognition by inexact graph matching
Optimal graph matching formulation • Attributed graphs: one single vertex for each image region • Typical attributes: adjacencies, distances, relative positions, grey levels • Vertex similarities and edge similarities are computed from the values of the attributed graphs, relating each pair of vertices and each pair of edges (one from the model, the other from the image). November 2002 9 Scene recognition by inexact graph matching
Optimal graph matching formulation • Model: G 1=(N 1, E 1) • Oversegmented image: G 2=(N 2, E 2), with |N 2|>|N 1| • Assign one single node of N 1 to each node of N 2: • Set of nodes associated with i N 1: November 2002 10 Scene recognition by inexact graph matching
Optimal graph matching formulation • Edges (a, b) E 1 and (p, q) E 2 are associated if node a N 1 is associated with p N 2 and node b N 1 is associated with q N 2. November 2002 11 Scene recognition by inexact graph matching
Optimal graph matching formulation • Search for the optimal graph matching which best represents the scene recognition is based on similarity values. November 2002 12 Scene recognition by inexact graph matching
Optimal graph matching formulation • Similarity matrices: sv(i, j ): vertex-similarity between vertices i N 1 and j N 2. se(u, v ): edge-similarity between edges u = (a, b ) E 1 and v = (c, d ) E 2. • A good matching is a solution in which the vertex associations and the edge associations correspond to high similarity values. November 2002 13 Scene recognition by inexact graph matching
Model: objective function node-similarities contribution November 2002 14 edge-similarities contribution Scene recognition by inexact graph matching
Model: objective function node-similarities contribution: High similarity associations are privileged: sv(i, j ) high xij = 1 maximizes Low similarity associations are penalized: sv(i, j ) low xij = 0 maximizes November 2002 15 Scene recognition by inexact graph matching
Model: constraints • Each vertex of N 2 has to be associated with exactly one vertex of N 1: – Image segmentation is performed by an appropriate algorithm which preserves the boundaries. – In consequence, it avoids situations in which one region of the segmented image is located in the frontier of two adjacent regions of the model and no undersegmentation occurs. Constraint (1): For every j N 2, there exists exactly one node i N 1 such that xij = 1, i. e. November 2002 16 Scene recognition by inexact graph matching
Model: constraints • Vertices of N 2 associated with the same vertex of N 1 should be all connected among themselves: – An oversegmentation method can split an object into several pieces, but it does not change their relative positions (connectivity constraint). Constraint (2): For every i N 1, the graph induced by A (i ) in G 2 = (N 2, E 2) is connected. November 2002 17 Scene recognition by inexact graph matching
Model: constraints • Pairs of vertices with null similarity cannot be associated: Constraint (3): sv(i, j ) = 0 xij = 0. • All objects in the model should appear in the image: Constraint (4): For every i N 1, there exists at least one node j N 2 such that xij = 1, i. e. November 2002 18 Scene recognition by inexact graph matching
• • Randomized construction algorithm Max. Trials: maximum number of attempts to build a feasible solution. Repeat for at most Max. Trials attempts: 1. Set A (i ) = for every i N 1. 2. Randomly select a node j N 2 and set N 2 – {j }. 3. Randomly select a node i N 1, until sv(i, j ) > 0 and the graph induced in G 2 by A (i ) {j } is connected. 4. Set A (i ) {j }. 5. If N 2 ≠ go back to step 2 above. 6. If A (i ) ≠ i N 1 and A -1(j ) ≠ j N 2, then stop. November 2002 19 Scene recognition by inexact graph matching
Randomized construction algorithm • The algorithm stops and returns the first feasible solution found within at most Max. Trials attempts. • The algorithm enforces feasibility, good solutions are not necessarily obtained. • It may also be extended to return the best among the first Max. Solutions feasible solutions built. • Each attempt has time complexity. November 2002 20 Scene recognition by inexact graph matching
Randomized construction algorithm November 2002 21 Scene recognition by inexact graph matching
Test problems • 12 test problems – Some randomly generated, others from real problems associated with medical images – Two difficult instances with isolated nodes – Largest instance has |N 1|=50 and |E 1|=88, |N 2|=250 and |E 2|=1681. • Construction algorithm is very fast and very often finds a feasible solution in the first attempt. • Instances with isolated nodes are harder, but even in these 22 cases a feasible solution is built. November 2002 Scene recognition by inexact graph matching
Test problems: I-6 (human brain) November 2002 23 Model: Image: 12 95 vertices 1434 edges 42 edges Scene recognition by inexact graph matching
Test problems: I-7 (muscle) Model: 14 vertices 27 edges November 2002 24 Image: 28 vertices 63 edges Scene recognition by inexact graph matching
Test problems: I-7 (muscle) model November 2002 oversegmented image 25 Scene recognition by inexact graph matching
Test problems: I-7 (feasible solutions) Best of 100 first Best ten first First of feasible solutions: feasible solution: solutions: 4 regions 12 regions 5 nodes correctly recognized November 2002 26 Scene recognition by inexact graph matching
Local search • Solutions built by the construction algorithm are not necessarily optimal. • Local search algorithm successively replaces the current solution by a better one in a neighborhood of the first, terminating at a local optimum. • First improving strategy: the current solution is replaced by the first neighbor whose cost function value improves that of the current solution. November 2002 27 Scene recognition by inexact graph matching
Local search: neighborhoods • Neighborhood A: all solutions that can be obtained from the current one by modifying A 1(j ) for some node j N (the new node 2 associated with j has to enforce all feasibility constraints). • Neighborhood B: all solutions that can be obtained from the current one by exchanging the nodes A -1(p) and A -1(q ) of N 1, currently associated with nodes p and q of N 2. November 2002 28 Scene recognition by inexact graph matching
Local search: neighborhoods Neighborhood A: associate a new node (a) with node 2 Neighborhood B: exchange the nodes b and c associated with nodes 4 and 7 November 2002 29 Scene recognition by inexact graph matching
Local search • VND procedure using neighborhoods A and B and a first improving strategy: 1. Start with a solution built by the construction algorithm. 2. Move to the first improving solution in neighborhood A of the current solution until a local optimum is found. 3. If the current solution is also locally optimal with respect to neighborhood B, then stop. 4. Otherwise, move to the first improving solution in neighborhood B and go back to step 2. November 2002 30 Scene recognition by inexact graph matching
Local search • Each local search iteration (neighborhood search and move) has time complexity. • Local search improved the solution value for all test problems. November 2002 31 Scene recognition by inexact graph matching
Local search: I-7 Local search applied to the best of the 100 first feasible solutions built: 13 regions correctly recognized (out of 28) November 2002 32 Scene recognition by inexact graph matching
GRASP heuristic • GRASP: multistart metaheuristic • Each iteration has two phases: construction and local search • Max. Iterations: maximum number of iterations • Repeat for Max. Iterations iterations: – Build a feasible solution using the randomized construction algorithm. – Use the local search procedure to improve the solution built. – Update the best solution found. November 2002 33 Scene recognition by inexact graph matching
GRASP heuristic • Good solutions obtained with a few hundred GRASP iterations. • Improved solutions for all test problems • Small computation times • Problem I-7, 20000 GRASP iterations: 216 seconds on a Pentium II 450 MHz November 2002 34 Scene recognition by inexact graph matching
GRASP heuristic • Solution values for α = 0. 9: construction algorithm with Max. Trials = 500, first improving local search strategy, and 200 GRASP iterations Problem |N 1| |E 1| |N 2| |E 2| Construction Local search GRASP I-6 12 42 95 1434 0. 412 0. 425 0. 426 I-7 14 27 28 63 0. 618 0. 634 0. 637 I-8 30 39 100 297 0. 498 0. 518 0. 521 I-9 50 88 250 1681 0. 505 0. 518 0. 519 November 2002 35 Scene recognition by inexact graph matching
GRASP heuristic: I-7 20000 GRASP iterations, with local search starting from the first feasible solution built in the construction phase: 26 regions correctly recognized (out of 28) November 2002 36 Scene recognition by inexact graph matching
GRASP heuristic: 1000 iterations Problem I-6 Problem I-7 November 2002 37 Scene recognition by inexact graph matching
Current and future current work • Improved construction algorithm incorporating a greedy (randomized) insertion criterion • Improvement of the GRASP heuristic using path-relinking • Parallelization on a 64 -node Linux cluster using the MPI library for communication • Comparison with other approaches in the literature, in particular with genetic algorithms November 2002 38 Scene recognition by inexact graph matching • Application to the recognition of facial
Slides and publications • Slides of this talk can be downloaded from: http: //www. inf. puc-rio/~celso/talks • Papers about the randomized construction heuristic with local search for scene recognition (shortly) available at: http: //www. inf. puc-rio. br/~celso/publicacoes (also available: other papers about GRASP, parallelization, and path-relinking) November 2002 39 Scene recognition by inexact graph matching
- Slides: 39