Text and Contentbased Approaches to Image Retrieval for

Text- and Content-based Approaches to Image Retrieval for the Image. CLEF 2009 Medical Retrieval Track Matthew Simpson, Md Mahmudur Rahman, Dina Demner-Fushman, Sameer Antani, George R. Thoma Lister Hill National Center for Biomedical Communications, National Library of Medicine, NIH, Bethesda, MD, USA CLEF 2009

Retrieval tasks and approaches • ITI project long term goal – Find a way to combine image and text features so that the whole is greater than the sum of its parts • Ad-hoc image retrieval – – Text-based Image content-based Automatic mixed Relevance feedback mixed • Case-based document retrieval – Text-based

Text-based approach • Indexing: – Create image documents for ad-hoc image retrieval – Create surrogate documents for case-based retrieval – Index using Essie • • term normalization using the SPECIALIST Lexicon query expansion based on UMLS synonymy term weighting based on location in the document Phrase-based search

Text documents • Image document – – Title and caption provided by organizers Mention extracted from paper MEDLINE citation (abstract +Me. SH) PICO frame of the caption + image modality (structured caption summary) • Surrogate document – MEDLINE citation – caption, mention, and structured caption summary of each image contained in the article

Text retrieval • PICO-based structured query and case representation – <topic. ID>19</topic. ID> <description>Crohn's disease CT</description> – <modality essie. Exp="false">ct</modality> <mod. Syn>c. a. t. </mod. Syn><mod. Syn>cat</mod. Syn><mod. Syn>compu terised axial tomography</mod. Syn>…. – <cond essie. Exp="true">Crohn's disease</cond><cond. PN>crohn disease</cond. PN><cond. Syn>Regional enteritis</cond. Syn> <cond. Syn>eleocolitis</cond. Syn><cond. Syn>Cicatrizing enterocolitis</cond. Syn><cond. Syn>granulomatous enteritis</cond. Syn><cond. Syn>INFLAMMATORY BOWEL DISEASE</cond. Syn><cond. Syn>regional enterocolitis</cond. Syn> …

CBIR - Image feature representation • Concepts - color and texture patches from local image regions • Low-level global features – Color (Color Layout Descriptor, MPEG-7) – Edge (histogram of local edge distribution and direction) – Texture (grey level co-occurrence matrix) – Average grey level (256 -dimensional vector of blocks in image normalized to gray-level 64 x 64) – Lucene (LIRE)-based Color Edge Direction Descriptor and Fuzzy Color Texture Histogram

Image similarity computation • Category-specific – Determine image category (training set of 5000 images manually assigned to 32 mutually exclusive categories) – Use category-specific weights in linear similarity matching • Relevance feedback – Feature weights updated using images judged relevant

Combining text and image • Based on text search results, – Compute mean vector of top 5 retrieved images, use as input to category-specific retrieval – Select 3 -5 relevant images manually, use as input to category-specific retrieval – Re-rank text retrieval results using visual retrieval scores • Provide feedback using all retrieval results, – expand query using image documents – Pad selected relevant images with new retrieval results

Relevance Feedback

Results category- RF specific visual text re. BRF ranked RF RF+QE mixed case-based

Image-text search engine

Thank you! Questions?