Content Based Image Clustering and Image Retrieval Using

Content Based Image Clustering and Image Retrieval Using Multiple Instance Learning Xin Chen Advisor: Chengcui Zhang Department of Computer and Information Sciences, University of Alabama at Birmingham Content Based Image Retrieval System Overview Image Segmentation and Feature Extraction CBIR Procedure Clustering with Genetic Algorithm: Overview • The possible solutions to a real world problem are evaluated. Each solution is given a “score”. Then they are encoded as chromosomes and fed into Genetic World. Motivation The amount of image data involved is very large. We use clustering to preprocess the data and reduce the search space in image retrieval. For this reason, we choose Genetic Algorithm in the hope that the clustering result can reach global optimum. • Preprocessing: Images are segmented into a set of image instances. Features of each instance are extracted. • Clustering: Image instances are clustered to reduce the search space of the learning algorithm. Segmentation • We use Blobworld[1] automatic image segmentation method to partition all the images in image database into several segments. Each segment represent an individual semantic region of the original image. E. g. grass, lake, deer, tree etc. • User Query: User provide a query image. • SVM Learning: One-class SVM is the similarity calculation algorithm, which learns user query and feedback and return results to the user. The objects in clustering are image segments. Therefore, if a certain image segment belongs to a cluster, the image that contains this object also belongs to this cluster. In this way, we can reduce the amount of image searched in the final retrieval process. Feature Extraction • Then we extract features for each image segment. In our experiment, 8 features are used – 3 color features, 3 texture features and 2 shape features. • Return Results: In each iteration, user give feedback to the returned results. SVM refines the retrieval results accordingly in the next iteration. • These chromosomes are parents or the tth generation. Genetic World operators (e. g. Selection, Crossover and Mutation) manipulate these chromosomes and generate the offspring i. e. the (t+1)th generation. When doing this, a mechanism is implemented to make sure that the higher the score the higher the chance a certain chromosome is inherited. • Offspring are decoded into real world solutions. • Evaluation in Real World is performed again upon the generated solutions. • The evaluated solutions are encoded and fed back into Genetic World for another round of evaluation. • End of iterations until a certain criteria is satisfied. Clustering with Genetic Algorithm: Algorithm Design Key Points in Design • Encoding • Object Function • Genetic Operators: Selection, Recombination and Mutation Selection Operator 98 56 10 23 653 357 228 469 167 823 Each integer in the allele is the ID of a centroid. Object Function Clustering is reduced to find the optimal combination. The goal is to find a set of centroids that make the function below has its minimum： • For each item, compute its fitness (i. e. the inverse value of object function): f 1, f 2, …, fn. • Total fitness: • Relative fitness of each item is fi/F. It is its probability of being inherited to the next generation. • Simulate roulette to choose pairs of chromosomes that will be used to generate the next generation. In the example below, C 1 on Roulette stands for Chromosome 1 whose fitness score is 30%. Experiment shows cut = 2/3. Adjust according to specific problem. C 1 and C 2 Y Nul l Iterate K times Randomly Generate a Small Number Y 30% C 3 Where R is the subset of P，and it has k items. d is Euclidean distance. Parents N 10% Mutation Recombination Operator Encoding For example，to partition 1000 objects into 10 clusters. Give each item an ID: 1，2，…，1000. A feasible solution is： Multiple Instance Learning N 98 20% Y 10 23 653 357 228 56 10 23 653 966 228 • where u and v are two data points. We choose to use Radial Basis Function (RBF) Machine: • Mathematically, One-Class SVM solves the following quadratic problem: • Motivation: Good image regions are good in the same way. But bad image regions are bad in their own way. • Basic Idea: The idea is to model the dense region as a “ball” – a hyper-sphere. In MIL problem, positive instances are inside the “ball” and negative instances are outside. Learning Procedure • Apply the strategy of Schölkopf’s One-Class SVM[2]. First do a mapping to transform the data into a feature space F corresponding to the kernel K: • subject to • where is the slack variable, and α (0, 1) is a parameter that controls the trade off between maximizing the distance from the origin and containing most of the data in the region created by the hyper-sphere and corresponds to the ratio of “outliers” in the training dataset. • When it is applied to the MIL problem, It is also subject the MIL Equation. • If w and are a solution to this problem, then the decision function is f(x) = sign(w× θ(x)- ρ) • It will be 1 for most examples xi contained in the training set. 167 823 469 167 823 • In each iteration, user provides feedback to the returned images. If a image is relevant, it is labeled positive, otherwise it is labeled negative. In this way, bag (image as a whole) label is known while instance (image segments) labels are unknown. But we know the relationship between these two kinds of labels. If the bag label is positive, there exist at least one positive instance label in the bag. If the bag label is negative, all the instances are negative. This can be mathematically depicted below. N • At the end of clustering we have k clusters. • Given a query image region, all the other image regions in this cluster can be located. • In some cases, the query region is also very close to image regions in another cluster. • We choose three clusters whose centroids are the closest to the query region. • We then group all the images that have at least one semantic region fall into the three clusters and take it as the reduced search space. One-Class Support Vector Machine (SVM) One-Class Support Vector Machine 469 • MIL: A training sample is a labeled bag in which the labels of the instances in the bag are unknown. Wrap-up Nul l offspring 40% 56 <cut Different from crossover operator in simple genetic algorithm, this is to guarantee there’s no repeated genes in one chromosome, i. e. centroids in one solution are all distinct. C 2 98 Randomly choose mutation point C 1 C 4 Change a gene to an item that is not included in this chromosome at a very low frequency. • Bag: Each image is a bag that has a certain number of instances. Experimental Results • Given a query image, in initial query, the user needs to identify a semantic region of interest. Since no training sample is available at this point, we simply compute the Euclidean distances between the query semantic region and all the other semantic regions in the image database. • The similarity score for each image is then set to the inverse of the minimum distance between its regions and the query region. The training sample set is then constructed according to user’s feedback. • If an image is labeled positive, its semantic region that is the least distant from the query region is labeled positive. In this way, most of the positive regions can be identified. • For some images, Blob-world may “over-segment” such that one semantic region is segmented into two or more “blobs”. In addition, some images may actually contain more than one positive region. Suppose the number of positive images is h and the number of all semantic regions in the training set is H. Then the ratio of “outliers” in the training set is set • to: • 9, 800 images with 82, 556 instances are used as test data. • We tested on 65 query images. The search space is reduced to 28. 6% on average. • In initial query, the system gets the feature vector of the query region and compares it with all the other image regions in the database using Euclidean distance. The top thirty images are returned to the user. Third Query Result • After the initial query, user gives feedback to the retrieved images and these feedbacks are returned to the system. Our One-Class SVM based algorithm learns from these feedbacks and starts another round of retrieval. • z is a small number used to adjust the α in order to alleviate the above mentioned problem. • The training set as well as the parameter α are fed into One-Class SVM to obtain and , which are used to calculate the value of the decision function for the test data. • Each image region will be assigned a “score” by in the decision function. The higher the score, the more likely this region is in the positive class. The similarity score of each image is then set to the highest score of all its regions. Retrieved Segments • Through each iteration, the number of positive images increases steadily.