Association Rule Mining on MultiMedia Data Auto Annotation

Association Rule Mining on Multi-Media Data Auto Annotation on Images Bhavika Patel Hau San Si Tou Juveria Kanodia Muhammad Ahmad

Auto Annotation on Images l l This project is on performing Association Rule Mining on Multi-relational, Multimedia Data, particularly pictures and text. Corpus: a group of 798 pictures of different kinds such as art, landscape … with descriptions Generate association rules on image data (the RGBY values), and on text data separately. Propose an algorithm to link these two different domains together. Goal: return words that will describe a given unknown picture

Offline Processing Collect 789 pictures (. jpg, . bmp) with picture descriptions Picture descriptions are saved in a file (pic. Description. txt) with the format: <pic. ID> picture description Extract keywords from the picture descriptions. Run through Keyword. Extractor program to remove stop words and duplicate keywords. (keyword. List. dat) Format: kewyrod 1 keyword 2 keyword 3 Run through Apriori implementation to generate association rules Each picture is saved with the name of its unique ID. Run the pictures through a program to extract features (R, G, B, Y, orientation, intensity) values Format: R G B Y 0 45 90 135 I Run the generated feature table through alterfeature program to append a unique identify for each values, as well as changing the values to relative percentage. Run through Apriori implementation to generate association rules

Multi-Arm Program Read in the 9 feature values extracted from a given image Look for all association rules in the file containing the rules on image data, with these 9 feature values as the body. Check in the feature table to find out all the pictures that have these feature values Obtain the keywords associated with each picture identified Output all the keywords as descriptive/related words for the given image Search for all association rules in the file containing the rules on text data, with any of these keywords

Association Rules on Text Rules Implies RAY <- CHANDRA Body Support % Confidence % CHANDRA 2. 90% 87. 00% <- RAY 3. 10% 80. 00% PAINTING <- PAINT 2. 50% 80. 00% GUIDE <- VE 2. 80% 90. 90% ARTIST <- VE 2. 80% 90. 90% PAINTING <- VE 2. 80% 95. 50% PAINTING <- COLOURS 4. 30% 82. 40% PAINTING <- GUIDE 6. 10% 95. 90% PAINTING <- ARTIST 8. 10% 83. 10% RAY <- CHANDRA IMAGE 2. 10% 88. 20% ARTIST <- VE GUIDE 2. 50% 95. 00% GUIDE <- VE ARTIST 2. 50% 95. 00% PAINTING <- VE GUIDE 2. 50% 100. 00% GUIDE <- VE PAINTING 2. 60% 95. 20% PAINTING <- VE ARTIST 2. 50% 100. 00% ARTIST <- VE PAINTING 2. 60% 95. 20% GUIDE <- FACE PAINTING 2. 00% 81. 20% ARTIST <- FACE PAINTING 2. 00% 81. 20% PAINTING <- COLOUR ARTIST 2. 00% 93. 80% ARTIST <- COLOURS GUIDE 2. 80% 86. 40% PAINTING <- COLOURS GUIDE 2. 80% 100. 00% PAINTING <- COLOURS ARTIST 3. 10% 96. 00%

Association Rules on Text Rules Implie s Body Support % Confidence % ARTIST <- COLOURS PAINTING 3. 50% 85. 70% ARTIST <- TOP GUIDE 2. 30% 88. 90% GUIDE <- TOP ARTIST 2. 30% 88. 90% PAINTING <- TOP GUIDE 2. 30% 100. 00% GUIDE <- TOP PAINTING 2. 60% 85. 70% PAINTING <- TOP ARTIST 2. 30% 100. 00% ARTIST <- TOP PAINTING 2. 60% 85. 70% PAINTING <- WORK ARTIST 2. 30% 83. 30% PAINTING <- GUIDE ARTIST 4. 90% 100. 00% ARTIST <- GUIDE PAINTING 5. 90% 83. 00% PAINTING <- VE GUIDE ARTIST 2. 40% 100. 00% ARTIST <- VE GUIDE PAINTING 2. 50% 95. 00% GUIDE <- VE ARTIST PAINTING 2. 50% 95. 00% PAINTING <- COLOURS GUIDE ARTIST 2. 40% 100. 00% ARTIST <- COLOURS GUIDE PAINTING 2. 80% 86. 40% PAINTING <- TOP GUIDE ARTIST 2. 00% 100. 00% ARTIST <- TOP GUIDE PAINTING 2. 30% 88. 90% GUIDE <- TOP ARTIST PAINTING 2. 30% 88. 90%

Association Rules on Image Data Rules Implies 1 D 135 <- 1 D 90 Body Support % Confidence % 0 B 1 D 0 2. 20% 82. 40% <- 0 B 1 D 0 2. 20% 94. 10% 1 D 45 <- 0 B 1 D 0 2. 20% 88. 20% 1 I <- 0 B 1 D 0 2. 20% 100. 00% 1 D 90 <- 0 B 1 D 135 2. 70% 85. 70% 1 D 135 <- 0 B 1 D 90 2. 70% 85. 70% 1 D 45 <- 0 B 1 D 135 2. 70% 85. 70% 1 D 135 <- 0 B 1 D 45 2. 80% 81. 80% 1 I <- 0 B 1 D 135 2. 70% 90. 50% 1 D 45 <- 0 B 1 D 90 2. 70% 90. 50% 1 D 90 <- 0 B 1 D 45 2. 80% 86. 40% 1 I <- 0 B 1 D 90 2. 70% 100. 00% 1 D 90 <- 0 B 1 I 3. 00% 87. 50% 1 I <- 0 B 1 D 45 2. 80% 86. 40% 1 I <- 1 R 1 D 0 2. 40% 84. 20% 1 D 45 <- 1 R 1 D 135 2. 80% 81. 80% 1 D 45 <- 1 R 1 D 90 2. 40% 84. 20% 1 I <- 1 R 1 D 90 2. 40% 84. 20% 1 D 90 <- 1 Y 1 D 135 3. 20% 80. 00% 1 D 90 <- 1 D 0 1 D 135 13. 60% 86. 90% 1 D 135 <- 1 D 0 1 D 90 13. 60% 86. 90% 1 D 0 <- 1 D 135 1 D 90 14. 40% 81. 60% 1 D 45 <- 1 D 0 1 D 135 13. 60% 83. 20%

Association Rules on Image Data Rules Implies 1 D 135 <- 1 I Body Support % Confidence % 1 D 0 1 D 45 12. 40% 90. 80% <- 1 D 0 1 D 135 13. 60% 90. 70% 1 D 135 <- 1 D 0 1 I 14. 70% 83. 60% 1 D 0 <- 1 D 135 1 I 14. 80% 82. 90% 1 D 45 <- 1 D 0 1 D 90 13. 60% 82. 20% 1 D 90 <- 1 D 0 1 D 45 12. 40% 89. 80% 1 D 0 <- 1 D 90 1 D 45 13. 90% 80. 00% 1 I <- 1 D 0 1 D 90 13. 60% 88. 80% 1 D 90 <- 1 D 0 1 I 14. 70% 81. 90% 1 I <- 1 D 0 1 D 45 12. 40% 91. 80% 1 D 45 <- 1 D 135 1 D 90 14. 40% 86. 00% 1 D 90 <- 1 D 135 1 D 45 15. 00% 83. 10% 1 D 135 <- 1 D 90 1 D 45 13. 90% 89. 10% 1 I <- 1 D 135 1 D 90 14. 40% 86. 00% 1 D 90 <- 1 D 135 1 I 14. 80% 83. 80% 1 D 135 <- 1 D 90 1 I 15. 20% 81. 70% 1 I <- 1 D 135 1 D 45 15. 00% 83. 90% 1 D 45 <- 1 D 135 1 I 14. 80% 84. 60% 1 D 135 <- 1 D 45 1 I 15. 20% 82. 50% 1 I <- 1 D 90 1 D 45 13. 90% 89. 10% 1 D 45 <- 1 D 90 1 I 15. 20% 81. 70% 1 D 90 <- 1 D 45 1 I 15. 20% 81. 70% 1 D 135 <- 1 D 0 1 D 45 12. 40% 90. 80%

# of text association rules generated from different combination of min supp & conf

# of image association rules generated from different combination of min supp & conf

Single pass rebuild l l Specify common key Rebuild the tables based on the common key Use Apriori EXAMPLE: Table 1: purchase(customer, item, amount) item(customer, item_id) Table 2 purchase_total(customer, items) Query: Customers who buy a lot of stuff what do they usually but? purchase_total(X, items) return item(X, item_id)

Conclusion l l l So we have a partial solution multimedia ARM problem, however there many things that can be done further, to improve upon it. Need to find a way to restrict the number of keywords that we get. Need to find an easier method than the present lookup method, as too many files are involved. Need for an efficient data structure to do the above point. Alternative Schemes

The End Please visit our project’s website at http: //www. cs. rit. edu/~p 759 -06 c to find detailed information.

Questions?