MultimediaImage Mining Mazen Abou Khamis Department of Computer

Multimedia/Image Mining Mazen Abou. Khamis Department of Computer Science and Engineering Southern Methodist University

Outline of Presentation n n Challenges Facing Image Mining Multi. Media. Miner: Ø Ø Ø Major Components Major Functional Modules Phases Harvesting v Preprocessing v • Median Filter Technique Storing v Creating a Data Cube v Mining v Discovering v Image Retrieval v n n Conclusions Future Work 2

Challenges Facing Image Mining n n n Representing the image objects clearly and efficiently because image objects are hard to define. Thus, we have to break the image object into meaningful components such as color, texture, shape, etc… Querying the image objects after representing them to retrieve the discovered knowledge. Below are some questions to ask. . . 1. how to compose an image query object? Can we use Keywords to compose a query? 2. How to compare the query image object to the objects in the database because unlike numeric or text data, exact match can be rarely found between two image objects. Visualization techniques to view the image components in a meaningful way such as DATA CUBES. 3

Multi. Media. Miner n n It is a system prototype responsible for mining multimedia information and knowledge from large multimedia databases. It allows the users to control, combine and manipulate different types of Media. 4

MM - Major Components n n Image excavator: responsible to extract images and videos from the multimedia sources. Pre-processor: responsible to extract image features and to store pre-computed data in the database. User interface: Used for querying. Search engine: Responsible to match queries with the image and video features in the database 5

MM - Major Modules n MM-Characterizer : It provides users with a multi-level view of the data in the database with roll-up and drill-down capabilities. The figure describes the characteristics for two dimensions: the internet domain from which the media was extracted and the size of the media in bytes. 6

MM - Major Modules n n MM-Associator : This module finds a set of association rules from the relevant sets of data in an image database. Example: “What are relationships among still images, the frequent colors used in them, their size and the keyword sky. One possible association rule would be “if image is big and is related to sky, it is blue with possibility of 68%” or “if image is small and is related to sky, it is dark blue with a possibility of 55%. The figure shows a visualization of some association rules. 7

MM – Major Modules n MM-Classifier : This module classifies multimedia data based on some provided class labels. The result is an elegant classification of a large set of multimedia data and a characteristic description of each class. The figure below shows an output of the classifier module where a classification of images and frames based on their topic, with reference to the distribution of image format, is made for a given web site. 8

MM - Phases n n Harvesting: Images are retrieved from the Web along with information about the web pages in which they are found. The World-Wide Web has been chosen to be the repository of images in the experiments done to build MM for the following reasons: 1. Free 2. Available 3. Has a large collection of images 4. Legal issues !!! n CT scans is an interesting application for the discovery of association rules based on colors in these scans but getting access to CT scans from hospitals is not easy due to privacy issues. 9

MM - Phases n Pre-Processing: Ø Ø Images are processed to extract the following features: v Color v Texture v Size v Length v Width v Duration v Format Web Pages are processed to extract the following features: v Internet Domain v Image Popularity v Page Richness v Keywords 10

MM - Phases n Keywords extracted from the web pages are “cleaned” and organized into concept hierarchy. Ø Ø Ø n Unnecessary words are elimimated. Words are normalized. Word. Net is used to validate words and build the Concept Hierarchy. v Word. Net , a semantic network, in version 1. 6 has 95, 600 different word forms organized into 71, 000 word meanings interconnected with links representing subsumptions. It gives relationships between meaning of words. v Word. Net can be enriched with some specific words that are not in its list such as : “Boeing 747” or “fighter F 15”. The built concept Hierarchy of words is used to browse images and select image data sets for mining. 11

MM - Phases Example: By selecting a keyword “Boeing” from the hierarchy of the keywords, which is on the left of this figure, all the images associated with the word “Boeing” and its descendents are selected. 12

MM – Phases n Images also have to be preprocessed in order to remove or to reduce the noise which occur mainly during the image capture. The Median filter technique proves to perform well. Let’s Consider this example: Steps: 123 125 126 130 140 122 124 126 127 135 118 120 125 134 119 115 119 123 133 111 116 110 120 130 150 1. The pixel value 150 does not represent its surroundings. 2. We have to sort neighbors of 150 in a numerical order. If we choose to use 3*3 square neighborhood, the neighborhood values in sorted order will be: 115, 119, 120, 123, 124, 125, 126, 127, and 150. 3. Calculate the median and replace 150 by the median value. 13

MM - Phases n The figures. Before of the coins below shows the image. After of coins before and after applying the median filter to it. 14

MM - Phases n Storing: Information or Meta-data extracted from the images and the web pages containing these images are stored in the database. n The image is not stored by itself in the database. Only its feature descriptors are stored in the database. n Since the World-Wide Web is the source of images and because it has dynamic structure ; i. e. some new images may appear and some images may disappear. Thus, the descriptors of those images who disappear are discarded from the database. 15

MM - Phases n n n Creating a Data Cube: A Multimedia data cube is created based on different dimensions (features stored in database). In reality it is impossible for the physical data cube to hold more than a given number of dimensions because the size of the cube grows exponentially with the number of dimensions. That is, each time a dimension is added, the size of the cube is multiplied by the number of distinct values in the new dimension. For example, the number of dimensions of color by itself = 256 and there are other features having large number of dimensions. So, how the data cube addresses this problem of dimensionality? Compromise is the solution. 16

MM - Phases The third column in the table below represents the compromise. 17

MM - Phases n The number of dimensions is still too large to be handled by a data cube. So, many data cubes needed in the implementation of MM-Miner: Ø Ø Ø n 1 st cube holds the color and the texture dimensions. 2 nd cube holds mainly the size, length, and width dimensions. 3 rd cube holds the dimensions of the remaining features. It is impossible to discover the correlation, if it exists, between different dimensions in different cubes such as the color and the size because data mining algorithms work at one cube at a time. What is the solution? Overlap. To have an overlap: Ø Ø 4 th cube needed having dimensions from the above 3 cubes. The internet domain and the size dimensions have to exist in all the 4 cubes. 18

MM - Processes n n n Mining: Data Mining algorithms and OLAP technology are used to discover implicit knowledge like classification of images based on their multimedia features, correlation between multimedia features, etc… Discovering: Classification, Association, and Summarization of image data is discovered. Moreover, Slicing, Drilling-down and Rolling-up allow identifying specific images by clusters. Image Retrieval: After the data mining process filters out the interesting images, these images and the web pages containing them can be retrieved now. 19

MM – Phases (Summary) 20

Conclusions n n n The user interface of the 3 major modules of Multi. Media. Miner allows interactive mining. Tradeoff exists to implement the data Cube. v Keywords are not represented in the data cube. v Choose only the most frequent values of some features such as color, texture, etc… in order to reduce the number of dimensions. v Designing many data cubes instead of just one data cube. Noise must be removed before storing the descriptors of the image in the database. 21

Future Work n n n A new model for data cube materialization is under study. Adding a clustering module which would group images into different clusters based on their features. Enhancing the system to work not only with images and videos but with audio data as well. 22

THE END 23