The Knowledge Discovery Process Data Preparation Preprocessing Bamshad

The Knowledge Discovery Process; Data Preparation & Preprocessing Bamshad Mobasher De. Paul University

The Knowledge Discovery Process - The KDD Process 2

Steps in the KDD Process i Learning the application domain 4 Translate the business problem into a data mining problem i Gathering and integrating of data i Cleaning and preprocessing data 4 may be the most resource intensive part i Reducing and selecting data 4 find useful features, dimensionality reduction, etc. i Choosing functions of data mining 4 summarization, classification, regression, association, clustering, etc. i i Choosing the mining algorithm(s) Data mining: discover patterns of interest; construct models Evaluate patterns and models Interpretation: analysis of results 4 visualization, alteration, removing redundant patterns, querying i Use of discovered knowledge 3

Data Mining: What Kind of Data? i. Structured Databases 4 relational, object-relational, etc. 4 can use SQL to perform parts of the process e. g. , SELECT count(*) FROM Items WHERE type=video GROUP BY category 4

Data Mining: What Kind of Data? i Flat Files 4 most common data source 4 can be text (or HTML) or binary 4 may contain transactions, statistical data, measurements, etc. i Transactional databases 4 set of records each with a transaction id, time stamp, and a set of items 4 may have an associated “description” file for the items 4 typical source of data used in market basket analysis 5

Data Mining: What Kind of Data? i Other Types of Databases 4 legacy databases 4 multimedia databases (usually very high-dimensional) 4 spatial databases (containing geographical information, such as maps, or satellite imaging data, etc. ) 4 Time Series Temporal Data (time dependent information such as stock market data; usually very dynamic) i World Wide Web 4 basically a large, heterogeneous, distributed database 4 need for new or additional tools and techniques hinformation retrieval, filtering and extraction hagents to assist in browsing and filtering h. Web content, usage, and structure (linkage) mining tools 6

Data Mining: What Kind of Data? i Data Warehouses 4 a data warehouse is a repository of data collected from multiple data sources (often heterogeneous) so that analysis can be done under the same unified schema 4 data from the different sources are loaded, cleaned, transformed and integrated 4 allows for interactive analysis and decision-making 4 usually modeled by a multi-dimensional data structure (data cube) to facilitate decision-making 4 aggregate values can be pre-computed and stored along many dimensions 4 each dimension of the data cube contains a hierarchy of values for one attribute 4 data cubes are well suited for fast interactive querying and analysis of data at different conceptual levels, known as On-Line Analytical Processing (OLAP) 4 OLAP operations allow the navigation of data at different levels of abstraction, such as drill-down, roll-up, slice, dice, etc. 7

DM Tasks: Classification i Classifying observations/instances into different “given” classes (i. e. , classification is “supervised”) 4 example, classifying credit applicants as low, medium, or high risk 4 normally use a training set where all objects are already associated with known class labels 4 classification algorithm learns from the training set and builds a model 4 the model is used to classify new objects i Suitable data mining tools 4 Decision Tree algorithms (CHAD, C 5. 0, and CART) 4 Memory-based reasoning 4 Neural Network i Example (Hypothetical Video Store) 4 build a model of users based their rental history (returning on-time, payments, etc. ) 4 observe the current customer’s rental and payment history 4 decide whether should charge a deposit to current customer 8

DM Tasks: Prediction i Same as classification 4 except classify according to some predicted or estimated future value i In prediction, historical data is used to build a (predictive) model that explains the current observed behavior 4 Model can then be applied to new instances to predict future behavior or forecast the future value of some missing attribute i Examples 4 predicting the size of balance transfer if the prospect accepts the offer 4 predicting the load on a Web server in a particular time period i Suitable data mining tools 4 Association rule discovery 4 Memory-based reasoning 4 Decision Trees 4 Neural Networks 9

DM Tasks: Affinity Grouping i Determine what items often go together (usually in transactional databases) i Often Referred to as Market Basket Analysis 4 used in retail for planning arrangement on shelves 4 used for identifying cross-selling opportunities 4 “should” be used to determine best link structure for a Web site i Examples 4 people who buy milk and beer also tend to buy diapers 4 people who access pages A and B are likely to place an online order i Suitable data mining tools 4 association rule discovery 4 clustering 4 Nearest Neighbor analysis (memory-based reasoning) 10

DM Tasks: Clustering i Like classification, clustering is the organization of data into classes 4 however, class labels are unknown and it is up to the clustering algorithm to discover acceptable classes 4 also called unsupervised classification, because the classification is not dictated by given class labels i There are many clustering approaches 4 all based on the principle of maximizing the similarity between objects in a same class (intra-class similarity) and minimizing the similarity between objects of different classes (inter-class similarity) i Examples: 4 doing market segmentation of customers based on buying patterns and demographic attributes 4 grouping user transactions on a Web site based their navigational patterns 11

Characterization & Discrimination i Data characterization is a summarization of general features of objects in a target class 4 The data relevant to a target class are retrieved by a database query and run through a summarization module to extract the essence of the data at different levels of abstraction 4 example: characterize the video store’s customers who regularly rent more than 30 movies a year i Data discrimination is used to compare of the general features of objects between two classes 4 comparison relevant features of objects between a target class and a contrasting class 4 example: compare the general characteristics of the customers who rented more than 30 movies in the last year with those who rented less than 5 4 The techniques used for data discrimination are very similar to the techniques used for data characterization with the exception that data discrimination results include comparative measures 12

Example: Moviegoer Database 13

Example: Moviegoer Database SELECT moviegoers. name, moviegoers. sex, moviegoers. age, sources. source, movies. name FROM movies, sources, moviegoers WHERE sources. source_ID = moviegoers. source_ID AND movies. movie_ID = moviegoers. movie_ID ORDER BY moviegoers. name; moviegoers. name sex age source movies. name Amy Andrew Andy Anne Ansje Beth Bob Brian Candy Cara Cathy Charles Curt David Erica f m m f f f m m m f 27 25 34 30 25 30 51 23 29 25 30 40 23 Oberlin Oberlin Pinewoods Oberlin Mt. Auburn Oberlin MRJ Mt. Auburn Independence Day 12 Monkeys The Birdcage Trainspotting I Shot Andy Warhol Chain Reaction Schindler's List Super Cop Eddie Phenomenon The Birdcage Kingpin T 2 Judgment Day Independence Day Trainspotting 14

Example: Moviegoer Database i Classification 4 determine sex based on age, source, and movies seen 4 determine source based on sex, age, and movies seen 4 determine most recent movie based on past movies, age, sex, and source i Prediction or Estimation 4 for predict, need a continuous variable (e. g. , “age”) 4 predict age as a function of source, sex, and past movies 4 if we had a “rating” field for each moviegoer, we could predict the rating a new moviegoer gives to a movie based on age, sex, past movies, etc. i Clustering 4 find groupings of movies that are often seen by the same people 4 find groupings of people that tend to see the same movies 4 clustering might reveal relationships that are not necessarily recorded in the data (e. g. , we may find a cluster that is dominated by people with young children; or a cluster of movies that correspond to a particular genre) 15

Example: Moviegoer Database i Affinity Grouping 4 market basket analysis (MBA): “which movies go together? ” 4 need to create “transactions” for each moviegoer containing movies seen by that moviegoer: name TID Transaction Amy Andrew Andy Anne … 001 002 003 004 … {Independence Day, Trainspotting} {12 Monkeys, The Birdcage, Trainspotting, Phenomenon} {Super Cop, Independence Day, Kingpin} {Trainspotting, Schindler's List}. . . 4 may result in association rules such as: {“Phenomenon”, “The Birdcage”} ==> {“Trainspotting”} {“Trainspotting”, “The Birdcage”} ==> {sex = “f”} i Sequence Analysis 4 similar to MBA, but order in which items appear in the pattern is important 4 e. g. , people who rent “The Birdcage” during a visit tend to rent “Trainspotting” in the next visit. 16

The Knowledge Discovery Process - The KDD Process 17

Data Preprocessing i Why do we need to prepare the data? 4 In real world applications data can be inconsistent, incomplete and/or noisy h. Data entry, data transmission, or data collection problems h. Discrepancy in naming conventions h. Duplicated records h. Incomplete data h. Contradictions in data i What happens when the data can not be trusted? 4 Can the decision be trusted? Decision making is jeopardized 4 Better chance to discover useful knowledge when data is clean 18

Data Preprocessing 19

Data Preprocessing Data Cleaning Data Integration -2, 32, 100, 59, 48 -0. 02, 0. 32, 1. 00, 0. 59, 0. 48 Data Transformation Data Reduction 20

Data Cleaning i Real-world application data can be incomplete, noisy, and inconsistent 4 No recorded values for some attributes 4 Not considered at time of entry 4 Random errors 4 Irrelevant records or fields i Data cleaning attempts to: 4 Fill in missing values 4 Smooth out noisy data 4 Correct inconsistencies 4 Remove irrelevant data 21

Dealing with Missing Values i Data is not always available (missing attribute values in records) 4 equipment malfunction 4 deleted due to inconsistency or misunderstanding 4 not considered important at time of data gathering i Solving Missing Data 4 Ignore the record with missing values; 4 Fill in the missing values manually; 4 Use a global constant to fill in missing values (NULL, unknown, etc. ); 4 Use the attribute value mean to filling missing values of that attribute; 4 Use the attribute mean for all samples belonging to the same class to fill in the missing values; 4 Infer the most probable value to fill in the missing value hmay need to use methods such as Bayesian classification or decision trees to automatically infer missing attribute values 22

Smoothing Noisy Data i The purpose of data smoothing is to eliminate noise Original Data for “price” (after sorting): 4, 8, 15, 21, 24, 25, 28, 34 Binning Each value in a bin is replaced by the mean value of the bin. Partition into equidepth bins Bin 1: 4, 8, 15 Bin 2: 21, 24 Bin 3: 25, 28, 34 means Bin 1: 9, 9, 9 Bin 2: 22, 22 Bin 3: 29, 29 boundaries Bin 1: 4, 4, 15 Bin 2: 21, 24 Bin 3: 25, 34 Min and Max values in each bin are identified (boundaries). Each value in a bin is replaced with the closest boundary value. 23

Smoothing Noisy Data i Other Methods Clustering Regression Similar values are organized into groups (clusters). Values falling outside of clusters may be considered “outliers” and may be candidates for elimination. Fit data to a function. Linear regression finds the best line to fit two variables. Multiple regression can handle multiple variables. The values given by the function are used instead of the original values. 24

Smoothing Noisy Data - Example Want to smooth “Temperature” by bin means with bins of size 3: 1. 2. 3. 4. First sort the values of the attribute (keep track of the ID or key so that the transformed values can be replaced in the original table. Divide the data into bins of size 3 (or less in case of last bin). Convert the values in each bin to the mean value for that bin Put the resulting values into the original table 25

Smoothing Noisy Data - Example Value of every record in each bin is changed to the mean value for that bin. If it is necessary to keep the value as an integer, then the mean values are rounded to the nearest integer. 26

Smoothing Noisy Data - Example The final table with the new values for the Temperature attribute. 27

Data Integration i Data analysis may require a combination of data from multiple sources into a coherent data store i Challenges in Data Integration: 4 Schema integration: CID = C_number = Cust-id = cust# 4 Semantic heterogeneity 4 Data value conflicts (different representations or scales, etc. ) 4 Synchronization (especially important in Web usage mining) 4 Redundant attributes (redundant if it can be derived from other attributes) -may be able to identify redundancies via correlation analysis: Correlation analysis: Pr(A, B) / (Pr(A). Pr(B)) = 1: independent, > 1: positive correlation, < 1: negative correlation. i Meta-data is often necessary for successful data integration 28

Normalization i Min-max normalization: linear transformation from v to v’ 4 v’ = [(v - min)/(max - min)] x (newmax - newmin) + newmin 4 Ex: transform $30000 between [10000. . 45000] into [0. . 1] ==> [(30000 – 10000) / 35000] x (1 - 0) + 0 = 0. 514 i z-score normalization: normalization of v into v’ based on attribute value mean and standard deviation 4 v’ = (v - Mean) / Standard. Deviation i Normalization by decimal scaling 4 moves the decimal point of v by j positions such that j is the minimum number of positions moved so that absolute maximum value falls in [0. . 1]. 4 v’ = v / 10 j 4 Ex: if v ranges between -56 and 9976, j=4 ==> v’ ranges between -0. 0056 and 0. 9976 29

Normalization: Example i z-score normalization: v’ = (v - Mean) / Stdev i Example: normalizing the “Humidity” attribute: Mean = 80. 3 Stdev = 9. 84 30

Normalization: Example II i Min-Max normalization on an employee database 4 max distance for salary: 100000 -19000 = 81000 4 max distance for age: 52 -27 = 25 4 New min for age and salary = 0; new max for age and salary = 1 31

Data Reduction i Data is often too large; reducing data can improve performance i Data reduction consists of reducing the representation of the data set while producing the same (or almost the same) results i Data reduction includes: 4 Data cube aggregation 4 Dimensionality reduction 4 Discretization 4 Numerosity reduction h. Regression h. Histograms h. Clustering h. Sampling 32

Data Cube Aggregation i Reduce the data to the concept level needed in the analysis 4 Use the smallest (most detailed) level necessary to solve the problem i Queries regarding aggregated information should be answered using data cube when possible 33

Dimensionality Reduction i Feature selection (i. e. , attribute subset selection): 4 Select only the necessary attributes. 4 The goal is to find a minimum set of attributes such that the resulting probability distribution of data classes is as close as possible to the original distribution obtained using all attributes. 4 Exponential number of possibilities. i Use heuristics: select local ‘best’ (or most pertinent) attribute, e. g. , using information gain, etc. 4 step-wise forward selection {}{A 1, A 3}{A 1, A 3, A 5} 4 step-wise backward elimination {A 1, A 2, A 3, A 4, A 5}{A 1, A 3, A 4, A 5} {A 1, A 3, A 5} 4 combining forward selection and backward elimination 4 decision-tree induction 34

Decision Tree Induction 35

Numerocity Reduction i Reduction via histograms: 4 Divide data into buckets and store representation of buckets (sum, count, etc. ) i Reduction via clustering 4 Partition data into clusters based on “closeness” in space 4 Retain representatives of clusters (centroids) and outliers i Reduction via sampling 4 Will the patterns in the sample represent the patterns in the data? 4 Random sampling can produce poor results 4 Stratified sample (stratum = group based on attribute value) 36

Sampling Techniques OR SRSW random e (simpl without e sampl ent) em replac Raw Data SRSWR Cluster/Stratified Sample Raw Data 37

Discretization i 3 Types of attributes 4 nominal - values from an unordered set (also “categorical” attributes) 4 ordinal - values from an ordered set 4 continuous - real numbers (but sometimes also integer values) i Discretization is used to reduce the number of values for a given continuous attribute 4 usually done by dividing the range of the attribute into intervals 4 interval labels are then used to replace actual data values i Some data mining algorithms only accept categorical attributes and cannot handle a range of continuous attribute value i Discretization can also be used to generate concept hierarchies 4 reduce the data by collecting and replacing low level concepts (e. g. , numeric values for “age”) by higher level concepts (e. g. , “young”, “middle aged”, “old”) 38

Discretization - Example i Example: discretizing the “Humidity” attribute using 3 bins. Low = 60 -69 Normal = 70 -79 High = 80+ 39

Converting Categorical Attributes to Numerical Attributes: Outlook (overcast, rain, sunny) Temperature real Humidity real Windy (true, false) Standard Spreadsheet Format Create separate columns for each value of a categorical attribute (e. g. , 3 values for the Outlook attribute and two values of the Windy attribute). There is no change to the numerical attributes. 40

Visualizing Patterns i Example: Cross Tabulation Windy Not Windy Outlook = sunny 2 3 Outlook = rain 2 3 Outlook = overcast 2 2 41

Evaluating Models i To train and evaluate models, data are often divided into three sets: the training set, the test set, and the evaluation set i Training Set 4 is used to build the initial model 4 may need to “enrich the data” to get enough of the special cases i Test Set 4 is used to adjust the initial model 4 models can be tweaked to be less idiosyncrasies to the training data and can be adapted for a more general model 4 idea is to prevent “over-training” (i. e. , finding patterns where none exist). i Evaluation Set 4 is used to evaluate the model performance 42

Training Sets i The training set will be used to train the models 4 most important consideration: need to cover the full range of values for all the features that the model might encounter 4 good idea to have several examples for each value of a categorical feature and for a range of values for numerical features i Data Enrichment 4 training set should have sufficient number of examples of rare events 4 a random sample of available data is not sufficient since common examples will swamp the rare examples 4 training set needs to over-sample the rare cases so that the model can learn the features for rare events instead of predicting everything to be common outputs 43

Test and Evaluation Sets i Reading too much into the training set (overfitting) 4 common problem with most data mining algorithms 4 resulting model works well on the training set but performs miserably on unseen data 4 test set can be used to “tweak” the initial model, and to remove unnecessary inputs or features i Evaluations Set for Final Performance Evaluation 4 test set can not be used to evaluate model performance for future unseen data 4 error rate on the evaluation is a good predictor of error rate on unseen data i Insufficient data to divide into three disjoint sets? 4 In such cases, validation techniques can play a major role h. Cross Validation h. Bootstrap Validation 44

Cross Validation i Cross validation is a heuristic that works as follows 4 randomly divide the data into n folds, each with approximately the same number of records 4 create n models using the same algorithms and training parameters; each model is trained with n-1 folds of the data and tested on the remaining fold 4 can be used to find the best algorithm and its optimal training parameter i Steps in Cross Validation 4 1. Divide the available data into a training set and an evaluation set 4 2. Split the training data into n folds 4 3. Select an architecture (algorithm) and training parameters 4 4. Train and test n models 4 5. Repeat step 2 to 4 using different architectures and parameters 4 6. Select the best model 4 7. Use all the training data to train the model 4 8. Assess the final model using the evaluation set 45

Bootstrap Validation i Based on the statistical procedure of sampling with replacement 4 data set of n instances is sampled n times (with replacement) to give another data set of n instances 4 since some elements will be repeated, there will be elements in the original data set that are not picked 4 these remaining instances are used as the test set i How many instances in the test set? 4 Probability of not getting picked in one sampling = 1 - 1/n 4 Pr(not getting picked in n samples) = (1 -1/n)n = e-1 = 0. 368 4 so, for large data set, test set will contain about 36. 8% of instances 4 to compensate for smaller training sample (63. 2%), test set error rate is combined with the re-substitution error in training set: e = (0. 632 * e test instance) + (0. 368 * e training instance) i Bootstrap validation increases variance that can occur in each fold 46

Measuring Effectiveness i Predictive models are measured based on the accuracy of their predictions on unseen data i Classification and Prediction Tasks 4 accuracy measured in terms of error rate (usually % of records classified incorrectly) 4 error rate on a pre-classified evaluation set estimates the real error rate 4 can also use cross-validation methods discussed before i Estimation Effectiveness 4 difference between predicted scores and the actual results (from evaluation set) 4 the accuracy of the model is measured in terms of variance (i. e. , average of the squared differences) 4 to be able to express this in terms of the original units, compute the standard deviation (i. e. , square root of the variance) 47

Ordinal or Nominal Outputs i When the output field is ordinal or nominal (e. g. , in two-class prediction), we use the classification table, the so-called confusion matrix to evaluate the resulting model i Example Predicted Class Actual Class 4 Overall correct classification rate = (18 + 15) / 38 = 87% 4 Given T, correct classification rate = 18 / 20 = 90% 4 Given F, correct classification rate = 15 / 18 = 83% 48

Measuring Effectiveness i Market Basket Analysis 4 MBA may be used for estimation or prediction (e. g. , “people who buy milk and diapers tend to also buy beer”) 4 confidence: percentage of the time “beer” occurs in transaction that contain “milk” and “diapers” (i. e. , conditional probability) 4 support: percentage of the time “milk”, “diaper”, and “beer” occurred together in the whole training set (i. e. , prior probability) i Distance-Based Techniques 4 clustering and memory-based reasoning: a measure of distance is used to evaluate the “closeness” or “similarity” of a point to cluster centers or a neighborhood 4 regression: line of best fit minimizes the sum of the distances between the line and the observations 4 often for numeric variables, can use Euclidean distance measure (square root of the sum of the squares of the difference along each dimension) 49

Measuring Effectiveness: Lift Charts i usually used for classification, but can be adopted to other methods 4 measure change in conditional probability of a target class when going from the general population (full test set) to a biased sample: 4 Example: h suppose expected response rate to a direct mailing campaign is 5% in the training set h use classifier to assign “yes” or “no” value to a target class “predicted to respond” h the “yes” group will contain a higher proportion of actual responders than the test set h suppose the “yes” group (our biased sample) contains 50% actual responders h this gives lift of 10 = 0. 5 / 0. 05 4 What if the lift sample is too small h need to increase the sample size h trade-off between lift and sample size No. of respondents ing mail ted targe ing lift s as M il ma Sample size 50