A Kit For Knowledge Discovery Data Data everywhere

A Kit For Knowledge Discovery

Data, Data everywhere yet. . . § I can’t find the data I need § data is scattered over the network § many versions, subtle differences z I can’t get the data I need y need an expert to get the data z I can’t understand the data I found y available data poorly documented z I can’t use the data I found y results are unexpected y data needs to be transformed from one form to other 2

? • There are sequence of steps (with eventual feedback loops) that should be followed to discover knowledge (e. g. , patterns) in data. • Achieving Standardized Process Model

What is KDD ? Knowledge Discovery in Data is the significant method of evaluating 1 2 Legitimate Probably Innovative useful 3 Accurate understandable patterns in data.

Knowledge Discovery Process Da ta Raw. Data Mi nin g Integration Tran sfo __ __ __ rm a Se __ __ __ tio n & lecti __ __ __ Cl on ea nin g Transformed Target Data Knowledge Patterns and Rules Understanding DATA Ware house Interpretation & Evaluation

Outcomes of Data Mining Forecasting Future Classification on Recognizing patterns Clustering Based On Attributes Events Correlation – Association Sequencing Events ~ Later Predictions

Data Mining § Look for hidden patterns and trends in data that is not immediately apparent from summarizing the data

Data Mining + Data = Interestingness criteria Hidden patterns

Data Mining Type of Patterns + Data = Interestingness criteria Hidden patterns

Data Mining Type of data Type of Interestingness criteria + Data = Interestingness criteria Hidden patterns

What is a Data Warehouse? A single, complete and consistent store of data obtained from a variety of different sources made available to end users in a what they can understand use in a business context.

What is Data Warehousing? Information A process of transforming data into information and making it available to users in a timely enough manner to make a difference Data 12

Data Mining Process 1 2 3 4 1. Problem Definition 2. Data Integration & Cleaning 3. Model Framing & Evaluation 4. Knowledge Discovery

Basic Operations in DM Data Mining Task Predictive: Regression Classification Collaborative Filtering Descriptive: Clustering / Similarity Matching Association rules Deviation detection

Why Machine Learning Growing flood of online data Budding industry Progress in algorithms and theory • Data mining: using historical data to improve decision – medical records ⇒ medical knowledge – log data to model user • Software applications we can’t program by hand – autonomous driving – speech recognition • Self customizing programs – Newsreader that learns user interests

Machine Learning Supervised Discover patterns in the data. Presence of Target Attribute Unsupervised Text Unsupervised Supervised Data Mining Data have no target attribute. Explore Data to find Patterns

Applications Of Data Mining

Applications of Data Mining §Fraud/Non-Compliance Anomaly detection § Isolate the factors that lead to fraud, waste and abuse § Target auditing and investigative efforts more effectively §Credit/Risk Scoring §Intrusion detection §Recruiting/Attracting customers §Maximizing profitability (cross selling, identifying profitable customers) §Service Delivery and Customer Retention § Build profiles of customers likely to use which services

Tools For Data Mining § § § Link. Out NCBI Sequin Rapid Miner Lib. Svm ADa. M etc….

Why Weka § Weka is a collection of machine learning algorithms for data mining tasks. § The algorithms can either be applied directly to a dataset or called from your own Java code. § Weka contains tools for data pre-processing, classification, regression, clustering, association rules, and visualization. § It is also well-suited for developing new machine learning schemes.

About WEKA § Waikato Environment for Knowledge Analysis (WEKA) § Developed by the Department of Computer Science, University of Waikato, New Zealand § Machine learning/data mining software coded in Java § Used for research, education, and applications § Exclusively for KDD. § Various Versions are available such as Version 2. 3, 1998; Version 3. 0, 1999; Version 3. 4, 2003; Version 3. 6, 2008.

Weka GUI Chooser

A Vital Part In Weka Explorer ww. themegallery. com

Weka !!!! § Weka is a collection of machine learning algorithms for data mining tasks. § The algorithms can either be applied directly to a dataset or called from your own Java code. § Weka contains tools for data pre-processing, classification, regression, clustering, association rules, and visualization. § Perfectly suited for developing new machine learning schemes.

Weka’s Structural Layout er lor p Ex Experimenter An environment for exploring data with WEKA Performing experiments and conducting statistical tests between learning schemes Knowledge Flow Supports the same functions as the Explorer but with dragand-drop Simple CLI Provides a simple commandline interface that allows direct execution of WEKA

Algorithms www. themegallery. com

WEKA ! File WEKA stores data in flat files (ARFF format). Easy to transform EXCEL file to ARFF format. ARFF file consists of a list of instances ARFF file can be created using Notepad or Word. Name of the dataset is with @relation Attribute information is with @attribute Data is with @data. Attribute Relation File Format (ARFF)

Sample ARFF

Intrinsic Operations Select Attributes 5 Associate 4 Cluster 3 Classify 2 Preprocess 1

Preprocessing § Changing Data formats as per the Needs. § Varies as Per Mining Datasets. § Some of the Preprocessing Steps Ø Adding/removing attributes Ø Attribute value substitution Ø Discretization (MDL, Kononenko, etc. ) Ø Time series filters (delta, shift) Ø Sampling, randomization Ø Missing value management Ø Normalization and other numeric transformations

Algorithms

Pre-Processing Opening Files Browse for the data file in local file system. Current Relation ØRelations ØInstances ØSchema Operations ØAttributes ØFilters

Weka – Formulating Files

Dataset -. txt Format

Weka ~ Dataset’s

Missing Values

Generic. Object. Editor v A Property Editor for objects as editable in the Generic. Object. Editor configuration file, which lists possible values that can be selected from, and themselves configured. v. The configuration file is called "Generic. Object. Editor. props" and may live in either the location given by "user. home" or the current directory (this last will take precedence), and a default properties file is read from the weka distribution.

Weka ~ Generic. Object. Editor v. This Editor allows configure a filter. v. Same kind of dialog box is used to configure other objects, such as classifiers and clusterers.

Sample - Cluster Attributes for Cluster

Weka’s Viewer

PCA Analysis

Pre-Processing Retrievals Before After

Retrieving Significant Attributes

Select Attribute !

Algorithms

Feature Selection § Some columns are noisy or redundant. This noise makes it more difficult to discover meaningful patterns from the data; § To discover quality patterns, most data mining algorithms require much larger training data set on high-dimensional data set. § Feature selection, also known as variable selection, feature reduction, attribute selection or variable subset selection, § is the technique of selecting a subset of relevant features for building robust learning models

Attribute Selection § Attribute selection involves searching through all possible combinations of attributes in the data to find which subset of attributes works best for prediction. § To do this, two objects must be set up: Ø The evaluator determines what method is used to assign a worth to each subset of attributes. Ø The search method determines what style of search to be done § The Attribute Selection Mode box has two options: § 1. Use full training set. § 2. Cross-validation.

Attribute Selection § Very flexible: arbitrary combination of search and evaluation methods § Both filtering and wrapping methods § Search methods § best-first § genetic § ranking. . . § Evaluation mmeasures § Relief § information gain § gain ratio. . .

Applying Algorithm

Best Attribute

Algorithm……

Classification § Classification is a data mining function that assigns items in a collection to target categories or classes. § The goal of classification is to accurately predict the target class for each case in the data. § A classification task begins with a data set in which the class assignments are known. § For example, a classification model that predicts credit risk could be developed based on observed data for many loan applicants over a period of time

Classification ~ Naive Bayes classifier § A naive Bayes classifier assumes that the presence (or absence) of a particular feature of a class is unrelated to the presence (or absence) of any other feature, given the class variable. § For example, a fruit may be considered to be an apple if it is red, round, and about 4" in diameter. § Even if these features depend on each other or upon the existence of the other features, a naive Bayes classifier considers all of these properties to independently contribute to the probability that this fruit is an apple.

Naive Bayes Classifier

Confusion Matrix –Pervasive Role

Confusion Matrix - Dataset

Second Fold -Classification

Algorithms

Clustering § Clustering is the task of assigning a set of objects into groups (called clusters) so that the objects in the same cluster are more similar (in some sense or another) to each other than to those in other clusters. § Belong to Unsupervised Learning

Example ~ Weka

Attributes Replacements

Updations

K- Means

Visualizer Open Saved File Save File => Will Store in ARFF

Visualizer – Samples

Association rules § Association rules are if/then statements that help uncover relationships between seemingly unrelated data in a relational database or other information repository. § Finding frequent patterns, associations, correlations, or causal structures among sets of items or objects in transaction databases. § An example of an association rule would be "If a customer buys a dozen eggs, he is 90% likely to also purchase milk. “ § Market Basket Analysis

Association

Description

Rules Framing Rules Set

Visualize

Result Analysis Result 2 Weka Result 1 Concept