Data Mining Lecture 3 Course Syllabus Course topics

Data Mining Lecture 3

Course Syllabus • Course topics: • Introduction (Week 1 -Week 2) – – What is Data Mining? Data Collection and Data Management Fundamentals The Essentials of Learning The Emerging Needs for Different Data Analysis Perspectives • Data Management and Data Collection Techniques for Data Mining Applications (Week 3 -Week 4) – Data Warehouses: Gathering Raw Data from Relational Databases and transforming into Information. – Information Extraction and Data Processing Techniques – Data Marts: The need for building highly specialized data storages for data mining applications

Week 3 - Remainder-Data to Knowledge Pyramid Increasing potential to support business decisions Making Decisions Data Presentation Visualization Techniques Data Mining Information Discovery End User Business Analyst Data Exploration Statistical Analysis, Querying and Reporting Data Warehouses / Data Marts OLAP, MDA Data Sources Paper, Files, Information Providers, Database Systems, OLTP DBA

Week 2 - Remainder - Data Mining Perspective to Knowledge Discovery Interpretation/ Evaluation Knowledge Data Mining Preprocessing Patterns Selection Preprocessed Data Target Data adapted from: U. Fayyad, et al. (1995), “From Knowledge Discovery to Data Mining: An Overview, ” Advances in Knowledge Discovery and Data Mining, U. Fayyad et al. (Eds. ), AAAI/MIT Press

Week 3 - Remainder - Essentials of Learning ? • can we formalize it? • is it just a chemical activation? • is it memorization? • is it continous node connecting/disconnecting on dynamically changing brain network topology?

Week 3 - Remainder -Essentials of Learning The Artifical Intelligence View: • central to human knowledge and intelligence, essential for building intelligent machines. • years of effort in AI has shown that trying to build intelligent computers by programming all the rules cannot be done; automatic learning is crucial. For example, we humans are not born with the ability to understand language — we learn it — and it makes sense to try to have computers learn language instead of trying to program it all it

Week 3 - Remainder- Essentials of Learning The Software Engineering View: • Machine Learning allows us to program computers by example, which can be easier than writing code the traditional way. The Stats View: • Machine Learning is the marriage of computer science and statistics • computational techniques are applied to statistical problems. Machine Learning has been applied to a vast number of problems in many contexts, beyond the typical statistics problems. Machine Learning is often designed with different considerations than statistics (e. g. , speed is often more important than accuracy).

Week 3 - Essentials of Learning Informal Learning Problem Definition: computer program that improves its performance at some task through experience Formal Learning Problem Definition: computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E

Week 3 - Essentials of Learning A chess learning problem: Task T: playing chess Performance measure P: percent of games won against opponents Training experience E: playing practice games against itself A handwriting recognition learning problem: Task T: recognizing and classifying handwritten words within images Performance measure P: percent of words correctly classified Training experience E: a database of handwritten words with given classifications A robot driving learning problem: Task T: driving on public four-lane highways using vision sensors Performance measure P: average distance traveled before an error (as judged by human overseer) Training experience E: a sequence of images and steering commands recorded while observing a human driver

Week 3 - Essentials of Learning Attributes of Experience learn from direct training examples consisting of states and the correct move for each – supervised learning -CHESS PROBLEM providing individual chess board states and the correct move for each learn from indirect information consisting of the moves and final outcomes of these moves. – unsupervised learning -CHESS PROBLEM providing sequences of moves and final outcomes of various games played causality – credit assignment

Week 3 - Essentials of Learning Attributes of Experience the degree to which the learner controls the sequence of training examples CHESS PROBLEM rely on the teacher to select informative board states and to provide the correct move for each the learner might itself propose board states that it finds particularly confusing and ask the teacher for the correct move learner may have complete control over both the board states and (indirect) training classifications, as it does when it learns by playing against itself with no teacher present

Week 3 - Essentials of Learning Attributes of Experience how well it represents the distribution of examples over which the final system performance P must be measured !!!! most current theory of machine learning rests on the crucial assumption that the distribution of training examples is identical to the distribution of test examples. Despite our need to make this assumption in order to obtain theoretical results, it is important to keep in mind that this assumption must often be violated in practice.

Week 3 - Essentials of Learning Central Limit Theorem The Central Limit Theorem is a theorem stating that the sum of a large number of independent, identically distributed random variables approximately follows a Normal distribution Consider a set of independent, identically distributed random variables Y 1. . . YN, governed by an arbitrary probability Distribution follows the Normal Distrubition even if we dont know the distrubition of individual Yi but we could compute the distribution of A common rule of thumb is that we can use the Normal approximation when n >= 30

Week 3 - Essentials of Learning Operational Definition of Learning Function. Target Function Given training experience and target definition deciding on learning architecture by considering correctness applicability performance CHESS PROBLEM Ideal Target Function: Choose. Move : B -> M to indicate that this function accepts as input any board from the set of legal boardstates B and produces as output some move from the set of legal moves M. What if indirect training experience available to our system ?

Week 3 - Essentials of Learning Operational Definition of Learning Function. Target Function Given training experience and target definition deciding on learning architecture by considering correctness applicability performance CHESS PROBLEM Operational Target Function: V : B ->R to denote that V maps any legal board state from the set B to some real value. assign higher scores to better board states. then use it to select the best move from any current board position. can be accomplished by generating the successor board state produced by every legal move, then using V to choose the best successor state and therefore the best legal move. Is it really operational Not So!! searching all the way down to the end of game. Computationally not operational

Week 3 - Essentials of Learning Operational Definition of Learning Function. Target Function Given training experience and target definition deciding on learning architecture by considering correctness applicability performance CHESS PROBLEM Operational Target Function: V : B ->R to denote that V maps any legal board state from the set B to some real value. assign higher scores to better board states. then use it to select the best move from any current board position. can be accomplished by generating the successor board state produced by every legal move, then using V to choose the best successor state and therefore the best legal move. Is it really operational Not So!! searching all the way down to the end of game. Computationally not operational

Week 3 - Essentials of Learning Operational Definition of Learning Function. Target Function Given training experience and target definition deciding on learning architecture by considering correctness applicability performance CHESS PROBLEM Choosing complex target function brings expressebility but also bring performance battleneck also brings the urgent need on extra more training examples (a lot more) to learn Real issue is choosing the operation target function -> MODELING->function approximation

Week 3 - Essentials of Learning Importance of Target Function Target function simply determines the size of our hypothesis space (solution space) What if needed solution cannot be represented in our hypothesis space lets have perfect hypothetical H hypothesis space that can represen every teachable function so expressebility is not our problem are we OK with that H NO we are now completely unable to generalize beyond the observed examples

Week 3 - Essentials of Learning Importance of Target Function if we need generalization and applicability to unseen instances we must choose biased-target function (generalizable target function) a learner that makes no a priori assumptions regarding the identity of the target function has no rational basis for classifying any unseen instances we simply wish to capture here is the policy by which the learner generalizes beyond the observed training data, to infer the classification of new instances

Week 3 - Essentials of Learning Inductive Bias target concept = target function Formal Definition:

Week 3 - Essentials of Learning Inductive Bias Examples ROTE-LEARNER: Learning corresponds simply to storing each observed training example in memory. Subsequent instances are classified by looking them up in memory. If the instance is found in memory, the stored classification is returned. Otherwise, the system refuses to classify the new instance. Inductive Bias: Bias- Free CANDIDATE-ELIMINATl. ION ALGORITHM: New instances are classified only in the case where all members of the current version space (subset of hypetheses consistent with our training examples) agree on the classification. Otherwise, the system refuses to classify the new instance. Inductive Bias: Target Concept in the Hypothesis Space FIND-S: This algorithm, finds the most specific hypothesis consistent with the training examples. It then uses this hypothesis to classify all subsequent instances. Inductive Bias : Target Concept in the Hypothesis Space and most specific hypothesis represent it

Week 3 - Essentials of Learning Search Bias vs Restriction Bias ID 3 searches a complete hypothesis space (i. e. , one capable of expressing any finite discrete-valued function). It searches incompletely through this space, from simple to complex hypotheses, until its termination condition is met (e. g. , until it finds a hypothesis consistent with the data). Its inductive bias is solely a consequence of the ordering of hypotheses by its search strategy. Its hypothesis space introduces no additional bias. (SEARCH BIAS, PREFERENCE BIAS) CANDIDATE-ELIMINATl. ON algorithm searches an incomplete hypothesis space (i. e. , one that can express only a subset of the potentially teachable concepts, version space). It searches this space completely, finding every hypothesis consistent with the training data. Its inductive bias is solely a consequence of the expressive power of its hypothesis representation. Its search strategy introduces no additional bias. (RESTRICTION BIAS LANGUAGE BIAS)

Week 3 -End • read – Supplemantary Book “Machine Learning”Tom Mitchell Chapter 1 – Chapter 2 – Course Text Book Chapter 2 (preparation for the next week)