VERSION SPACE Concept Learning by Induction Learning has

VERSION SPACE Concept Learning by Induction • Learning has been classified into several types: deductive, inductive, analytical, etc. • Much of human learning involves acquiring general concepts from specific training examples (this is called inductive learning) 1

VERSION SPACE Concept Learning by Induction • Example: Concept of ball * red, round, small * green, round, small * red, round, medium • Complicated concepts: “situations in which I should study more to pass the exam” 2

VERSION SPACE Concept Learning by Induction • Each concept can be thought of as a Boolean-valued function whose value is true for some inputs and false for all the rest (e. g. a function defined over all the animals, whose value is true for birds and false for all the other animals) • This lecture is about the problem of automatically inferring the general definition of some concept, given examples labeled as members or nonmembers of the concept. This task is called concept learning, or approximating (inferring) a Boolean valued function from examples 3

VERSION SPACE Concept Learning by Induction • Target Concept to be learnt: “Days on which Aldo enjoys his favorite water sport” • Training Examples present are: 4

VERSION SPACE Concept Learning by Induction • The training examples are described by the values of seven “Attributes” • The task is to learn to predict the value of the attribute Enjoy. Sport for an arbitrary day, based on the values of its other attributes 5

VERSION SPACE Concept Learning by Induction: Hypothesis Representation • The possible concepts are called Hypotheses and we need an appropriate representation for the hypotheses • Let the hypothesis be a conjunction of constraints on the attribute-values 6

VERSION SPACE Concept Learning by Induction: Hypothesis Representation • If sky = sunny temp = warm humidity = ? wind = strong water = ? forecast = same then Enjoy Sport = Yes else Enjoy sport = No • Alternatively, this can be written as: {sunny, warm, ? , strong, ? , same} 7

VERSION SPACE Concept Learning by Induction: Hypothesis Representation • For each attribute, the hypothesis will have either ? Any value is acceptable Value Any single value is acceptable No value is acceptable 8

VERSION SPACE Concept Learning by Induction: Hypothesis Representation • If some instance (example/observation) satisfies all the constraints of a hypothesis, then it is classified as positive (belonging to the concept) • The most general hypothesis is {? , ? , ? , ? } It would classify every example as a positive example • The most specific hypothesis is { , , , } It would classify every example as negative 9

VERSION SPACE Concept Learning by Induction: Hypothesis Representation • Alternate hypothesis representation could have been Disjunction of several conjunction of constraints on the attribute-values Example: {sunny, warm, normal, strong, warm, same} {sunny, warm, high, strong, cool, change} 10

VERSION SPACE Concept Learning by Induction: Hypothesis Representation • Another alternate hypothesis representation could have been Conjunction of constraints on the attribute-values where each constraint may be a disjunction of values Example: {sunny, warm, normal high, strong, warm cool, same change} 11

VERSION SPACE Concept Learning by Induction: Hypothesis Representation • Yet another alternate hypothesis representation could have incorporated negations Example: { sunny, warm, (normal high), ? , ? } 12

VERSION SPACE Concept Learning by Induction: Hypothesis Representation By selecting a hypothesis representation, the space of all hypotheses (that the program can ever represent and therefore can ever learn) is implicitly defined In our example, the instance space X can contain 3. 2. 2. 2 = 96 distinct instances There are 5. 4. 4. 4 = 5120 syntactically distinct hypotheses. Since every hypothesis containing even one classifies every instance as negative, hence semantically distinct hypotheses are: 4. 3. 3. 3 + 1 = 973 13

VERSION SPACE Concept Learning by Induction: Hypothesis Representation Most practical learning tasks involve much larger, sometimes infinite, hypothesis spaces 14

VERSION SPACE Concept Learning by Induction: Search in Hypotheses Space Concept learning can be viewed as the task of searching through a large space of hypotheses implicitly defined by the hypothesis representation The goal of this search is to find the hypothesis that best fits the training examples 15

VERSION SPACE Concept Learning by Induction: Basic Assumption Once a hypothesis that best fits the training examples is found, we can use it to predict the class label of new examples The basic assumption while using this hypothesis is: Any hypothesis found to approximate the target function well over a sufficiently large set of training examples will also approximate the target function well over other unobserved examples 16

VERSION SPACE Concept Learning by Induction: General to Specific Ordering If we view learning as a search problem, then it is natural that our study of learning algorithms will examine different strategies for searching the hypothesis space Many algorithms for concept learning organize the search through the hypothesis space by relying on a general to specific ordering of hypotheses 17

VERSION SPACE Concept Learning by Induction: General to Specific Ordering Example: Consider h 1 = {sunny, ? , strong, ? } h 2 = {sunny, ? , ? , ? } any instance classified positive by h 1 will also be classified positive by h 2 (because it imposes fewer constraints on the instance) Hence h 2 is more general than h 1 and h 1 is more specific than h 2 18

VERSION SPACE Concept Learning by Induction: General to Specific Ordering Consider the three hypotheses h 1, h 2 and h 3 19

VERSION SPACE Concept Learning by Induction: General to Specific Ordering • Neither h 1 nor h 3 is more general than the other • h 2 is more general than both h 1 and h 3 • Note that the “more-general-than” relationship is independent of the target concept. It depends only on which instances satisfy the two hypotheses and not on the classification of those instances according to the target concept 20

VERSION SPACE Find-S Algorithm How to find a hypothesis consistent with the observed training examples? - A hypothesis is consistent with the training examples if it correctly classifies these examples One way is to begin with the most specific possible hypothesis, then generalize it each time it fails to cover a positive training example (i. e. classifies it as negative) The algorithm based on this method is called Find-S 21

VERSION SPACE Find-S Algorithm We say that a hypothesis covers a positive training example if it correctly classifies the example as positive A positive training example is an example of the concept to be learnt Similarly a negative training example is not an example of the concept 22

VERSION SPACE Find-S Algorithm 23

VERSION SPACE Find-S Algorithm 24

VERSION SPACE Find-S Algorithm The nodes shown in the diagram are the possible hypotheses allowed by our hypothesis representation scheme Note that our search is guided by the positive examples and we consider only those hypotheses which are consistent with the positive training examples The search moves from hypothesis to hypothesis, searching from the most specific to progressively more general hypotheses 25

VERSION SPACE Find-S Algorithm At each step, the hypothesis is generalized only as far as necessary to cover the new positive example Therefore, at each stage the hypothesis is the most specific hypothesis consistent with the training examples observed up to this point Hence, it is called Find-S 26

VERSION SPACE Find-S Algorithm Note that the algorithm simply ignores every negative example However, since at each step our current hypothesis is maximally specific it will never cover (falsely classify) any negative example. In other words, it will be always consistent with each negative training example However the data must be noise free and our hypothesis representation should be such that the true concept can be described by it 27

VERSION SPACE Find-S Algorithm Problems with Find-S: 1. 2. 3. 4. Has the learner converged to the true target concept? Why prefer the most specific hypothesis? Are the training examples consistent with each other? What if there are several maximally specific consistent hypotheses? 28