Principles of Machine Learning PoChen Wu Media IC
- Slides: 67
Principles of Machine Learning Po-Chen Wu Media IC and System Lab Graduate Institute of Electronics Engineering National Taiwan University
Outline • • • Introduction to Machine Learning Theory of Generalization Learning Algorithm Hazard of Overfitting Blending and Bagging Media IC & System Lab Po-Chen Wu (吳柏辰) 2
Outline • • • Introduction to Machine Learning Theory of Generalization Learning Algorithm Hazard of Overfitting Blending and Bagging Media IC & System Lab Po-Chen Wu (吳柏辰) 3
Mars One Project • A one-way ticket to Mars. • There is a total of 2, 782 applicants. • The application consists of applicant’s General information Motivational letter Résumé Video Media IC & System Lab Po-Chen Wu (吳柏辰) 4
Admission Ticket Approval Applicant Information Age 37 years Gender Male Occupation Professor Annual Salary Year in Job Current Debt NTD 2, 000 11 Years NTD 110, 000 • Unknown target function to be learned: “Should we approve the admission ticket or not? ” Media IC & System Lab Po-Chen Wu (吳柏辰) 5
Formalize the Learning Problem • ML Media IC & System Lab Po-Chen Wu (吳柏辰) 6
Learning Flow for Ticker Approval • (ideal credit approval formula) (historical records) Media IC & System Lab (‘learned’ formula to be used) Po-Chen Wu (吳柏辰) 7
The Learning Model (ideal credit approval formula) (historical records) (‘learned’ formula to be used) (set of candidate formula) Media IC & System Lab Po-Chen Wu (吳柏辰) 8
Practical Definition of Machine Learning (ideal credit approval formula) (historical records) (‘learned’ formula to be used) (set of candidate formula) Media IC & System Lab Po-Chen Wu (吳柏辰) 9
Outline • • • Introduction to Machine Learning Theory of Generalization Learning Algorithm Hazard of Overfitting Blending and Bagging Media IC & System Lab Po-Chen Wu (吳柏辰) 10
Sex Ratio of EE Students Sample • Population Media IC & System Lab Po-Chen Wu (吳柏辰) 11
Hoeffding’s Inequality • Population Sample Hoeffding’s Inequaltiy Media IC & System Lab Po-Chen Wu (吳柏辰) 12
Connection to Learning EE Learning • • Media IC & System Lab Po-Chen Wu (吳柏辰) ? 13
Error Measure • In-sample Error Media IC & System Lab Out-of-sample Error Po-Chen Wu (吳柏辰) 14
Find a Separation Line Classifier Media IC & System Lab Po-Chen Wu (吳柏辰) 15
The Formal Guarantee • Media IC & System Lab Po-Chen Wu (吳柏辰) 16
Find a Separation Line Media IC & System Lab Po-Chen Wu (吳柏辰) 17
• Vapnik-Chervonenkis (VC) bound: Media IC & System Lab Po-Chen Wu (吳柏辰) 18
Find a Separation Linear Seperable! Media IC & System Lab Po-Chen Wu (吳柏辰) 19
Noise & Model Complexity High Complexity Low Complexity Media IC & System Lab Po-Chen Wu (吳柏辰) 20
Statistical Learning Flow (set of candidate formula) Media IC & System Lab Po-Chen Wu (吳柏辰) 21
Outline • • • Introduction to Machine Learning Theory of Generalization Learning Algorithm Hazard of Overfitting Blending and Bagging Media IC & System Lab Po-Chen Wu (吳柏辰) 22
A Simple Hypothesis Set : Perceptron • called ‘perceptron’ hypothesis historically Media IC & System Lab Po-Chen Wu (吳柏辰) 23
Vector Form of Perception Hypothesis • Media IC & System Lab Po-Chen Wu (吳柏辰) 24
• Media IC & System Lab Po-Chen Wu (吳柏辰) 25
Perceptron Learning Algorithm • Media IC & System Lab Po-Chen Wu (吳柏辰) 26
Line with Noise Tolerance • Media IC & System Lab Po-Chen Wu (吳柏辰) 27
Pocket Algorithm • Modify PLA algorithm (black lines) by keeping best weights in pocket. Media IC & System Lab Po-Chen Wu (吳柏辰) 28
• Linear regression find lines/hyperplanes with small residuals Media IC & System Lab Po-Chen Wu (吳柏辰) 29
Error Measure • In-sample Error Media IC & System Lab Out-of-sample Error Po-Chen Wu (吳柏辰) 30
• Media IC & System Lab Po-Chen Wu (吳柏辰) 31
• Media IC & System Lab Po-Chen Wu (吳柏辰) 32
Recap: Matrix Calculus • Denominator-layout notation Media IC & System Lab Po-Chen Wu (吳柏辰) 33
• Media IC & System Lab Po-Chen Wu (吳柏辰) 34
Linear Regression Algorithm • Media IC & System Lab Po-Chen Wu (吳柏辰) 35
Logistic Hypothesis • Media IC & System Lab Po-Chen Wu (吳柏辰) 36
Logistic Function • smooth, monotonic, sigmoid function of s Media IC & System Lab Po-Chen Wu (吳柏辰) 37
Cross-Entropy Error • Media IC & System Lab Po-Chen Wu (吳柏辰) 38
• Media IC & System Lab Po-Chen Wu (吳柏辰) 39
• No closed-form solution! Media IC & System Lab Po-Chen Wu (吳柏辰) 40
Iterative Optimization • Media IC & System Lab Po-Chen Wu (吳柏辰) 41
Gradient Descent • Gradient descent: descent a simple & popular optimization tool Media IC & System Lab Po-Chen Wu (吳柏辰) 42
• too slow too unstable better the fixed learning rate Media IC & System Lab Po-Chen Wu (吳柏辰) 43
Logistic Regression Algorithm • Media IC & System Lab Po-Chen Wu (吳柏辰) 44
Stochastic Gradient Descent (SGD) • Media IC & System Lab Po-Chen Wu (吳柏辰) 45
Three Linear Models • linear classification Media IC & System Lab linear regression Po-Chen Wu (吳柏辰) logistic regression 46
Outline • • • Introduction to Machine Learning Theory of Generalization Learning Algorithm Hazard of Overfitting Blending and Bagging Media IC & System Lab Po-Chen Wu (吳柏辰) 47
Circular Separable 5 0 -5 -5 0 5 • Media IC & System Lab Po-Chen Wu (吳柏辰) 48
Circular Separable and Linear Separable 5 5 0 2. 5 -5 -5 0 0 2. 5 5 • Media IC & System Lab Po-Chen Wu (吳柏辰) 49
• Media IC & System Lab Po-Chen Wu (吳柏辰) 50
Hazard of Overfitting • Vapnik-Chervonenkis (VC) bound (remember? ): Media IC & System Lab Po-Chen Wu (吳柏辰) 51
Regularization: The Magic • Data Target Fit overfitting Media IC & System Lab Data Target Fit step back Po-Chen Wu (吳柏辰) ‘regularized fit’ 52
Stepping Back as Constraint • Media IC & System Lab Po-Chen Wu (吳柏辰) 53
Regression with Looser Constraint • Media IC & System Lab Po-Chen Wu (吳柏辰) 54
Regression with Softer Constraint • Media IC & System Lab Po-Chen Wu (吳柏辰) 55
The Lagrange Multiplier • Media IC & System Lab Po-Chen Wu (吳柏辰) 56
Ridge Regression • Media IC & System Lab Po-Chen Wu (吳柏辰) 57
The Results • Data Target Fit overfitting Media IC & System Lab Data Target Fit ‘regularized fit’ Po-Chen Wu (吳柏辰) Data Target Fit underfitting 58
The VC Message • out-of-sample error model complexity underfitting overfitting in-sample error Media IC & System Lab Po-Chen Wu (吳柏辰) 59
Model Selection Problem • Which one is better? Media IC & System Lab Po-Chen Wu (吳柏辰) 60
• Media IC & System Lab Po-Chen Wu (吳柏辰) 61
pick the best Media IC & System Lab Po-Chen Wu (吳柏辰) 62
V-fold Cross Validation • training validation Media IC & System Lab Po-Chen Wu (吳柏辰) 63
Outline • • • Introduction to Machine Learning Theory of Generalization Learning Algorithm Hazard of Overfitting Blending and Bagging Media IC & System Lab Po-Chen Wu (吳柏辰) 64
Blending (Aggregation) • Media IC & System Lab Po-Chen Wu (吳柏辰) 65
Bagging (Bootstrap Aggregation) • Media IC & System Lab Po-Chen Wu (吳柏辰) 66
Reference • Machine learning slides by Prof. Hsuan-Tien Lin http: //www. csie. ntu. edu. tw/~htlin/course/ml 14 fall/ Media IC & System Lab Po-Chen Wu (吳柏辰) 67
Concept learning task in machine learning
Analytical learning in machine learning
Pac learning model in machine learning
Machine learning t mitchell
Inductive and analytical learning in machine learning
Difference between inductive and analytical learning
Instance based learning in machine learning
Inductive learning machine learning
First order rule learning in machine learning
Eager learner and lazy learner
Cmu machine learning
Cuadro comparativo e-learning y b-learning
Finite state machine vending machine example
Moore and mealy machine
Mealy to moore conversion
Chapter 10 energy work and simple machines answer key
Bsp classification
Expected risk minimization
Sql server machine learning services
Azure machine learning studio logo
Machine learning octave
Jmp neural network
Machine learning tom mitchell
Machine learning infrastructure monitoring
Machine learning actuary
Zillow api python
Tom mitchell machine learning solutions chapter 3
Introduction to machine learning ethem alpaydin
Hypothesis space in machine learning
Kth machine learning
What is unsupervised learning algorithm
Introduction to machine learning andrew ng
Hypothesis space in machine learning
Ilp machine learning
Qradar uba machine learning
Analogizers
Demographic parity
Stacking bagging boosting
Econometrics machine learning
Dimensionality reduction in machine learning
What is tensor in machine learning
Usman roshan njit
Azure synapse vs databricks
Hypothesis space in machine learning
Convex optimization in machine learning javatpoint
Describe generic-specific-ordering of hypotheses.
Upenn machine learning
Aws lambda
High bias low variance introduction to machine learning
Www.youtube.com
Cisco machine learning security
Machine learning conclusion
Advice for applying machine learning
5 tribes of machine learning
Machine learning algorithms for restaurants
Bootstrapping machine learning
Vc dimension rectangle
Multivariate methods in machine learning
Designing machine learning systems
Type of machine learning models
Traditional programming vs machine learning
Feature creation
Unbiased learner in machine learning
Cost funtion
Machine learning ethem
Machine learning lecture
Vc dimension in machine learning
Unbiased learner in machine learning
