Least Square Method Line fitting Hyperplane fitting Function

Least Square Method Line fitting Hyper-plane fitting Function approximation Adavanced Numerical Computation 2008, AM NDHU 1

Line fitting Given paired data, (xi yi ), minimize Adavanced Numerical Computation 2008, AM NDHU 2

Paired data n=100; x=rand(1, n); y=1. 5*x+2+rand(1, n)*0. 1 -0. 05; plot(x, y, '. ') Adavanced Numerical Computation 2008, AM NDHU 3

Fitting criteria Adavanced Numerical Computation 2008, AM NDHU 4

Pseudo Inverse Adavanced Numerical Computation 2008, AM NDHU 5

Form X and b X=[x' ones(n, 1)]; b=y'; Adavanced Numerical Computation 2008, AM NDHU 6

Line fitting >> a=inv(X'*X)*X'*b a= 1. 4816 2. 0113 Adavanced Numerical Computation 2008, AM NDHU 7

Line fitting >> a=pinv(X)*b a= 1. 4816 2. 0113 Adavanced Numerical Computation 2008, AM NDHU 8

Demo_line_fitting demo_line_fitting. m Adavanced Numerical Computation 2008, AM NDHU 9

Stand alone executable file mcc -m demo_line_fitting. exe demo_line_fitting. ctf Adavanced Numerical Computation 2008, AM NDHU 10

Linear system Adavanced Numerical Computation 2008, AM NDHU 11

m=n l If X is invertible Adavanced Numerical Computation 2008, AM NDHU 12

inv >> X=rand(5, 5); b=rand(5, 1); >> a=inv(X)*b a= -2. 2355 9. 2038 -7. 0138 -2. 8158 13. 3273 Adavanced Numerical Computation 2008, AM NDHU 13

Mean Square Error >> mean((X*a-b). ^2) ans = 1. 2843 e-031 Adavanced Numerical Computation 2008, AM NDHU 14

m<n l Unknown number less than constraint or data number l Minimization of the mean square error Adavanced Numerical Computation 2008, AM NDHU 15

Adavanced Numerical Computation 2008, AM NDHU 16

Pseudo Inverse Adavanced Numerical Computation 2008, AM NDHU 17

Mean square errors Adavanced Numerical Computation 2008, AM NDHU 18

Minimization Adavanced Numerical Computation 2008, AM NDHU 19

Derivative Adavanced Numerical Computation 2008, AM NDHU 20

Vector Form Adavanced Numerical Computation 2008, AM NDHU 21

Linear system: normal equations Adavanced Numerical Computation 2008, AM NDHU 22

Adavanced Numerical Computation 2008, AM NDHU 23

Adavanced Numerical Computation 2008, AM NDHU 24

Adavanced Numerical Computation 2008, AM NDHU 25

Adavanced Numerical Computation 2008, AM NDHU 26

Adavanced Numerical Computation 2008, AM NDHU 27

Hyper-plane fitting Adavanced Numerical Computation 2008, AM NDHU 28

Mean square error 1 Adavanced Numerical Computation 2008, AM NDHU 29

Adavanced Numerical Computation 2008, AM NDHU 30

Hyper-plane fitting Step 1. Input paired data, (xi , yi), i=1…n l Step 2. Form matrix X and vector b l Step 3. Set a to pinv(X)*b l Step 4. Set a 2 to l Adavanced Numerical Computation 2008, AM NDHU 31

>> >> n=30; X=rand(n, 2); b=rand(n, 1); X=[X ones(n, 1)]; a=pinv(X)*b; aa=inv(X'*X)*(X'*b); sum(abs(a-aa)) ans = 1. 0547 e-015 Adavanced Numerical Computation 2008, AM NDHU 32

demo_hp_fitting >> demo_hp_fitting a 1: 1 a 2: 2 a 3: 3 a= 0. 9959 2. 0035 3. 0141 Adavanced Numerical Computation 2008, AM NDHU 33

HP Tool Adavanced Numerical Computation 2008, AM NDHU 34

HP Tool MLP_Tool. m MLP_Tool. fig Adavanced Numerical Computation 2008, AM NDHU 35

Mesh fstr=input('input a 2 D function: x 1. ^2+x 2. ^2+cos(x 1) : ', 's'); fx=inline(fstr); range=2*pi; x 1=-range: 0. 1: range; x 2=x 1; for i=1: length(x 1) C(i, : )=fx(x 1(i), x 2); end mesh(x 1, x 2, C); Adavanced Numerical Computation 2008, AM NDHU 36

Post-nonlinear Projection tanh Adavanced Numerical Computation 2008, AM NDHU y 37

Nonlinear function approximation Adavanced Numerical Computation 2008, AM NDHU 38

Nonlinear function approximation Target function & sample Unfaithful approximation by hyper-plane fitting Adavanced Numerical Computation 2008, AM NDHU 39

Linear projection Adavanced Numerical Computation 2008, AM NDHU 40

Two linear projections l Add two linear projections Adavanced Numerical Computation 2008, AM NDHU 41

Adavanced Numerical Computation 2008, AM NDHU 42

Adavanced Numerical Computation 2008, AM NDHU 43

Two post-nonlinear projections Adavanced Numerical Computation 2008, AM NDHU 44

Data driven function approximation Adavanced Numerical Computation 2008, AM NDHU 45

Data driven function approximation Adavanced Numerical Computation 2008, AM NDHU 46

Classification l Discriminate l analysis Linear discriminate analysis l win. dat 178 paired data l (x, y) l x {R 13} : predictor or future l y { 1, 2, 3} : three categories l Adavanced Numerical Computation 2008, AM NDHU 47

Linear assumption Predictor x=[x 1, …, x 13]T l y = a 1*x 1+a 2*x 2+…+a 13*x 13 l Find a to l Adavanced Numerical Computation 2008, AM NDHU 48

Demo_wine_fitting Error Rate : 3. 93% Adavanced Numerical Computation 2008, AM NDHU 49

Linearly non-separable • Classify blue and red dots to two category • Linearly non-separable by hyper-plane fitting Adavanced Numerical Computation 2008, AM NDHU 50

Error rate 22. 48 % Adavanced Numerical Computation 2008, AM NDHU 51