Derivatives and Gradients Derivatives Derivatives tell us which

  • Slides: 20
Download presentation
Derivatives and Gradients

Derivatives and Gradients

Derivatives • Derivatives tell us which direction to search for a solution 2

Derivatives • Derivatives tell us which direction to search for a solution 2

Derivatives • Slope of Tangent Line 3

Derivatives • Slope of Tangent Line 3

Derivatives 4

Derivatives 4

Derivatives in Multiple Dimensions • Gradient Vector • Hessian Matrix 5

Derivatives in Multiple Dimensions • Gradient Vector • Hessian Matrix 5

Derivatives in Multiple Dimensions • Directional Derivative 6

Derivatives in Multiple Dimensions • Directional Derivative 6

Numerical Differentiation • Finite Difference Methods • Complex Step Method 7

Numerical Differentiation • Finite Difference Methods • Complex Step Method 7

Numerical Differentiation: Finite Difference • Derivation from Taylor series expansion 8

Numerical Differentiation: Finite Difference • Derivation from Taylor series expansion 8

Numerical Differentiation: Finite Difference • Neighboring points are used to approximate the derivative •

Numerical Differentiation: Finite Difference • Neighboring points are used to approximate the derivative • h too small causes numerical cancellation errors 9

Numerical Differentiation: Finite Difference • Error Analysis • Forward Difference: O(h) • Central Difference:

Numerical Differentiation: Finite Difference • Error Analysis • Forward Difference: O(h) • Central Difference: O(h 2) 10

Numerical Differentiation: Complex Step • Taylor series expansion using imaginary step 11

Numerical Differentiation: Complex Step • Taylor series expansion using imaginary step 11

Numerical Differentiation Error Comparison 12

Numerical Differentiation Error Comparison 12

Automatic Differentiation • Evaluate a function and compute partial derivatives simultaneously using the chain

Automatic Differentiation • Evaluate a function and compute partial derivatives simultaneously using the chain rule of differentiation 13

Automatic Differentiation • Forward Accumulation is equivalent to expanding a function using the chain

Automatic Differentiation • Forward Accumulation is equivalent to expanding a function using the chain rule and computing the derivatives inside-out • Requires n-passes to compute n-dimensional gradient • Example 14

Automatic Differentiation • Forward Accumulation 15

Automatic Differentiation • Forward Accumulation 15

Automatic Differentiation • Forward Accumulation 16

Automatic Differentiation • Forward Accumulation 16

Automatic Differentiation • Forward Accumulation 17

Automatic Differentiation • Forward Accumulation 17

Automatic Differentiation • Reverse accumulation is performed in single run using two passes over

Automatic Differentiation • Reverse accumulation is performed in single run using two passes over an n-dimensional function (forward and back) • Note: this is central to the backpropagation algorithm used to train neural networks • Many open-source software implementations are available 18

Summary • Derivatives are useful in optimization because they provide information about how to

Summary • Derivatives are useful in optimization because they provide information about how to change a given point in order to improve the objective function • For multivariate functions, various derivative-based concepts are useful for directing the search for an optimum, including the gradient, the Hessian, and the directional derivative • One approach to numerical differentiation includes finite difference approximations 19

Summary • Complex step method can eliminate the effect of subtractive cancellation error when

Summary • Complex step method can eliminate the effect of subtractive cancellation error when taking small steps • Analytic differentiation methods include forward and reverse accumulation on computational graphs 20

Gradients What are Gradients n n n GradientsGradients What are Gradients n n n Gradients
Gradients What are Gradients n n n GradientsGradients What are Gradients n n n Gradients
Grade DE Interpreting Gradients Identify and interpret gradientsGrade DE Interpreting Gradients Identify and interpret gradients
Latitudinal Gradients in Species Diversity Latitudinal Gradients inLatitudinal Gradients in Species Diversity Latitudinal Gradients in
Gradients Gradients A gradient is a type ofGradients Gradients A gradient is a type of
Linear Graphs Gradients westiesworkshop Linear Graphs Gradients MayLinear Graphs Gradients westiesworkshop Linear Graphs Gradients May
Module Derivatives and Related Accounting Issues Derivatives DerivativesModule Derivatives and Related Accounting Issues Derivatives Derivatives
Which part Which part Which part Which partWhich part Which part Which part Which part
CREDIT DERIVATIVES WHAT ARE CREDIT DERIVATIVES Credit derivativesCREDIT DERIVATIVES WHAT ARE CREDIT DERIVATIVES Credit derivatives
Derivatives part I 1 Derivatives Derivatives are financialDerivatives part I 1 Derivatives Derivatives are financial
3 DERIVATIVES DERIVATIVES 3 4 Derivatives of Trigonometric3 DERIVATIVES DERIVATIVES 3 4 Derivatives of Trigonometric
Derivatives part I 1 Derivatives Derivatives are financialDerivatives part I 1 Derivatives Derivatives are financial
Derivatives part I 1 Derivatives Derivatives are financialDerivatives part I 1 Derivatives Derivatives are financial
WHICH IS BETTER Tell me which introduction isWHICH IS BETTER Tell me which introduction is
DEMOCRATIC CULTURAL POLICY WHICH DEMOCRACY WHICH CULTURE WHICHDEMOCRATIC CULTURAL POLICY WHICH DEMOCRACY WHICH CULTURE WHICH
DEMOCRATIC CULTURAL POLICY WHICH DEMOCRACY WHICH CULTURE WHICHDEMOCRATIC CULTURAL POLICY WHICH DEMOCRACY WHICH CULTURE WHICH
w Which witch wished which wicked wish Whichw Which witch wished which wicked wish Which
w Which witch wished which wicked wish Whichw Which witch wished which wicked wish Which
w Which witch wished which wicked wish Whichw Which witch wished which wicked wish Which