Stochastic Gradient Descent Weihang Chen Xingchen Chen Jinxiu

  • Slides: 24
Download presentation
Stochastic Gradient Descent Weihang Chen, Xingchen Chen, Jinxiu Liang, Cheng xu, Zehao Chen and

Stochastic Gradient Descent Weihang Chen, Xingchen Chen, Jinxiu Liang, Cheng xu, Zehao Chen and Donglin He School of Computer &Enginnering, South China University of Technology Guangdong, P. R. China March 21, 2017

Outline • Why Stochastic Gradient Descent (SGD) • Convergence Analysis • Performance • Extensions

Outline • Why Stochastic Gradient Descent (SGD) • Convergence Analysis • Performance • Extensions and Variants

Why SGD •

Why SGD •

Example •

Example •

Example •

Example •

Experiment for Performance • Question

Experiment for Performance • Question

Solution •

Solution •

Experiment for Performance（BGD） Step count: 38 The amount of calculation: 38 * 100

Experiment for Performance（BGD） Step count: 38 The amount of calculation: 38 * 100

Experiment for Performance（SGD） Step count: 998 The amount of calculation: 38 * 100

Experiment for Performance（SGD） Step count: 998 The amount of calculation: 38 * 100

Convergence Analysis •

Convergence Analysis •

Experiment for Convergence(Linear Regression) Classified by SVM

Experiment for Convergence(Linear Regression) Classified by SVM

Solution •

Solution •

Result SGD loss – iteration times

Result SGD loss – iteration times

Result Linear Regression with SGD

Result Linear Regression with SGD

Large Learning Rate

Large Learning Rate

MBGD •

MBGD •

Extensions and Variants • Momentum • Average • Ada. Grad • RMSProp • Adam

Extensions and Variants • Momentum • Average • Ada. Grad • RMSProp • Adam • More…

Momentum •

Momentum •

Average •

Average •

Ada. Grad •

Ada. Grad •

RMSProp •

RMSProp •

Adam •

Adam •

Reference • Cnblogs Murongxixi(2013): Stochastic Gradient Descent • Hongmin Cai(2016): Sub-gradient Method • Abdelkrim

Reference • Cnblogs Murongxixi(2013): Stochastic Gradient Descent • Hongmin Cai(2016): Sub-gradient Method • Abdelkrim Bennar(2007): Almost sure convergence of a stochastic approximation process in a convex set • A. SHAPIRO, Y. WARDI(1996): Convergence Analysis of Gradient Descent Stochastic Algorithms • Int 8(2016): Optimization techniques comparison in Julia: SGD, Momentum, Adagrad, Adadelta, Adam • Wikipedia: Stochastic_gradient_descent • Sebastian Ruder (2016): An overview of gradient descent optimization algorithms • Zhihua Zhou(2016): Machine Learning, Chapter 6 • Ycszen(2016): Summary and comparison of SGD，Adagrad， Adadelta，Adam

Thank you for your time

Thank you for your time

Gradient Methods Steepest Descent gradient descent Conjugate GradientGradient Methods Steepest Descent gradient descent Conjugate Gradient
Gradient Descent Review Gradient Descent In step 3Gradient Descent Review Gradient Descent In step 3
Training and optimization Stochastic gradient descent Gradient onTraining and optimization Stochastic gradient descent Gradient on
gradient Descent py def gradient Descentx y thetagradient Descent py def gradient Descentx y theta
Python implementation of Decision Tree Stochastic Gradient DescentPython implementation of Decision Tree Stochastic Gradient Descent
Tight Analyses for NonSmooth Stochastic Gradient Descent NickTight Analyses for NonSmooth Stochastic Gradient Descent Nick
Stochastic Gradient Descent Training for L 1 regularizaedStochastic Gradient Descent Training for L 1 regularizaed
High Performance Parallel Stochastic Gradient Descent IN SHAREDHigh Performance Parallel Stochastic Gradient Descent IN SHARED
Efficient Logistic Regression with Stochastic Gradient Descent WilliamEfficient Logistic Regression with Stochastic Gradient Descent William
Communication Efficient Stochastic Gradient Descent Milan Vojnovi DepartmentCommunication Efficient Stochastic Gradient Descent Milan Vojnovi Department
Efficient Logistic Regression with Stochastic Gradient Descent WilliamEfficient Logistic Regression with Stochastic Gradient Descent William
Efficient Logistic Regression with Stochastic Gradient Descent TheEfficient Logistic Regression with Stochastic Gradient Descent The
Stochastic Gradient Descent Presenter Khishigsuren Ph D CandidateStochastic Gradient Descent Presenter Khishigsuren Ph D Candidate
Tight Analyses for NonSmooth Stochastic Gradient Descent NickTight Analyses for NonSmooth Stochastic Gradient Descent Nick
Stochastic Gradient Descent Training for L 1 regularizaedStochastic Gradient Descent Training for L 1 regularizaed
Advanced Section 6 Convexity Subgradients Stochastic Gradient DescentAdvanced Section 6 Convexity Subgradients Stochastic Gradient Descent
Topology Management Weihang Jiang Weili lin Vishvesh SahasrabudheTopology Management Weihang Jiang Weili lin Vishvesh Sahasrabudhe
Future RD path on accelerator magnets Xingchen XuFuture RD path on accelerator magnets Xingchen Xu
1 Families Kinship and Descent Families Descent Kinship1 Families Kinship and Descent Families Descent Kinship
Recursive descent parsing Some notes on recursive descentRecursive descent parsing Some notes on recursive descent
Single Seed Descent SSD Single seed descent canSingle Seed Descent SSD Single seed descent can
Descent into Savagery Lord of the Flies DescentDescent into Savagery Lord of the Flies Descent
Kinship and Descent Chapter Outline What are descentKinship and Descent Chapter Outline What are descent