stanford_CS231n_learning note_Lec_06 Training Neural Networks, Part2

原创

GuokLiu 2017-03-17 22:20:29 博主文章分类：CS231n ©著作权

文章标签 过拟合 ci lua 文章分类 JavaScript 前端开发

©著作权归作者所有：来自51CTO博客作者GuokLiu的原创作品，请联系作者获取转载授权，否则将追究法律责任

Lec_6_Training Neural Networks, Part II

Happy Moment: 功夫老鼠—看我的锁喉功

stanford_CS231n_learning note_Lec_06 Training Neural Networks, Part2_ci

Parameter Updates

The common method

Update problem-1 TOO SLOW

stanford_CS231n_learning note_Lec_06 Training Neural Networks, Part2_lua_02

Momentum update

Physical interpretation as ball rolling down the loss function + friction (mu coefficient).
mu = usually ~0.5, 0.9, or 0.99 (Sometimes annealed over time, e.g. from 0.5 -> 0.99)

Nesterov Momentum update

stanford_CS231n_learning note_Lec_06 Training Neural Networks, Part2_ci_03

AdaGrad update

stanford_CS231n_learning note_Lec_06 Training Neural Networks, Part2_ci_04

RMSProp update

stanford_CS231n_learning note_Lec_06 Training Neural Networks, Part2_lua_05

stanford_CS231n_learning note_Lec_06 Training Neural Networks, Part2_lua_06

Adam update

stanford_CS231n_learning note_Lec_06 Training Neural Networks, Part2_lua_07

Update problem-2 HYPERPARAMETER NEEDED

stanford_CS231n_learning note_Lec_06 Training Neural Networks, Part2_过拟合_08

Second order optimization methods

stanford_CS231n_learning note_Lec_06 Training Neural Networks, Part2_lua_09

L-BFGS

stanford_CS231n_learning note_Lec_06 Training Neural Networks, Part2_lua_10

Summary for update problems 1+2

IN PRACTICE
- Adam is a good default choice in most cases
- If you can afford to do full batch updates then try out L-BFGS (and don’t forget to disable all sources of noise)