caffe中的solver负责处理模型优化过程&＃xff0c;通过网络前向传播产生的loss和后向传播产生的梯度进行模型优化&＃xff08;更新模型中的权重&＃xff09;来降低loss.
caffe中的solver有&＃xff1a;

• Stochastic Gradient Descent (type: “SGD”)
• AdaDelta (type:”AdaDelta”)
• Adaptive Gradient (type: “AdaGrad”)
• Adam (type: “Adam”)
• Nesterov’s Accelerated Gradient (type: “Nesterov”)
• RMSprop (type: “RMSProp”)

solver是关于损失函数最小的优化问题的求解.
其中前向传播计算给定数据集D的损失表达式为

其中&＃xff0c;fW是给定数据集D上的损失函数，r(W)为正则化项&＃xff0c;λ是正则化项的权重&＃xff0c;称为weight decay,用于防止过拟合.由于|D|往往很大&＃xff0c;所以将其划分为多个mini-batch N,|N|<<|D|&＃xff0c;每次迭代计算下面的损失&＃xff1a;

后向传播更新网络权重

其中权重更新量

ΔW的计算涉及到

∇fW,

∇r(W)梯度信息以及其他针对不同优化策略的计算量。

不同的优化策略只是权重更新方法不同。

SGD

随机梯度下降&＃xff08;Stochastic gradient descent&＃xff09;

其中&＃xff0c;

μ是momentum&＃xff0c;是更新量

Vt&＃43;1的权重,

α是学习率&＃xff0c;是负梯度的的权重。

学习率的更新策略有&＃xff1a;

// The learning rate decay policy. The currently implemented learning rate
// policies are as follows:
// - fixed: always return base_lr.
// - step: return base_lr * gamma ^ (floor(iter / step))
// - exp: return base_lr * gamma ^ iter
// - inv: return base_lr * (1 &＃43; gamma * iter) ^ (- power)
// - multistep: similar to step but it allows non uniform steps defined by
// stepvalue
// - poly: the effective learning rate follows a polynomial decay, to be
// zero by the max_iter. return base_lr (1 - iter/max_iter) ^ (power)
// - sigmoid: the effective learning rate follows a sigmod decay
// return base_lr ( 1/(1 &＃43; exp(-gamma * (iter - stepsize))))
//
// where base_lr, max_iter, gamma, step, stepvalue and power are defined
// in the solver parameter protocol buffer, and iter is the current iteration.


参见&＃xff1a;/caffe-master/src/caffe/proto/caffe.proto&＃xff0c;在线的点击这里

AdaDelta