Caffe源码中Solver文件分析

Caffe源码中Solver文件分析：Caffe源码中有一些重要的头文件，这里介绍下include/caffe/solver.hpp文件的内容：

1.include文件：

：此文件的介绍可以参考：http://blog.csdn.net/fengbingchun/article/details/62423060

2.模板类Solver：虚基类

3.模板类WorkerSolver：继承父类Solver，用于多GPU训练时仅计算梯度

4.模板类SGDSolver：继承父类Solver

5.模板类NesterovSolver：继承SGDSolver

6.模板类AdaGradSolver：继承SGDSolver

7.模板类RMSPropSolver：继承SGDSolver

8.模板类AdaDeltaSolver：继承SGDSolver

9.模板类AdamSolver：继承SGDSolver

10.函数GetSolver：new solver对象

Solver通过协调Net的前向推断计算和反向梯度计算(forward inference and backward gradients)，来对参数进行更新，从而达到减少loss的目的。Caffe模型的学习被分为两个部分：由Solver进行优化、更新参数，由Net计算出loss和gradient。

solver.prototxt是一个配置文件用来告知Caffe怎样对网络进行训练。

有了Net就可以进行神经网络的前后向传播计算了，但是还缺少神经网络的训练和预测功能，Solver类进一步封装了训练和预测相关的一些功能。Solver定义了针对Net网络模型的求解方法，记录神经网络的训练过程，保存神经网络模型参数，中断并恢复网络的训练过程。自定义Solver能够实现不同的神经网络求解方式。

Caffe支持的solvers包括：

(1)、Stochastic Gradient Descent(type: “SGD”)即随机梯度下降：利用负梯度和上一次权重的更新值的线性组合来更新权重。学习率(learning rate)是负梯度的权重。动量是上一次更新值的权重。一般将学习速率初始化为0.01，然后在训练(training)中当loss达到稳定时，将学习速率除以一个常数(例如10)，将这个过程重复多次。对于动量一般设置为0.9，动量使weight得更新更为平缓，使学习过程更为稳定、快速。

(2)、AdaDelta(type:“AdaDelta”)：是一种”鲁棒的学习率方法”，同SGD一样是一种基于梯度的优化方法。

(3)、Adaptive Gradient(type: “AdaGrad”)即自适应梯度下降，与随机梯度下降一样是基于梯度的优化方法。

(4)、Adam(type:“Adam”)：也是一种基于梯度的优化方法。它包含一对自适应时刻估计变量，可以看做是AdaGrad的一种泛化形式。

(5)、Nesterov’s Accelerated Gradient(type: “Nesterov”)：Nesterov提出的加速梯度下降(Nesterov’s accelerated gradient)是凸优化的一种最优算法，其收敛速度可以达到O(1/t^2)，而不是O(1/t)。尽管在使用Caffe训练深度神经网络时很难满足O(1/t^2)收敛条件，但实际中NAG对于某些特定结构的深度学习模型仍是一个非常有效的方法。

(6)、RMSprop(type:“RMSProp”)：是一种基于梯度的优化方法(同SGD类似)。

Solver：

(1)、用于优化过程的记录、创建训练网络(用于学习)和测试网络(用于评估)；

(2)、通过forward和backward过程来迭代地优化和更新参数；

(3)、周期性地用测试网络评估模型性能；

(4)、在优化过程中记录模型和solver状态的快照(snapshot)。

每一次迭代过程中：

(1)、调用Net的前向过程计算出输出和loss；

(2)、调用Net的反向过程计算出梯度(loss对每层的权重w和偏置b求导)；

(3)、根据下面所讲的Solver方法，利用梯度更新参数；

(4)、根据学习率(learning rate)，历史数据和求解方法更新solver的状态，使权重从初始化状态逐步更新到最终的学习到的状态。

Solvers的运行模式有CPU/GPU两种模式。

Solver方法：用于最小化损失(loss)值。给定一个数据集D，优化的目标是D中所有数据损失的均值，即平均损失，取得最小值。

注：以上关于Solver内容的介绍主要摘自由CaffeCN社区翻译的《Caffe官方教程中译本》。

文件的详细介绍如下：

#ifndef CAFFE_OPTIMIZATION_SOLVER_HPP_
#define CAFFE_OPTIMIZATION_SOLVER_HPP_

#include 
#include 

#include "caffe/net.hpp"

namespace caffe {

/**
 * @brief An interface for classes that perform optimization on Net%s.
 *
 * Requires implementation of ApplyUpdate to compute a parameter update
 * given the current state of the Net parameters.
 */
template 
class Solver { // Solver模板类，虚基类
 public:
// 显示构造函数, 内部会调用Init函数
  explicit Solver(const SolverParameter& param, const Solver* root_solver = NULL);
  explicit Solver(const string& param_file, const Solver* root_solver = NULL);
// 成员变量赋值，包括param_、iter_、current_step_,并调用InitTrainNet和InitTestNets函数
  void Init(const SolverParameter& param);
// 为成员变量net_赋值
  void InitTrainNet();
// 为成员变量test_nets_赋值
  void InitTestNets();
  // The main entry of the solver function. In default, iter will be zero. Pass
  // in a non-zero iter number to resume training for a pre-trained net.
// 依次调用函数Restore、Step、Snapshot，然后执行net_的前向传播函数ForwardPrefilled，最后调用TestAll函数
  virtual void Solve(const char* resume_file = NULL);
  inline void Solve(const string resume_file) { Solve(resume_file.c_str()); }
// 反复执行net前向传播反向传播计算,期间会调用函数TestAll、ApplyUpdate、Snapshot及类Callback两个成员函数
  void Step(int iters);
  // The Restore method simply dispatches to one of the
  // RestoreSolverStateFrom___ protected methods. You should implement these
  // methods to restore the state from the appropriate snapshot type.
// 加载已有的模型
  void Restore(const char* resume_file);
// 虚析构函数
  virtual ~Solver() {}
 // 获得slover parameter
  inline const SolverParameter& param() const { return param_; }
// 获得train Net
  inline shared_ptr<> > net() { return net_; }
// 获得test Net
  inline const vector<><> > >& test_nets() {
    return test_nets_;
  }
// 获得当前的迭代数
  int iter() { return iter_; }
  // Invoked at specific points during an iteration
// 内部Callback类，仅在多卡GPU模式下使用
  class Callback {
   protected:
    virtual void on_start() = 0;
    virtual void on_gradients_ready() = 0;

    template 
    friend class Solver;
  };
// 获得Callback
  const vector& callbacks() const { return callbacks_; }
// 添加一个Callback
  void add_callback(Callback* value) { callbacks_.push_back(value); }

 protected:
  // Make and apply the update value for the current iteration.
// 更新net的权值和偏置
  virtual void ApplyUpdate() = 0;
  // The Solver::Snapshot function implements the basic snapshotting utility
  // that stores the learned net. You should implement the SnapshotSolverState()
  // function that produces a SolverState protocol buffer that needs to be
  // written to disk together with the learned net.
// 快照，内部会调用SnapshotToBinaryProto或SnapshotToHDF5、SnapshotSolverState函数
  void Snapshot();
// 获取快照文件名
  string SnapshotFilename(const string extension);
// 写proto到.caffemodel
  string SnapshotToBinaryProto();
// 写proto到HDF5文件
  string SnapshotToHDF5();
  // The test routine
// 内部会循环调用Test函数
  void TestAll();
// 执行测试网络，net前向传播
  void Test(const int test_net_id = 0);
// 存储snapshot solver state
  virtual void SnapshotSolverState(const string& model_filename) = 0;
// 读HDF5文件到solver state
  virtual void RestoreSolverStateFromHDF5(const string& state_file) = 0;
// 读二进制文件.solverstate到solver state
  virtual void RestoreSolverStateFromBinaryProto(const string& state_file) = 0;
// dummy function，只有声明没有实现
  void DisplayOutputBlobs(const int net_id);

// Caffe中类的成员变量名都带有后缀"_"，这样就容易区分临时变量和类成员变量
  SolverParameter param_; // solver parameter
  int iter_; // 当前的迭代数
  int current_step_; // 
  shared_ptr<> > net_; // train net
  vector<><> > > test_nets_; // test net
  vector callbacks_; // Callback

  // The root solver that holds root nets (actually containing shared layers)
  // in data parallelism
  const Solver* const root_solver_;

// 禁止使用Solver类的拷贝和赋值操作
  DISABLE_COPY_AND_ASSIGN(Solver);
};

/**
 * @brief Solver that only computes gradients, used as worker
 *        for multi-GPU training.
 */
template 
class WorkerSolver : public Solver { // 模板类WorkerSolver，继承父类Solver
 public:
// 显示构造函数
  explicit WorkerSolver(const SolverParameter& param, const Solver* root_solver = NULL)
      : Solver(param, root_solver) {}

 protected:
  void ApplyUpdate() {}
  void SnapshotSolverState(const string& model_filename) {
    LOG(FATAL) << "Should not be called on worker solver.";
  }
  void RestoreSolverStateFromBinaryProto(const string& state_file) {
    LOG(FATAL) << "Should not be called on worker solver.";
  }
  void RestoreSolverStateFromHDF5(const string& state_file) {
    LOG(FATAL) << "Should not be called on worker solver.";
  }
};

/**
 * @brief Optimizes the parameters of a Net using
 *        stochastic gradient descent (SGD) with momentum.
 */
template 
class SGDSolver : public Solver { // 模板类SGDSolver,继承父类Solver
 public:
// 显示构造函数，调用PreSolve函数
  explicit SGDSolver(const SolverParameter& param) : Solver(param) { PreSolve(); }
  explicit SGDSolver(const string& param_file) : Solver(param_file) { PreSolve(); }
// 获取history数据
  const vector<><> > >& history() { return history_; }

 protected:
// 成员变量history_, update_, temp_初始化
  void PreSolve();
// 获取学习率
  Dtype GetLearningRate();
// 内部会调用ClipGradients、Normalize、Regularize、ComputeUpdateValue，更新net权值和偏置
  virtual void ApplyUpdate();
// 调用caffe_scal函数
  virtual void Normalize(int param_id);
// 调用caffe_axpy函数
  virtual void Regularize(int param_id);
// 计算并更新相应Blob值，调用caffe_cpu_axpby和caffe_copy函数
  virtual void ComputeUpdateValue(int param_id, Dtype rate);
// clip parameter gradients to that L2 norm，如果梯度值过大，就会对梯度做一个修剪，
// 对所有的参数乘以一个缩放因子，使得所有参数的平方和不超过参数中设定的梯度总值
  virtual void ClipGradients();
// 存储snapshot solver state，内部会掉用SnapshotSolverStateToBinaryProto或SnapshotSolverStateToHDF5函数
  virtual void SnapshotSolverState(const string& model_filename);
// 写solver state到二进制文件.solverstate
  virtual void SnapshotSolverStateToBinaryProto(const string& model_filename);
// 写solver state到HDF5
  virtual void SnapshotSolverStateToHDF5(const string& model_filename);
  // 读HDF5文件到solver state
  virtual void RestoreSolverStateFromHDF5(const string& state_file);
  // 读二进制文件.solverstate到solver state
  virtual void RestoreSolverStateFromBinaryProto(const string& state_file);
  // history maintains the historical momentum data.
  // update maintains update related data and is not needed in snapshots.
  // temp maintains other information that might be needed in computation
  //   of gradients/updates and is not needed in snapshots
// Caffe中类的成员变量名都带有后缀"_"，这样就容易区分临时变量和类成员变量
  vector<><> > > history_, update_, temp_;

// 禁止使用SGDSolver类的拷贝和赋值操作
  DISABLE_COPY_AND_ASSIGN(SGDSolver);
};

template 
class NesterovSolver : public SGDSolver { // 模板类NesterovSolver，继承SGDSolver
 public:
// 显示构造函数
  explicit NesterovSolver(const SolverParameter& param) : SGDSolver(param) {}
  explicit NesterovSolver(const string& param_file) : SGDSolver(param_file) {}

 protected:
// 计算并更新相应Blob值,调用caffe_cpu_axpby和caffe_copy函数
  virtual void ComputeUpdateValue(int param_id, Dtype rate);

// 禁止使用NesterovSolver类的拷贝和赋值操作
  DISABLE_COPY_AND_ASSIGN(NesterovSolver);
};

template 
class AdaGradSolver : public SGDSolver { // 模板类AdaGradSolver,继承SGDSolver
 public:
// 显示构造函数，调用constuctor_sanity_check函数
  explicit AdaGradSolver(const SolverParameter& param) : SGDSolver(param) { constructor_sanity_check(); }
  explicit AdaGradSolver(const string& param_file) : SGDSolver(param_file) { constructor_sanity_check(); }

 protected:
// 计算并更新相应Blob值
  virtual void ComputeUpdateValue(int param_id, Dtype rate);
  void constructor_sanity_check() {
    CHECK_EQ(0, this->param_.momentum())
        << "Momentum cannot be used with AdaGrad.";
  }

// 禁止使用AdaGradSolver类的拷贝和赋值操作
  DISABLE_COPY_AND_ASSIGN(AdaGradSolver);
};


template 
class RMSPropSolver : public SGDSolver { // 模板类RMSPropSolver,继承SGDSolver
 public:
// 显示构造函数，调用constructor_sanity_check函数
  explicit RMSPropSolver(const SolverParameter& param) : SGDSolver(param) { constructor_sanity_check(); }
  explicit RMSPropSolver(const string& param_file) : SGDSolver(param_file) { constructor_sanity_check(); }

 protected:
// 计算并更新相应Blob值
  virtual void ComputeUpdateValue(int param_id, Dtype rate);
  void constructor_sanity_check() {
    CHECK_EQ(0, this->param_.momentum())
        << "Momentum cannot be used with RMSProp.";
    CHECK_GE(this->param_.rms_decay(), 0)
        << "rms_decay should lie between 0 and 1.";
    CHECK_LT(this->param_.rms_decay(), 1)
        << "rms_decay should lie between 0 and 1.";
  }

// 禁止使用RMSPropSolver类的拷贝和赋值操作
  DISABLE_COPY_AND_ASSIGN(RMSPropSolver);
};

template 
class AdaDeltaSolver : public SGDSolver { // 模板类AdaDeltaSolver，继承SGDSolver
 public:
// 显示构造函数，调用AdaDeltaPreSolve函数
  explicit AdaDeltaSolver(const SolverParameter& param) : SGDSolver(param) { AdaDeltaPreSolve(); }
  explicit AdaDeltaSolver(const string& param_file) : SGDSolver(param_file) { AdaDeltaPreSolve(); }

 protected:
  void AdaDeltaPreSolve();
// 计算并更新相应Blob值
  virtual void ComputeUpdateValue(int param_id, Dtype rate);

// 禁止使用AdaDeltaSolver类的拷贝和赋值操作
  DISABLE_COPY_AND_ASSIGN(AdaDeltaSolver);
};

/**
 * @brief AdamSolver, an algorithm for first-order gradient-based optimization
 *        of stochastic objective functions, based on adaptive estimates of
 *        lower-order moments. Described in [1].
 *
 * [1] D. P. Kingma and J. L. Ba, "ADAM: A Method for Stochastic Optimization."
 *     arXiv preprint arXiv:1412.6980v8 (2014).
 */
template 
class AdamSolver : public SGDSolver { // 模板类AdamSolver，继承SGDSolver
 public:
// 显示构造函数，调用AdamPreSolve函数
  explicit AdamSolver(const SolverParameter& param) : SGDSolver(param) { AdamPreSolve();}
  explicit AdamSolver(const string& param_file) : SGDSolver(param_file) { AdamPreSolve(); }

 protected:
  void AdamPreSolve();
// 计算并更新相应Blob值
  virtual void ComputeUpdateValue(int param_id, Dtype rate);

// 禁止使用AdamSolver类的拷贝和赋值操作
  DISABLE_COPY_AND_ASSIGN(AdamSolver);
};

// new一个指定的solver方法对象
template 
Solver* GetSolver(const SolverParameter& param) {
  SolverParameter_SolverType type = param.solver_type();

  switch (type) {
  case SolverParameter_SolverType_SGD:
      return new SGDSolver(param);
  case SolverParameter_SolverType_NESTEROV:
      return new NesterovSolver(param);
  case SolverParameter_SolverType_ADAGRAD:
      return new AdaGradSolver(param);
  case SolverParameter_SolverType_RMSPROP:
      return new RMSPropSolver(param);
  case SolverParameter_SolverType_ADADELTA:
      return new AdaDeltaSolver(param);
  case SolverParameter_SolverType_ADAM:
      return new AdamSolver(param);
  default:
      LOG(FATAL) << "Unknown SolverType: " << type;
  }
  return (Solver*) NULL;
}

}  // namespace caffe

#endif  // CAFFE_OPTIMIZATION_SOLVER_HPP_<><><><>

// NOTE
// Update the next available ID when you add a new SolverParameter field.
//
// SolverParameter next available ID: 40 (last added: momentum2)
message SolverParameter { // Solver参数
  //////////////////////////////////////////////////////////////////////////////
  // Specifying the train and test networks
  //
  // Exactly one train net must be specified using one of the following fields:
  //     train_net_param, train_net, net_param, net
  // One or more test nets may be specified using any of the following fields:
  //     test_net_param, test_net, net_param, net
  // If more than one test net field is specified (e.g., both net and
  // test_net are specified), they will be evaluated in the field order given
  // above: (1) test_net_param, (2) test_net, (3) net_param/net.
  // A test_iter must be specified for each test_net.
  // A test_level and/or a test_stage may also be specified for each test_net.
  //////////////////////////////////////////////////////////////////////////////

  // Proto filename for the train net, possibly combined with one or more test nets.
  optional string net = 24; // .prototxt文件名， train or test net
  // Inline train net param, possibly combined with one or more test nets.
  optional NetParameter net_param = 25; // net parameter类

  optional string train_net = 1; // Proto filename for the train net, .prototxt文件名，train net
  repeated string test_net = 2; // Proto filenames for the test nets, .prototxt文件名，test net
  optional NetParameter train_net_param = 21; // Inline train net params, train net parameter类
  repeated NetParameter test_net_param = 22; // Inline test net params, test net parameter类

  // The states for the train/test nets. Must be unspecified or
  // specified once per net.
  //
  // By default, all states will have solver = true;
  // train_state will have phase = TRAIN,
  // and all test_state's will have phase = TEST.
  // Other defaults are set according to the NetState defaults.
  optional NetState train_state = 26; // train net state
  repeated NetState test_state = 27; // test net state

  // The number of iterations for each test net.
  repeated int32 test_iter = 3; // 对于测试网络(用于评估)执行一次需要迭代的次数, test_iter * batch_size = 测试图像总数量

  // The number of iterations between two testing phases.
  optional int32 test_interval = 4 [default = 0];　// 指定执行多少次训练网络执行一次测试网络
  optional bool test_compute_loss = 19 [default = false]; // 执行测试网络时是否计算loss
  // If true, run an initial test pass before the first iteration,
  // ensuring memory availability and printing the starting value of the loss.
  optional bool test_initialization = 32 [default = true]; // 在总的开始前，是否先执行一次测试网络
  optional float base_lr = 5; // The base learning rate,基础学习率
  // the number of iterations between displaying info. If display = 0, no info
  // will be displayed.
  optional int32 display = 6; // 指定迭代多少次显示一次结果信息
  // Display the loss averaged over the last average_loss iterations
  optional int32 average_loss = 33 [default = 1]; // 
  optional int32 max_iter = 7; // the maximum number of iterations
  // accumulate gradients over `iter_size` x `batch_size` instances
  optional int32 iter_size = 36 [default = 1]; // 

  // The learning rate decay policy. The currently implemented learning rate
  // policies are as follows: // 学习率衰减策略
  //    - fixed: always return base_lr.
  //    - step: return base_lr * gamma ^ (floor(iter / step))
  //    - exp: return base_lr * gamma ^ iter
  //    - inv: return base_lr * (1 + gamma * iter) ^ (- power)
  //    - multistep: similar to step but it allows non uniform steps defined by
  //      stepvalue
  //    - poly: the effective learning rate follows a polynomial decay, to be
  //      zero by the max_iter. return base_lr (1 - iter/max_iter) ^ (power)
  //    - sigmoid: the effective learning rate follows a sigmod decay
  //      return base_lr ( 1/(1 + exp(-gamma * (iter - stepsize))))
  //
  // where base_lr, max_iter, gamma, step, stepvalue and power are defined
  // in the solver parameter protocol buffer, and iter is the current iteration.
  optional string lr_policy = 8; // 学习策略,可取的值包括：fixed、step、exp、inv、multistep、poly、sigmoid
  optional float gamma = 9; // The parameter to compute the learning rate.
  optional float power = 10; // The parameter to compute the learning rate.
  optional float momentum = 11; // The momentum value, 动量
  optional float weight_decay = 12; // The weight decay. // 
  // regularization types supported: L1 and L2
  // controlled by weight_decay
  optional string regularization_type = 29 [default = "L2"]; // L1 or L2
  // the stepsize for learning rate policy "step"
  optional int32 stepsize = 13; //
  // the stepsize for learning rate policy "multistep"
  repeated int32 stepvalue = 34; //

  // Set clip_gradients to >= 0 to clip parameter gradients to that L2 norm,
  // whenever their actual L2 norm is larger.
  optional float clip_gradients = 35 [default = -1]; //

  optional int32 snapshot = 14 [default = 0]; // The snapshot interval, 迭代多少次保存下结果(如权值、偏置)
  optional string snapshot_prefix = 15; // The prefix for the snapshot,指定保存文件名的前缀
  // whether to snapshot diff in the results or not. Snapshotting diff will help
  // debugging but the final protocol buffer size will be much larger.
  optional bool snapshot_diff = 16 [default = false]; //
  enum SnapshotFormat {
    HDF5 = 0;
    BINARYPROTO = 1;
  }
  optional SnapshotFormat snapshot_format = 37 [default = BINARYPROTO]; // HDF5 or BINARYPROTO
  // the mode solver will use: 0 for CPU and 1 for GPU. Use GPU in default.
  enum SolverMode {
    CPU = 0;
    GPU = 1;
  }
  optional SolverMode solver_mode = 17 [default = GPU]; // 指定solve mode是CPU还是GPU
  // the device_id will that be used in GPU mode. Use device_id = 0 in default.
  optional int32 device_id = 18 [default = 0]; // GPU mode下使用
  // If non-negative, the seed with which the Solver will initialize the Caffe
  // random number generator -- useful for reproducible results. Otherwise,
  // (and by default) initialize using a seed derived from the system clock.
  optional int64 random_seed = 20 [default = -1]; // 

  // Solver type
  enum SolverType { // solver优化方法
    SGD = 0;
    NESTEROV = 1;
    ADAGRAD = 2;
    RMSPROP = 3;
    ADADELTA = 4;
    ADAM = 5;
  }
  optional SolverType solver_type = 30 [default = SGD]; // 指定solver优化方法
  // numerical stability for RMSProp, AdaGrad and AdaDelta and Adam
  optional float delta = 31 [default = 1e-8]; // 
  // parameters for the Adam solver
  optional float momentum2 = 39 [default = 0.999]; // 

  // RMSProp decay value
  // MeanSquare(t) = rms_decay*MeanSquare(t-1) + (1-rms_decay)*SquareGradient(t)
  optional float rms_decay = 38; // 

  // If true, print information about the state of the net that may help with
  // debugging learning problems.
  optional bool debug_info = 23 [default = false]; // 

  // If false, don't save a snapshot after training finishes.
  optional bool snapshot_after_train = 28 [default = true]; //
}

#include "funset.hpp"
#include 
#include 
#include