西瓜书笔记：神经网络（第 5 章）

神经网络，反向传播等。

神经元模型

$$\operatorname{logistic}(x) = \operatorname{sigmoid}(x) = \frac{1}{1+\exp(-x)}$$

单层感知机（Perception）模型，只能处理线性可分问题

$$y = f\left( \sum_{i=1}^{n} w_i x_i - \theta \right)$$

多层网络

多层前馈神经网络（Multi-Layer Feedforward Neural Networks）

反向传播算法（Back Propagation）

损失函数（Loss Function）

• 均方误差 $$\mathcal{E}(y_i, \hat{y}_i) = \frac{1}{2} \cdot (y_i - \hat{y}_i)^2$$
• 交叉熵（Cross Entropy） $$\begin{aligned} \mathcal{L}(y_i, \hat{y}_i) =& \begin{cases} \log(P(y_i=1 \mid \hat{y}_i)) = \log(\hat{y}_i), & {\mathrm{if}} \; y_i=1 \\ \log(P(y_i=0 \mid \hat{y}_i)) = 1 - \log(\hat{y}_i), & {\mathrm{if}} \; y_i=0 \end{cases} \\ =& y_i \cdot \log(\hat{y}_i) + (1 - y_i) \cdot \log(1 - \hat{y}_i) \end{aligned}$$

反向传播

对特定的一层神经元，其输出为

$${\pmb y} = f({\pmb W}{\pmb x} + {\pmb b})$$

其中，${\pmb x}$ 为上一层的输出，${\pmb W}$ 和 ${\pmb b}$ 为上一层与当前层之间的权重矩阵和偏置向量，$f$ 为激活函数。已知 $\frac{\partial \mathcal{L}}{\partial {\pmb y}}$，那么

$$\begin{aligned} \frac{\partial \mathcal{L}}{\partial {\pmb W}} =& \frac{\partial \mathcal{L}}{\partial {\pmb y}} \cdot \frac{\partial {\pmb y}}{\partial ({\pmb W}{\pmb x}+{\pmb b})} \cdot \frac{\partial ({\pmb W}{\pmb x}+{\pmb b})}{\partial {\pmb W}} \\ \frac{\partial \mathcal{L}}{\partial {\pmb b}} =& \frac{\partial \mathcal{L}}{\partial {\pmb y}} \cdot \frac{\partial {\pmb y}}{\partial ({\pmb W}{\pmb x}+{\pmb b})} \cdot \frac{\partial ({\pmb W}{\pmb x}+{\pmb b})}{\partial {\pmb b}} \\ \frac{\partial \mathcal{L}}{\partial {\pmb x}} =& \frac{\partial \mathcal{L}}{\partial {\pmb y}} \cdot \frac{\partial {\pmb y}}{\partial ({\pmb W}{\pmb x}+{\pmb b})} \cdot \frac{\partial ({\pmb W}{\pmb x}+{\pmb b})}{\partial {\pmb x}} \end{aligned}$$

其中，$\frac{\partial \mathcal{L}}{\partial {\pmb W}}$ 和 $\frac{\partial \mathcal{L}}{\partial {\pmb b}}$ 可以用于更新 ${\pmb W}$ 和 ${\pmb b}$。以 $\eta$ 为学习速率，则

$$\begin{aligned} \Delta {\pmb W} =& -\eta \cdot \frac{\partial \mathcal{L}}{\partial {\pmb W}} \\ \Delta {\pmb b} =& -\eta \cdot \frac{\partial \mathcal{L}}{\partial {\pmb b}} \end{aligned}$$

$\frac{\partial \mathcal{L}}{\partial {\pmb x}}$ 则用于进一步反向传播。

全局最小与局部最小

防止过拟合的策略

• Early Stopping
• 正则化

防止局部最小的策略

• 多组不同参数初始化网络
• 模拟退火（Simulated Annealing）：以一定概率接受比当前解更差的结果
• 随即梯度下降（SGD）

深度学习

增加隐藏层的层数比增加隐藏层的神经元数目更有效。

防止梯度发散/消失策略

• Pre-Training + Fine-Tuning
• 深度信念网络（Deep Belief Network, DBN），每层都是受限玻尔兹曼机（Restricted Boltzmann Machine, RBM）

节省训练开销

• 权值共享，如 CNN