rnn反向传播详细推导 bn反向传播推导

Batch Normalization 和 Batch Renormalization 前向和反向公式详细推导

文章目录

• Batch Normalization 和 Batch Renormalization 前向和反向公式详细推导

• 一、BN前向传播
• 二、BN的反向传播
• 三、Batch Renormalization

• 1、前向
• 2、反向

• 三、卷积网络中的BN

一、BN前向传播

根据论文‘’Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift" 的推导过程，主要有下面四个公式：

$\mu_B=\frac{1}{m}\sum_i^mx_i\tag{1}$

$\delta_B^2=\frac{1}{m}\sum_i^m(x_i-\mu_B)^2\tag{2}$

$\widehat{x_i}=\frac{x_i-\mu_B}{\sqrt{\delta_B^2+\epsilon}}\tag{3}$

$y_i=\gamma\widehat{x_i}+\beta\tag{4}$

以MLP为例，假设输入的mini-batch样本数为$m$，则此处的$x_i,i=1,2,...m$是第$i$个样本对应的某一层激活值中的一个激活值。也就是说，假设输入$m$个样本作为一次训练，其中第$i$个样本输入网络后，在$l$层得到了$N$个激活单元，则$x_i$代表其中任意一个激活单元。事实上应该写为$x_i^l(n)$更为直观。

所以BN实际上就是对第$l$层的第$n$个激活单元$x_i^l(n)$求其在一个batch中的平均值和方差，并对其进行标准归一化，得到$\widehat{x_i^l(n)}$，可知归一化后的m个激活单元均值为0方差为1，一定程度上消除了Internal Covariate Shift，减少了网络的各层激活值在训练样本上的边缘分布的变化。

二、BN的反向传播

• 设前一层的梯度为$\frac{\partial{L}}{\partial{y_i}}$.
• 需要计算$\frac{\partial{L}}{\partial{x_i}},\frac{\partial{L}}{\partial{\gamma}}以及\frac{\partial{L}}{\partial{\beta}}$

由链式法则以及公式\eqref{4}：

$\frac{\partial{L}}{\partial{\gamma}}=\frac{\partial{L}}{\partial{y_i}}\frac{\partial{y_i}}{\partial{\gamma}}=\frac{\partial{L}}{\partial{y_i}}\widehat{x_i} \tag{5}$

由于对于所有$i=1,2...m. \frac{\partial{L}}{\partial{y_i}}\widehat{x_i}对\frac{\partial{L}}{\partial{\gamma}}$均有贡献，因此一个batch的训练中将$\frac{\partial{L}}{\partial{\gamma}}$定义为：

$\frac{\partial{L}}{\partial{\gamma}}=\sum_{i=1}^m \frac{\partial{L}}{\partial{y_i}}\widehat{x_i}\tag{6}$

同样有：

$\frac{\partial{L}}{\partial{\beta}}=\sum_{i=1}^m \frac{\partial{L}}{\partial{y_i}}\tag{7}$

另外，求$\frac{\partial{L}}{\partial{x_i}}$过程则较为复杂。根据链式法则，以及公式KaTeX parse error: Undefined control sequence: \eqref at position 1: \̲e̲q̲r̲e̲f̲{3},将$\widehat{x_i}$视为$g(x_i,\delta_B^2,\mu_B)$有：

$\frac{\partial{L}}{\partial{x_i}}=\frac{\partial{L}}{\partial{y_i}}\frac{\partial{y_i}}{\partial{\widehat{x_i}}}(\frac{\partial{\widehat{x_i}}}{\partial{x_i}}+\frac{\partial{\widehat{x_i}}}{\partial{\delta_B^2}}\frac{\partial{\delta_B^2}}{\partial{x_i}}+\frac{\partial{\widehat{x_i}}}{\partial{\mu_B}}\frac{\partial{\mu_B}}{\partial{x_i}}) =\frac{\partial{L}}{\partial{y_i}}\frac{\partial{y_i}}{\partial{\widehat{x_i}}}(g_1$

而因为公式KaTeX parse error: Undefined control sequence: \eqref at position 1: \̲e̲q̲r̲e̲f̲{2}可知上式括号中的第二项求偏导可以进一步拆分。（将$\delta_B^2$视为$f(x_i,\mu_B)$）

$\frac{\partial{\delta_B^2}}{\partial{x_i}}= \frac{\partial{\delta_B^2}}{\partial{x_i}}+ \frac{\partial{\delta_B^2}}{\partial{\mu_B}} \frac{\partial{\mu_B}}{\partial{x_i}}= f_1$

注意公式KaTeX parse error: Undefined control sequence: \eqref at position 1: \̲e̲q̲r̲e̲f̲{9}中的两个$\frac{\partial{\delta_B^2}}{\partial{x_i}}$代表不同的含义。由公式KaTeX parse error: Undefined control sequence: \eqref at position 1: \̲e̲q̲r̲e̲f̲{8},\eqref{9}可知，只要求出$f_1$.即可求出$\frac{\partial{L}}{\partial{x_i}}$.

原论文中将公式KaTeX parse error: Undefined control sequence: \eqref at position 1: \̲e̲q̲r̲e̲f̲{8}拆分成如下几项:

$\frac{\partial{L}}{\partial{x_i}}= \frac{\partial{L}}{\partial{\widehat{x_i}}} \frac{\partial{\widehat{x_i}}}{\partial{x_i}}+ \frac{\partial{L}}{\partial{\delta_B^2}} \frac{\partial{\delta_B^2}}{\partial{x_i}}+ \frac{\partial{L}}{\partial{\mu_B}} \frac{\partial{\mu_B}}{\partial{x_i}} \tag{10}$

其中：
$\frac{\partial{L}}{\partial{\widehat{x_i}}}= \frac{\partial{L}}{\partial{y_i}} \frac{\partial{y_i}}{\partial{\widehat{x_i}}}= \frac{\partial{L}}{\partial{y_i}} \gamma\tag{10.1}$

$\frac{\partial{\widehat{x_i}}}{\partial{x_i}}=g$

$\frac{\partial{L}}{\partial{\delta_B^2}}= \frac{\partial{L}}{\partial{\widehat{x_i}}}g$
$\sum_{i=1}^m\frac{\partial{L}}{\partial{\widehat{x_i}}} \frac{\mu_B-x_i}{2}(\delta_B^2+\epsilon)^{-\frac{3}{2}} \tag{10.3}$

$\frac{\partial{\delta_B^2}}{\partial{x_i}}=f$

$\frac{\partial{L}}{\partial{\mu_B}}= \frac{\partial{L}}{\partial{\widehat{x_i}}}g$
$\sum_{i=1}^m( \frac{\partial{L}}{\partial{\widehat{x_i}}}\frac{-1}{\sqrt{\delta_B^2+\epsilon}} +\frac{\partial{L}}{\partial{\delta_B^2}}\frac{2(\mu_B-x_i)}{m}) \tag{10.5}$

$\frac{\partial{\mu_B}}{\partial{x_i}}=\frac{1}{m} \tag{10.6}$

最终BN的反向过程由公式KaTeX parse error: Undefined control sequence: \eqref at position 1: \̲e̲q̲r̲e̲f̲{6},\eqref{7},\…给出。

三、Batch Renormalization

参照论文—— Batch Renormalization: Towards Reducing Minibatch Dependence
in Batch-Normalized Models

Batch Renormalization是对传统BN的优化，该方法保证了train和inference阶段的等效性，解决了非独立同分布和小minibatch的问题。

1、前向

跟原来的公式类似，添加了两个非训练参数$r,d$:
$\mu_B=\frac{1}{m}\sum_i^mx_i\tag{1.1}$

$\sigma_B=\sqrt{\epsilon+\frac{1}{m}\sum_i^m(x_i-\mu_B)^2}\tag{2.1}$

$\widehat{x_i}=\frac{x_i-\mu_B}{\sigma_B}r+d\tag{3.1}$

$y_i=\gamma\widehat{x_i}+\beta\tag{4.1}$

$r=Stop\_Gradient(Clip_{[1/r_{max} ,r_{max}]}(\frac{\sigma_B}{\sigma}))\tag{5.1}$

$d=Stop\_Gradient(Clip_{[-d_{max} ,d_{max}]}(\frac{\mu_B-\mu}{\sigma}))\tag{6.1}$

---

Update moving averages:

$\mu:=\mu+\alpha(\mu_B-\mu)\tag{7.1}$

$\sigma:=\sigma+\alpha(\sigma_B-\sigma)\tag{8.1}$

Inference:

$y=\gamma\frac{x-\mu}{\sigma}+\beta\tag{9.1}$

相比于之前的BN只在训练时计算滑动均值与方差，推断时才使用他们；BRN在训练和推断时都用到了滑动均值与方差。

2、反向

反向的推导与BN类似，
$\frac{\partial{L}}{\partial{\widehat{x_i}}}= \frac{\partial{L}}{\partial{y_i}} \frac{\partial{y_i}}{\partial{\widehat{x_i}}}= \frac{\partial{L}}{\partial{y_i}} \gamma\tag{10.11}$

$\frac{\partial{L}}{\partial{\sigma_B}} \longrightarrow\sum_{i=1}^m \frac{\partial{L}}{\partial{\widehat{x_i}}} \frac{-r(x_i-\mu_B)}{\sigma_B^2} \tag{10.22}$

$\frac{\partial{L}}{\partial{\mu_B}}\longrightarrow\sum_{i=1}^{m}\frac{\partial{L}}{\partial{\widehat{x_i}}}\frac{-r}{\sigma_B} \tag{10.33}$

$\frac{\partial{L}}{\partial{x_i}}= \frac{\partial{L}}{\partial{\widehat{x_i}}} \frac{r}{\sigma_B}+ \frac{\partial{L}}{\partial{\sigma_B}} \frac{x_i-\mu_B}{m\sigma_B}+ \frac{\partial{L}}{\partial{\mu_B}} \frac{1}{m} \tag{10.44}$

$\frac{\partial{L}}{\partial{\gamma}}=\sum_{i=1}^m \frac{\partial{L}}{\partial{y_i}}\widehat{x_i}\tag{10.55}$

$\frac{\partial{L}}{\partial{\beta}}=\sum_{i=1}^m \frac{\partial{L}}{\partial{y_i}}\tag{10.66}$

三、卷积网络中的BN

上面的推导过程都是基于MLP的。对于卷积网络而言，BN过程中的m个激活单元被推广为m幅特征图像。 假设某一层卷积后的feature map是$[N,H,W,C]$的张量，其中N表示batch数目，H,W分别表示长和宽，C表示特征通道数。则对卷积网络的BN操作时，令$m = N\times H\times W$,也就是说将第$i$个batch内某一通道$c$上的任意一个特征图像素点视为$x_i$,套用上面的BN公式即可。所以对于卷积网络来说，中间激活层每个通道都对应一组BN参数$\gamma,\beta$.