深度学习 约束优化

无约束的优化问题

$minf(x) \ \ \ \ ,f\in C^2\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \\ 1.必要条件，局部极小值点x^*需要:\nabla f(x^*)=0,\nabla^2 f(x^*)\geq 0\\ 2.充分条件，局部极小值点x^*:\nabla f(x^*)=0,\nabla^2 f(x^*)> 0$
$例：f(x)=\frac{1}{2}||Ax-b||^2\ \ ,\ \ rank(A_{m,n})=m\leq n\ \ ,b\in R^m\\ 解：\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \\ 标准的二次函数的形式：f(x)=\frac{1}{2}(Ax-b)^t(Ax-b) \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \\ =\frac{1}{2}(x^tA^t-b^t)(Ax-b)=\frac{1}{2}x^tA^tAx-(A^tb)^t x+\frac12b^tb \\ 形式：\frac{1}{2}x^tPx-(Q)^t x+\frac12B \\ 则：\nabla f(x^*)=A^tAx-A^tb ,\nabla^2 f(x^*)=P=A^tA\\ rank(A_{m,n})=m \ \ \ A^tA> 0(自然满足)\\ \left\{\begin{array}{l}min \ f(x)=\frac{1}{2}||Ax-b||^2\\A^tAx=A^tb\end{array}\right. x^*=(A^tA)^{-1}A^tb$

$例： min \ f(x)=\frac{1}{2}||Ax-b||^2+\frac{\lambda}{2}||x||^2 \ \ \ \lambda >0 \\ 将以上的x看成Ix-0*I,\left\{\begin{array}{l}\nabla f(x^*)=A^tAx-A^tb+\lambda x \\\nabla^2 f(x^*)=A^tA+\lambda I\end{array}\right.,\\ 其中\nabla^2 f(x^*)=A^tA+\lambda I，x^t(\nabla^2 f(x^*))x=||Ax||^2+\lambda||x||^2\geq0,因为x\neq 0,所以其严格大于0\\$
$\nabla f(x^*)=A^tAx-A^tb+\lambda x =0 \Rightarrow (A^tA+\lambda) x =A^tb\Rightarrow x=(A^tA+\lambda) ^{-1}A^tb$

有约束的优化问题

线性约束

$\left\{\begin{array}{l}minf(x)\\线性约束Ax=b\end{array}\right. \ \ \ \ ,f\in C^2\ \ \ rank(A_{m,n})=m \ \ \ \\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \\ 对于线性方程组Ax=b可求出解x=x^*+y\{y：Ay=0,矩阵A的零空间,dim(N(A))=n-m\}\\ x=x^*+y=x^*+B_{n,n-p}y,y\in R^{n-p}\\ \left\{\begin{array}{l}minf(x)\\线性约束Ax=b\end{array}\right. （有约束）\Rightarrow min\ \ f(x^*+By)\ ,y\in R^{n-p}（无约束）\\ 令e=By,f(x^*+e)=f(x^*)+\nabla f(x^*)^te+\frac12\nabla ^2 f(x^*)e+o(||e||^2),带入By\\ 并且记f(x^*+By)=g(y),(f-x^*极小，g-0极小)\\ 则\nabla g(0)=B^t\nabla f(x^*),\nabla ^2g(0)=B^t\nabla ^2f(x^*)B \\$
$实际上B^t\nabla f(x^*)是等价于\href {}{拉格朗日乘子法}的$
$\left\{\begin{array}{l}minf(x)\\线性约束Ax=b\end{array}\right. ,x^*局部极小\Rightarrow\partial _x L(x^*,\mu)=0\\ 其中L(x^*,\mu )=f(x)+\mu ^t(Ax-b)$
01h19min
$例：\left\{\begin{array}{l}min \ \ \sum_1^n x_ilogx_i\\\sum x_i=1\end{array}\right.\\ L(x,\mu)=\sum_1^n x_ilogx_i +\mu(\sum x_i-1)\\ \partial _xL=0 \Rightarrow \partial _{x_i} L=0 \Rightarrow 1+log{x_i}+\mu=0 \\ x_i=e^{-1-\mu}（x_i=\frac1n）\\ 然后需要验证：B^t\nabla^2f(x)B>0,\nabla^2f(x)=\begin{bmatrix}\frac{1}{x_1}&&\\&\frac{1}{x_i}&\\&&\frac{1}{x_n}\end{bmatrix},B满秩\rightarrow B^t\nabla^2f(x)B>0$
$例：\left\{\begin{array}{l}min ||y-x||^2\\a^tx=b\end{array}\right.\\ 常值a,y\in R^n,b\in R\\ 约束a^tx=\sum_1^na_ix_i=b \ \ 是n维空间中的一个n-1维超平面，即最小值为y到超平面的投影长度 \\ L(x,\mu)=||y-x||^2+\mu(a^tx-b)\\ \partial _xL=0 \Rightarrow \partial _{x} L=2(x-y)+\mu a=0 \\ \left\{\begin{array}{l}2(x-y)+\mu a=0\\a^tx=b\end{array}\right.\Rightarrow 2a^t(x-y)=\mu a^ta,2b-2a^ty=\mu ||a||^2\\ \mu=\frac{2b-2a^ty}{||a||^2}（常量，可解出x）$
$例：\left\{\begin{array}{l}min ||y-x||^2\\Ax=b\end{array}\right.\\ L(x,\mu)=||y-x||^2+\mu^t(Ax-b)\\$

非线性约束

$例：\left\{\begin{array}{l}min ||y-x||^2\\h_i(x)=0,i\in [1,p],x\in R^n\end{array}\right.\\ 解空间局部参数化\\ g:(-1,1)\rightarrow R^n,g(0)=x^*,R(g)=D（f(x)的定义域）\\ h_i(g)=0\\ x^*局部极小 \Rightarrow 0为f*g局部极小\\ \Rightarrow (f*g)$