LR的原理，损失函数，求解方法

  逻辑回归本质上是线性回归&＃xff0c;只是在特征到结果的映射中加入了一层逻辑函数&＃xff0c;g(z)&＃61;11&＃43;e−z{\rm{g}}(z) &＃61; \frac{1}{{1 &＃43; {e^{ - z}}}}g(z)&＃61;1&＃43;e−z1​&＃xff0c;即&＃xff1a;先把特征线性求和z&＃61;w0&＃43;w1∗x1&＃43;...,&＃43;wn∗xn{\rm{z}} &＃61; {w_0} &＃43; {w_1}*{x_1} &＃43; ..., &＃43; {w_n}*{x_n}z&＃61;w0​&＃43;w1​∗x1​&＃43;...,&＃43;wn​∗xn​,然后使用函数g(z)作为假设函数来预测。
  逻辑回归用来分类0/1问题&＃xff0c;也就是预测结果属于0或者属于1的二值分类问题&＃xff0c;有模型&＃xff1a;
p(y&＃61;1∣x)&＃61;g(wTx)&＃61;11&＃43;e−wTxp(y &＃61; 1|x) &＃61; g({w^T}x) &＃61; \frac{1}{{1 &＃43; {e^{ - {w^T}x}}}} p(y&＃61;1∣x)&＃61;g(wTx)&＃61;1&＃43;e−wTx1​
p(y&＃61;0∣x)&＃61;1−g(wTx)&＃61;e−wTx1&＃43;e−wTxp(y &＃61; 0|x) &＃61; 1 - g({w^T}x) &＃61; \frac{{{e^{ - {w^T}x}}}}{{1 &＃43; {e^{ - {w^T}x}}}} p(y&＃61;0∣x)&＃61;1−g(wTx)&＃61;1&＃43;e−wTxe−wTx​

  对于训练数据集&＃xff0c;特征数据x&＃61;{x1,x2,...,xm}x &＃61; \{ {x_1},{x_2},...,{x_m}\}x&＃61;{x1​,x2​,...,xm​}和对应的分类标签y&＃61;{y1,...,ym}{\rm{y}} &＃61; \{ {y_1},...,{y_m}\}y&＃61;{y1​,...,ym​}。假设m个样本相互独立&＃xff0c;那么它们的联合分布为各边缘分布的乘积&＃xff0c;得到似然函数&＃xff1a;
L(w)&＃61;∏i&＃61;1mg(wTx)yi∗(1−g(wTx))1−yiL(w) &＃61; {\prod\limits_{i &＃61; 1}^m {g({w^T}x)} ^{{y_i}}}*{(1 - g({w^T}x))^{1 - {y_i}}} L(w)&＃61;i&＃61;1∏m​g(wTx)yi​∗(1−g(wTx))1−yi​
取对数&＃xff1a;
e(w)&＃61;ln⁡L(w)&＃61;∑i&＃61;1myi∗ln⁡g(wTx)&＃43;(1−y)ln⁡(1−g(wTx))e(w) &＃61; \ln L(w) &＃61; \sum\limits_{i &＃61; 1}^m {{y_i}*\ln g({w^T}x)} &＃43; (1 - y)\ln (1 - g({w^T}x)) e(w)&＃61;lnL(w)&＃61;i&＃61;1∑m​yi​∗lng(wTx)&＃43;(1−y)ln(1−g(wTx))

  与线性回归类似&＃xff0c;我们使用梯度上升的方法(类似与梯度下降方法)&＃xff0c;那么随机梯度上升更新规则为&＃xff1a;w:&＃61;w&＃43;α∗∇we(w)w: &＃61; w &＃43; \alpha *{\nabla _w}e(w)w:&＃61;w&＃43;α∗∇w​e(w)

∂∂wje(w)&＃61;∂∂wj{∑i&＃61;1m{yilng(wTxi)&＃43;(1−yi)ln(1−g(wTxi))}&＃61;∂∂wj∑i&＃61;1m[yig(wTxi)−1−yi1−g(wTxi)]g(wTxi)′&＃61;∂∂wj∑i&＃61;1m[yi−g(wTxi)](wTxi)′&＃61;∑i&＃61;1m[yi−g(wTxi)]wj\begin{array}{l} \frac{\partial }{{\partial {w_j}}}e(w) &＃61; \frac{\partial }{{\partial {w_j}}}\{ \sum\limits_{i &＃61; 1}^m {\{ {y_i}ln\;g({w^T}{x_i})} &＃43; (1 - {y_i})ln\;(1 - g({w^T}{x_i}))\} \\ \;\;\;\;\;\;\;\;\;\;\;\;\; &＃61; \frac{\partial }{{\partial {w_j}}}\sum\limits_{i &＃61; 1}^m {[\frac{{{y_i}}}{{g({w^T}{x_i})}} - \frac{{1 - {y_i}}}{{1 - g({w^T}{x_i})}}]g({w^T}{x_i})&＃39;} \\ \;\;\;\;\;\;\;\;\;\;\;\;\; &＃61; \frac{\partial }{{\partial {w_j}}}\sum\limits_{i &＃61; 1}^m {[{y_i} - g({w^T}{x_i})]({w^T}{x_i})&＃39;} \\ \;\;\;\;\;\;\;\;\;\;\;\;\; &＃61; \sum\limits_{i &＃61; 1}^m {[{y_i} - g({w^T}{x_i})]{w_j}} \end{array} ∂wj​∂​e(w)&＃61;∂wj​∂​{i&＃61;1∑m​{yi​lng(wTxi​)&＃43;(1−yi​)ln(1−g(wTxi​))}&＃61;∂wj​∂​i&＃61;1∑m​[g(wTxi​)yi​​−1−g(wTxi​)1−yi​​]g(wTxi​)′&＃61;∂wj​∂​i&＃61;1∑m​[yi​−g(wTxi​)](wTxi​)′&＃61;i&＃61;1∑m​[yi​−g(wTxi​)]wj​​