f(x) = wx + b

$w,b$ : parameters(参数/coefficients(系数)/weights(权重)

Cost Function

最常用于线性回归的成本函数Squared error cost function(平方误差成本函数):

$J(w,b)=\frac{1}{2m} \sum_1^m (\hat{y}^{(i)}-y^{(i)})^2$

其中,

$\hat{y}-y^{(i)}$ 称为error(误差);

$m$ 为训练集规模;

分母多除2为了使后续计算更简洁

将 $\hat{y}^{(i)}$ 替换为 $f_{w,b}(x^{(i)}$ 等价于:

$J(w,b)=\frac{1}{2m} \sum_1^m (f_{w,b}(x^{(i)})-y^{(i)})^2$

Gradient descent 梯度下降

An algorithm that can use to try to minimize any function

Have some function: $J(w,b)$

Want: $\min\limits_{w,b}J(w,b)$

Outline:

Start with some $ w,b $ (set w = 0 b = 0)
Keep changing $w,b$ to reduce $J(w,b)$
Until we settle at or near a minimum (may have >1 minimums) ==>converge (收敛到一个极小值)

Simultaneous Update 所有参数同时同时更新

Gradient descent algorithm

$w = w - \alpha \frac{\partial}{\partial w}J(w,b) \tag{1}$

$b = b - \alpha \frac{\partial}{\partial b}J(w,b) \tag{2}$

其中,

$\alpha$ : learning rate 学习率, 通常介于0-1, 控制更新参数时的步长

if too small : work but slow

if too large: may fail to converge(收敛) and may even diverge(发散)

因此,可以visualize Loss Function关于参数每次更新时变化的图像

Linear Regression Gradient Descent Algorithm

$\hat{y}^{(i)} = f(x^{(i)}) = wx^{(i)} + b \\ J(w,b)=\frac{1}{2m} \sum_1^m (\hat{y}^{(i)}-y^{(i)})^2$

于是,

$\frac{\partial}{\partial w} J(w,b) \\ =\frac{\partial}{\partial w} \frac{1}{2m} \sum_1^m (f_{w,b}(x^{(i)})-y^{(i)})^2 \\ = \frac{\partial}{\partial w} \frac{1}{2m} \sum_1^m (wx^{(i)}+b-y^{(i)})^2 \\ = \frac{1}{2m}\sum_1^m(wx^{(i)}+b-y^{(i)})2x^{(i)} \\ = \frac{1}{m}\sum_1^m(f_{w,b}(x^{(i)})-y^{(i)})x^{(i)}$

代入可得,

$w - \alpha \frac{\partial}{\partial w} J(w,b) => w - \alpha \frac{1}{m} \sum_1^m (f_{w,b}(x^{(i)})-y^{(i)})x^{(i)}$

同理可得,

$b - \alpha \frac{\partial}{\partial b} J(w,b) => b - \alpha \frac{1}{m} \sum_1^m (f_{w,b}(x^{(i)})-y^{(i)})$

Batch Gradient Descent

Batch: Each step of gradient descent uses all the training exmaples