Loss function is sum squared error: $\left(\frac{1}{2}\left(t-\mathbb{M}_{\mathbf{w}}(\mathbf{d})\right)^2\right)$

measure of the model's fit, aiding in optimization, and facilitating model comparison

Derivative of loss function: $\left(\left(t_i-\mathbb{M}_{\mathbf{w}}\left(\mathbf{d}_i\right)\right) \times d_{j, i}\right)$

• $\mathbb{M}_{\mathbf{w}}\left(\mathbf{d}_i\right)$ is the model output, so makes sense that we need to compute the model output
• $\left(t_i-\mathbb{M}_{\mathbf{w}}\left(\mathbf{d}_i\right)\right.$ is the model error, so also makes sense that we calculate this

Let's assume we have the following weights: $W_0(t=0)=-0.146$ $W_1(t=0)=0.185$ $W_3(t=0)=-0.044$

Clearly our model is: $W_{0} + W_{1}\cdot d_1+W_{2}\cdot d_2$ –> $W(t=0)=-0.146+0.185 \cdot d_1-0.044 \cdot d_2$

1. Calculate model predictions

So what we do is apply $W_{0} + W_{1}\cdot d_1+W_{2}\cdot d_2$ to each instance

$-0.146 + 0.185*501 – 0.044 * 4 = 92.363$ $-0.146 + 0.185*551 – 0.044 * 7 = 101.481$

2. Calculate the error

Here we calculate the error between the correct values and the model prediction $y-\hat{y}$

3. Calculate $\delta$ values for 1st variable

i.e. for $(w_{0}, j=0)$ on each instance $(i=1,2)$ $\delta=\left(\left(t_i-\mathbb{M}_{\mathbf{w}}\left(\mathbf{d}_i\right)\right) \times d_{j, i}\right)$

But we know that $d_{0}=1$ so the equation is simply the error. You can also think of it as an elementwise multiplication of the error and a column of ones $(d_{0})$.

Which results in:

4. Calculate $\delta$ values for 2nd variable

i.e. for $(w_{1}, j=1)$ on each instance $(i=1,2)$

So we perform an element wise multiplication of the error with the feature corresponding to $d_{1}$:

Which results in:

5. Calculate $\delta$ values for 3rd variable

i.e. for $(w_{2}, j=2)$ on each instance $(i=1,2)$

So we perform an element wise multiplication of the error with the feature corresponding to $d_{2}$:

Which results in:

6. Calculate actual $\delta$ values

• Now calculate the actual delta values used to perform the weight updates.
• This is the sum of the $\delta$ values for $\delta(W_{0})$, $\delta(W_{1})$, and $\delta(W_{2})$

7. Optional: compute sum of squared errors

• Compute the sum of squared errors by squaring each of the error values in the error column, and computing the sum.
• This would provide you with a single metric to determine how good the current model is with respect to its current weights.
• Remember that we are trying to minimise the loss function.
• For $t=0$, the loss would typically be bad because we randomly initialised the weights.

8. Weight updates

• Now that we have the $\delta$ values we can perform the weight updates

new weight $=w_j+\eta \underbrace{\sum_{i=1}^n\left(\left(t_i-\mathbb{M}_{\mathbf{w}}\left(\mathbf{d}_i\right)\right) \times \mathbf{d}_{j, i}\right)}_{\delta\left(\mathcal{D}, w_j\right)}$

Assume the learning rate is $\alpha =0.000001$

We already know the current weights for time step 0, and we know all the $\delta$ values.

$W_0(t=0)=-0.146$ $W_1(t=0)=0.185$ $W_3(t=0)=-0.044$

$w_0(t=1) = w_0(t=0) + \alpha \times \delta w_0$ Substituting the values: $-0.146 + 0.000001 \times 506.156 = -0.145493844$

$w_1(t=1) = w_1(t=0) + \alpha \times \delta w_1$ Substituting the values: $0.185 + 0.000001 \times 267510.106 = 0.452510106$

$w_2(t=1) = w_2(t=0) + \alpha \times \delta w_2$ Substituting the values: $-0.044 + 0.000001 \times 2860.181 = -0.041139819$

$W_{0}(t=1)= -0.145493844$ $W_{1}(t=1)= 0.452510106$ $W_{2}(t=1)= -0.041139819$

• Now we could repeat all the steps again.
• If you compute the sum of squared errors you will see that this value would have reduced (I.e. we minimised the loss).
• We are trying to optimise the weights in such a was as to minimise the loss.
• repeat all the steps and compute the weights for $t=2$, and also ensure that the loss function has indeed decreased in time step $t=1$when compared to time step $t=0$.