Linear equivalence

$\text{multi-layered FFNN, all neurons use linear} f_{a}s$ $\text{EQUIVALENT TO}$ $\text{single-layer NN, all neurons linear}$

• Proof

Need for non-linearities

• multi-layer network using solely linear neurons

• lacks the expressive power gained from non-linear activations,

• reducing its capabilities to that of a simpler, single-layer network.

• Linear activation functions don't introduce the complexity needed for neural networks to learn and represent intricate patterns

• therefore use deep NNs when your data IS NOT linearly separable/ CANNOT be modeled using a linear model

• from FFNN slides