1. Linear function

$f_{A N}(\text {net}-\theta)=\lambda(\text {net}-\theta)$

• if you tweak the input, the output changes proportionally, following a straight line with a constant slope (λ) for $θ = 0$: see also: week 11 explanation

2. Step function

$f_{A N}(\text {net-}\theta)= \begin{cases}\gamma_1 & \text { if net}\geq \theta \\ \gamma_2 & \text { if net}<\theta\end{cases}$

• Binary output: $γ_{1} = 1$ and $γ_{2} = 0$
• Bipolar output: $γ_{1} = 1$ and $γ_{2} = -1$ for $\theta>0$:

3. Ramp function

$f_{A N}(\text {net}-\theta)= \begin{cases}\gamma & \text {net}-\theta \geq \epsilon \\ \text {net-}\theta & \text { if }-\epsilon<\text {net}-\theta<\epsilon \\ -\gamma & \text { if } \text {net}-\theta \leq-\epsilon\end{cases}$

• combination of the linear and step functions for $\theta>0$:

4. Sigmoid function

$f_{A N}(net-\theta)=\frac{1}{1+e^{-\lambda(n e t-\theta)}}$

• continuous version of the ramp function
• $f_{AN}(\text{net}) ∈ (0, 1)$
• λ = slope; usually $= 1$. for $\theta =0$:

5. Hyperbolic tangent

$f_{A N}(\text{net}-\theta)=\frac{e^{\lambda(n e t-\theta)}-e^{-\lambda(n e t-\theta)}}{e^{\lambda(n e t-\theta)}+e^{-\lambda(n e t-\theta)}}$ $≈f_{A N}(\text{net}-\theta)=\frac{2}{1+e^{-\lambda(n e t-\theta)}}-1$

• $f_{AN}(\text{net}) ∈ (-1, 1)$ for $\theta=0$:

6. Gaussian function

$f_{A N}(\text{net}-\theta)=e^{-(\text{net}-\theta)^2 / \sigma^2}$

• $\text{net-}\theta=$ mean of Gaussian distribution
• $σ=$ standard deviation of Gaussian distribution for $\theta=0$:

7. ReLU function