Neural Network

The Parity Problem

• Perceptrons - AI Winter

XOR	0	1
0	0	1
1	1	0

Solution: If you add one new quadratic feature $x_1{x}_2$, XOR is linearly separable in 3D.

There is another more powerful way to solve this XOR problem : Stacking Linear Combos

This is close, but it won’t work yet. A linear combo of a linear combo is a linear combo … only works for linearly separable points.

We need some non-linearity.

• Traditional Choice: Logistic Function (Sigmoid) : Smooth, well defined gradients and Hassian.
• RELU

---

Network with 1 Hidden Layer

Input Layer: $x_1,\dots ,x_d;x_{d+1}=1$ Hidden Layer: $h_1,\dots ,h_m;h_{m+1}=1$ Output Layer: ${\hat{y}}_1,\dots ,{\hat{y}}_k$

Layer 1 weights: $m\times (d+1)$ matrix V; $V_i^T$ is row i; weights into $h_i$ Layer 2 weights: $k\times (m+1)$ matrix W; $W_i^T$ is row i; weights into ${\hat{y}}_i$

The weights are in row. (since we want to do $Vx$). So, each $v_i$ in the row is gonna be the weight for $x_1,x_2,\dots$.

---