Multivariate Gaussian Distribution

The multivariate Gaussian Distribution is commonly expressed in terms of $\mu$ and $\Sigma$. The probability density function (PDF) of the multivariate normal (Gaussian) distribution is defined as:

$$f(x)=\frac{1}{\sqrt{(2\pi {)}^d|\Sigma |}}\exp \left(-\frac{1}{2}(x-u{)}^T{\Sigma }^{-1}(x-u)\right)$$ $$X\sim \mathbf{N}(\mu ,\Sigma )$$

• $X\text{ and }u\text{are vectors in }{\mathbb{R}}^d$.
• $X\text{is a random variable with mean }\mu$.
• $\Sigma \text{is the d x d PSD Covariance Matrix}$.
• ${\Sigma }^{-1}\text{is the d x d PSD Precision Matrix}$.
• $|\Sigma |\text{is the determinant of }\Sigma$.

---

Multivariate Normal Log PDF (with prior probability)

$$Q_c(x)=\ln \left(\sqrt{(2\pi {)}^d}\cdot f_{X|Y=C}(x).{\pi }_{Y=c}\right)$$

General logPDF with Prior Probability*

$$Q_c(x)=-\frac{1}{2}\ln |{\Sigma }_c|-\frac{1}{2}(x-{\mu }_c{)}^T{\Sigma }_c^{-1}(x-{\mu }_c)+\ln {\pi }_c$$

---

Anisotropic QDA logPDF (different covariances for each class)

$$Q_c(x)=-\frac{1}{2}\ln |{\Sigma }_c|-\frac{1}{2}(x-{\mu }_c{)}^T{\Sigma }_c^{-1}(x-{\mu }_c)+\ln {\pi }_c$$

---

Isotropic QDA logPDF

Isotropic $⟹{\Sigma }_c={\sigma }_c^{2}I$

The logpdf is simplified to:

$$Q_c(x)=-\frac{1}{2}\ln (|{\sigma }_c^{2}I|)-\frac{1}{2}(x-{\mu }_c{)}^T\left(\frac{1}{{\sigma }_c^2}I\right)(x-{\mu }_c)+\ln {\pi }_c$$ $$Q_c(x)=-\frac{1}{2}(2d)\ln ({\sigma }_c)-\frac{1}{2{\sigma }_c^2}(x-u_c{)}^{T}I(x-u_c)+\ln {\pi }_c$$

Reference ^ae513b

$$Q_c(x)=-\frac{\Vert x-{\mu }_c{\Vert }^2}{2{\sigma }_c^2}-d\ln {\sigma }_c+\ln {\pi }_c$$

---

Anisotropic LDA logPDF (pool covariance)

Let $\Sigma$ be the pool covariance of all the conditional covariances. (sum of expected covariances)

$$Q_c(x)=-\frac{1}{2}\ln |\Sigma |-\frac{1}{2}(x-{\mu }_c{)}^T{\Sigma }^{-1}(x-{\mu }_c)+\ln {\pi }_c$$

Decision Boundary (include prior probability)

$$Q_c(x)-Q_d(x)=({\mu }_c-{\mu }_d{)}^T{\Sigma }^{-1}x-\frac{{\mu }_c^T{\Sigma }^{-1}{\mu }_c-{\mu }_d^T{\Sigma }^{-1}{\mu }_d}{2}+\ln {\pi }_c-\ln {\pi }_d$$

---

Isotropic LDA logPDF

Let ${\sigma }^2$ be the pool variance of all the conditional variances. (sum of expected variances)

The Decision Boundary: (include prior probability)

$$Q_c(x)-Q_d(x)=\frac{({\mu }_c-{\mu }_d).x}{{\sigma }^2}-\frac{\Vert u_c{\Vert }^2-\Vert {\mu }_d{\Vert }^2}{2{\sigma }^2}+\ln {\pi }_c-\ln {\pi }_d$$

---

Isotropic LDA with Same Prior logPDF

The Decision Boundary: (Centroid Method)

$$Q_c(x)-Q_d(x)=\frac{({\mu }_c-{\mu }_d).x}{{\sigma }^2}-\frac{\Vert u_c{\Vert }^2-\Vert {\mu }_d{\Vert }^2}{2{\sigma }^2}$$

---

Anisotropic → Covariance Matrix is not Diagonal. Isotropic → Covariance Matrix is Diagonal.

Visualizing Quadratic Form Induced Norm

---