Understanding Covariance Matrix for GDA

Suppose we have two random variables — could be column vectors or scalars.

By definition, Covariance is:

$$Cov(R,S)=\mathbf{E}\left[(R-\mathbf{E}[R])(S-\mathbf{E}[S{]}^T)\right]=\mathbf{E}[R{S}^T]-{\mu }_R{\mu }_S^T$$

By definition, Variance is:

$$Var(R)=Cov(R,R)$$

Suppose R is a vector, the Covariance Matrix of R is:

$$Var(R)=\left[\begin{array}{cccc}Var(R_1)& Cov(R_1,R_2)& \dots & Cov(R_1,R_d)\\ Cov(R_2,R_1)& Var(R_2)& \dots & Cov(R_2,R_d)\\ & & \ddots & \\ Cov(R_d,R_1)& Cov(R_d,R_2)& \dots & Var(R_d)\end{array}\right]$$

Warning

The Covariance matrix is symmetric, and positive semidefinite (PSD).

• If $R_i\text{ and }{R}_j$ are independent $⟹$ $Cov(R_i,R_j)=0$.
• The reverse is not true.
• Thus, pairwise independent $⟹$ Var(R) is diagonal.
• Again, the reverse is not true.

If all features pairwise independent, then we can write the joint normal pdf as $f(x)=f(x_1)f(x_2)\dots f(x_d)$.

If Var(R) is diagonal, that means the ellipsoid (for ${\Sigma }^{-1}?$) are axes-aligned with squared radii on diagonal of $\Sigma$.

Cite

So when the features are independent, you can write the multivariate Gaussian PDF as a product of univariate Gaussian PDFs. When they aren’t, you can do a change of coordinates to the eigenvector coordinate system, and write it as a product of univariate Gaussian PDFs in eigenvector coordinates. You did something very similar in Q6.2 of Homework 2.

---