Maximum Likelihood Estimation

How do we use Maximum Likelihood Estimation for Generative Models (Gaussian Discriminant Analysis, QDA, LDA)?

In Gaussian Discriminant Analysis, we understood the approach of generative models. Particularly, we want to find the probability distribution of the underlying classes rather than just the decision boundary. However, in real life, we do not know those probability distributions accurately. Therefore, we need some sort of estimation tool to figure out the prior probability and the probability distribution of the feature classes.

Equivalently,

• We model the class conditional probability distributions $P(X|Y=k)$ for each class k (often assuming that they’re Gaussian).
• Estimate the parameters of these distributions—such as the mean vectors, covariance matrices, and the class priors—from the training data (commonly using methods like Maximum Likelihood Estimation).

MLE is used for estimating the parameters of a statistical distribution. In GDA, we need to estimate the normal distribution (continuous) of each class and prior probability (discrete) for each class.

---

Coin Flipping Exercise (for prior probability)

Flip a Biased Coin with Head probability p and tail 1-p. Suppose that I flip the coin 10 times, and got 8 heads and 2 tails

Question: what is the value of bias p that is the most likely to lead to this outcome.

Recall that the number of heads is binomial distribution: $X\sim B(n,p)$

$$P[X=x]=\left(\genfrac{}{}{0ex}{}{n}{x}\right)p^x(1-p{)}^{n-x}$$

Thus, the probability of getting 8 heads in 10 flips is

$$P[X=8]=\left(\genfrac{}{}{0ex}{}{10}{8}\right)p^8(1-p{)}^2$$ $$P[X=8]=45p^8(1-p{)}^2$$

Let’s define this expression as the “Likelihood”. We will see why we call likelihood instead of probability when it comes to the continuous distribution case.

Therefore, the LIKELIHOOD FUNCTION is:

$$\mathcal{L}(p)=45p^8(1-p{)}^2$$

Optimization problem. $\arg {\max }_p\mathcal{L}(p)$

We can solve this by finding the critical point of $\mathcal{L}$

$$\frac{d\mathcal{L}}{dp}=0$$ $$360p^7(1-p{)}^2-90p^8(1-p)=0$$ $$p=0.8$$

Intuitively, this calculation yields what we expected, which is $0.8=\frac{x}{n}=\frac{8}{10}$

---

Estimated Prior Probability

Important

Given that we observe an event “A” x times out of n trials and we want to estimate the Prior Probability of event A happening, then the estimated Prior Probability is :

$${\hat{\pi }}_A=\frac{x}{n}$$

Another Definition: Suppose our training set is n points, with x in class C. Then our estimated prior for class C is ${\hat{\pi }}_C=\frac{x}{n}$.

---

Likelihood of a Gaussian

We have training data (sample points) $X_1\dots X_n$. We would like to find the best-fit Gaussian (meaning find the best $\mu$ and ${\sigma }^2$ for given training data.)

Difference between Probability and Likelihood

In Continuous Distribution, the probability of getting a particular point is zero. However, in Likelihood, we will ignore this phenomenon and consider that it’s not zero.

Likelihood of Gaussian is defined as:

$$\mathcal{L}(\mu ,\sigma ,X_1,\dots ,X_n)=f(X_1)f(X_2)\dots f(X_n)$$

To simplify the computation, we take the log of this and called it the Log Likelihood, $l(.)$

$$l(\mu ,\sigma ,X_1,\dots X_n)=\ln f(X_1)+\ln f(X_2)+\ln f(X_n)$$

Recall the PDF of Multivariate Gaussian Distribution is:

$$f(x)=\frac{1}{(\sqrt{2\pi }\sigma {)}^d}\cdot \exp \left(-\frac{\Vert x-\mu {\Vert }^2}{2{\sigma }^2}\right)$$

Each $f(X_i)$ is a Normal Distribution, thus,

$$l(\mu ,\sigma ,X_1,\dots X_n)=\sum _i^n\underset{\ln \text{of Gaussian}}{\underbrace{\left(-\frac{\Vert X_i-\mu {\Vert }^2}{2{\sigma }^2}-d\ln \sqrt{2\pi }-d\ln \sigma \right)}}$$

Similar to the discrete case, we can take the derivative of log likelihood to find the critical point (maximum).

Estimation of $\hat{\mu }$

$${\nabla }_{u}l=\sum _{i=1}^n\frac{X_i-\mu }{{\sigma }^2}=0⟹\hat{\mu }=\frac{1}{n}\sum _{i=1}^n{X}_i$$

Note that this expression is the same as the Sample Mean.

Estimation of Variance ${\sigma }^2$

$$\frac{\partial l}{\partial \sigma }=\sum _{i=1}^n\frac{\Vert X_i-\mu {\Vert }^2-d{\sigma }^2}{{\sigma }^3}=0⟹{\hat{\sigma }}^2=\frac{1}{dn}\sum _{i=1}^n\Vert X_i-\hat{\mu }{\Vert }^2$$

$\hat{\mu }$ is used since we do not know the exact mean of the distribution. Our best is the estimated u ($\hat{\mu }$)

Takeaway

Use Sample Mean and Sample Variance* in class C to estimate the mean and variance of class C Gaussian. * Almost Sample Variance except we’re using the estimated $\hat{\mu }$.

Conclusion:

• QDA: Estimate the conditional mean, ${\hat{\mu }}_C$ and conditional variance ${\hat{\sigma }}_C^2$ of each class Separately and estimate Prior Probabilities ${\hat{\pi }}_c$.
• LDA: Estimate the conditional mean, ${\hat{\mu }}_C$ and Prior Probabilities ${\hat{\pi }}_c$ and One Variance for all Class.
• We define “one variance for all class” as:

$${\hat{\sigma }}^2=\frac{1}{dn}\sum _C\sum _{i:i\in C}\Vert X_i-{\mu }_C\Vert$$

Shewchuk

Notice that although LDA is computing one variance for all the data, each sample point contributes with respect to its own class’s mean. This gives a very different result than if you simply use the global mean! It’s usually smaller than the global variance. We say “within-class” because we use each point’s distance from its class’s mean, but “pooled” because we then pool all the classes together.

• Basically, the mean and prior probability calculation remains the same, whereas for variance, the calculation of variance is different in such a way that we use “class mean instead of overall mean” in calculating the “one” variance.