37 Conjugate Priors
We have mentioned, many of the times, we do not have closed-form solution for the posterior. However, there is a class of models — pairs of likelihoods and priors — that an analytic posterior exists. These pairs of likelihoods and pairs are referred as conjugate.
| Likelihood | Prior | Posterior |
|---|---|---|
| Bernoulli | \(\text{Beta}(\alpha,\beta)\) | \(\text{Beta}(\alpha+\sum x_i,\beta+n-\sum x_i)\) |
| Binomial | \(\text{Beta}(\alpha,\beta)\) | \(\text{Beta}(\alpha+\sum x_i,\beta+\sum n_i-\sum x_i)\) |
| Multinomial | \(\text{Dirichlet}(\alpha)\) | \(\text{Dirichlet}(\alpha+\sum x_i)\) |
| Normal (known \(\sigma^2\)) | \(N(\mu_0, \sigma_0^2)\) | \(N\left(\frac{1}{\frac{1}{\sigma_0^2}+\frac{n}{\sigma^2}}\left(\frac{\mu_0}{\sigma_0^2}+\frac{\sum x_i}{\sigma^2}\right),\left(\frac{1}{\sigma_0^2}+\frac{n}{\sigma^2}\right)^{-1}\right)\) |
| Normal (known \(\mu\)) | \(IG(\alpha,\beta)\) | \(IG \left(\frac{\alpha+n}{2}, \frac{\beta+\sum(x_i-\mu)^2}{2}\right)\) |
| Possion | \(\Gamma(\alpha,\beta)\) | \(\Gamma(\alpha+\sum x_i,\beta+n)\) |
Using conjugate priors, we can plug-in the data into the formula and get the exact posterior distribution. But the limitation is obvious. We are confined to use a given set of distributions, whereas other distributions do not have conjugate properties.