Two Levels Of Inference
- tags
- Occam’s Razor
There are 2 levels of inference: model fitting and model comparison.
In model fitting, assuming a model is true (say
The normalizing constant is irrelevant to the first level of
inference. It is common to use gradient-based methods to find the
maximum of the posterior
locally approximating the posterior as a Gaussian with covariance
matrix
In model comparison, we compare models in light of the data, assign some sort of preference.
Bayesian methods can consistently and quantitatively solve both types of inferences, although adopting the Bayesian method for the first type leads to similar results from orthodox statistical methods. Orthodox statistical methods will find it difficult to perform model comparisons, because it is not possible simply to choose the model that fits the data itself. For example, maximum likelihood can fail by choosing implausible, over-parameterized models that overfit the data.
How do Bayesian methods perform model comparison? The posterior probability for each model is:
Hence, if we assign equal priors to the alternative models, models