Gibbs Sampling
Gibbs sampling is a special case of the Metropolis-Hastings method, where a sequence of proposal distributions \(q\) is defined in terms of the conditional distributions of the joint distribution \(p(\mathbf{x})\), and proposals are always accepted.
In the general case of a system with \(K\) variables, a single iteration involves sampling one parameter at a time:
\begin{equation} \begin{array}{l}{x_{1}^{(t+1)} \sim P\left(x_{1} | x_{2}^{(t)}, x_{3}^{(t)}, \ldots, x_{K}^{(t)}\right)} \ {x_{2}^{(t+1)} \sim P\left(x_{2} | x_{1}^{(t+1)}, x_{3}^{(t)}, \ldots, x_{K}^{(t)}\right)} \ {x_{3}^{(t+1)} \sim P\left(x_{3} | x_{1}^{(t+1)}, x_{2}^{(t+1)}, \ldots, x_{K}^{(t)}\right), \text { etc. }}\end{array} \end{equation}
Pros and Cons
- Suffers the same defects as Metropolis-Hastings methods
- No adjustable parameters, so it’s easy to start with