Meta Learning
Learning to learn: learn an update rule from related tasks
For example, tasks are related through a low-dimensional embedding.
Model-Agnostic Meta Learning (MAML)
Based on 2nd-order gradient descent:
2-stage gradient-based approach on batches of tasks \(\mathcal{T}\):
- Inner loop:
\begin{equation} \theta_i’ = \theta - \alpha \nabla_\theta L_{\mathcal{T}}(f_\theta) \end{equation}
- Outer Loop:
\begin{equation} \theta=\theta-\beta \nabla_{\theta} \sum_{\mathcal{T}_{i} \sim p(\mathcal{T})} \mathcal{L}_{\mathcal{J}_{i}}\left(f_{\theta_{i}^{\prime}}\right) \end{equation}