Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches (Dacrema et al., 2019)

The authors analyzed various recent publications on recommendation systems techniques, and found that these have:

  1. Weak baselines
  2. Establish weak methods as baselines
  3. Are outperformed by simple, sometimes non-neural approaches

The simple approaches that work well include ItemKNN, a collaborative-filtering approach that uses k-nearest neighbours and item-item similarities:

\begin{equation} s_{ij} = \frac{r_i \dot r_j}{\lvert r_i \rvert \lvert r_j \rvert + h} \end{equation}

To alleviate these issues:

  1. Use appropriate evaluation methods
  2. Evaluate on appropriate datasets (size is important)
  3. Release reproducible code

Bibliography

Dacrema, M. F., Cremonesi, P., & Jannach, D., Are we really making much progress? a worrying analysis of recent neural recommendation approaches, CoRR, (), (2019).