Jethro's Braindump

Article: An Opinionated Guide to Machine Learning Research

The keys to success are working on the right problems, making continual progress on them, and achieving continual personal growth. This essay is comprised of three sections, each covering one of these topics.

Choosing Problems

Develop good taste for what problems to work on.

  1. Read a lot of papers, and assess them critically.
  2. Work in a research group with other people working on similar topics.
  3. Seek advice from experienced researchers on what to work on.
  4. Reflect on what research is useful;
    1. When is theory useful?
    2. What causes some ideas to get wide uptake?

Idea-Driven vs Goal-Driven Research

Idea driven
follow some sector of literature, and improve on existing ideas.
  • Cons: High risk of getting snooped, or duplicating work, requires deep understanding of subject
Goal-driven
Have vision of new AI capabilities, and experiment with diferent methods across the literature. Develop your own methods to improve on them.
  • Pros: much more motivating. goals also give differentiating perspective from rest of the community.

John Schulman recommends goal-driven research.

Tips for Goal-Driven research

  1. Don’t take the goal too literally. Restrict yourself to general solutions.
  2. Aim high, and climb incrementally towards high goals
    1. Keep a notebook, and record daily ideas and experiments (Note-taking). John Schulman creates an entry for each day, and conducts a review to condense entries every 1 or 2 weeks.
  3. Know when to switch problems. Also, don’t switch problems too often. Notice the dead-ends in half-finished projects, and if there aren’t any, make a commitment towards following through in the future.
  4. Have a fixed time budget for trying out new ideas that diverge from the main line of work.

Personal Development

  • Read textbooks, theses and papers, reimplement algorithms from these sources. Especially textbooks, because they condense information in an ordered fashion, and using proper notation!
  • Theses are a good place to find a literature review of an active field.