Article: An Opinionated Guide to ML Research
- tags
- Machine Learning, Research
- author
- John Schulman
- source
- An Opinionated Guide to ML Research
The keys to success are *working on the right problems, making continual progress on them, and achieving continual personal growth.* This essay is comprised of three sections, each covering one of these topics.
Choosing Problems
Develop good taste for what problems to work on.
- Read a lot of papers, and assess them critically.
- Work in a research group with other people working on similar topics.
- Seek advice from experienced researchers on what to work on.
- Reflect on what research is useful;
- When is theory useful?
- What causes some ideas to get wide uptake?
Idea-Driven vs Goal-Driven Research
- Idea driven
- follow some sector of literature, and improve on existing ideas. - Cons: High risk of getting snooped, or duplicating work, requires deep understanding of subject
- Goal-driven
- Have vision of new AI capabilities, and experiment with diferent methods across the literature. Develop your own methods to improve on them. - Pros: much more motivating. goals also give differentiating perspective from rest of the community.
John Schulman recommends goal-driven research.
Tips for Goal-Driven research
- Don’t take the goal too literally. Restrict yourself to general solutions.
- Aim high, and climb incrementally towards high goals
- Keep a notebook, and record daily ideas and experiments (Note-taking). John Schulman creates an entry for each day, and conducts a review to condense entries every 1 or 2 weeks.
- Know when to switch problems. Also, don’t switch problems too often. Notice the dead-ends in half-finished projects, and if there aren’t any, make a commitment towards following through in the future.
- Have a fixed time budget for trying out new ideas that diverge from the main line of work.
Personal Development
- Read textbooks, theses and papers, reimplement algorithms from these sources. Especially textbooks, because they condense information in an ordered fashion, and using proper notation!
- Theses are a good place to find a literature review of an active field.