Article: An Opinionated Guide to ML Research
The keys to success are *working on the right problems, making continual progress on them, and achieving continual personal growth.* This essay is comprised of three sections, each covering one of these topics.
Develop good taste for what problems to work on.
- Read a lot of papers, and assess them critically.
- Work in a research group with other people working on similar topics.
- Seek advice from experienced researchers on what to work on.
- Reflect on what research is useful;
- When is theory useful?
- What causes some ideas to get wide uptake?
Idea-Driven vs Goal-Driven Research
- Idea driven
- follow some sector of literature, and improve on existing ideas. - Cons: High risk of getting snooped, or duplicating work, requires deep understanding of subject
- Have vision of new AI capabilities, and experiment with diferent methods across the literature. Develop your own methods to improve on them. - Pros: much more motivating. goals also give differentiating perspective from rest of the community.
John Schulman recommends goal-driven research.
Tips for Goal-Driven research
- Don’t take the goal too literally. Restrict yourself to general solutions.
- Aim high, and climb incrementally towards high goals
- Know when to switch problems. Also, don’t switch problems too often. Notice the dead-ends in half-finished projects, and if there aren’t any, make a commitment towards following through in the future.
- Have a fixed time budget for trying out new ideas that diverge from the main line of work.
- Read textbooks, theses and papers, reimplement algorithms from these sources. Especially textbooks, because they condense information in an ordered fashion, and using proper notation!
- Theses are a good place to find a literature review of an active field.