5. Research & Projects
Problem first #
We prioritize in this order: the problem, then the data, then the model. A clever model on a problem nobody cares about, or on data that can’t support it, won’t get far. When you describe a project, work through it in that order: the problem, the ML task, the data, the modeling, and how you’ll evaluate it.
New ideas #
I’m open to research ideas. Sell me on them: why is it worth the time, and is it feasible? Anything can become a research paper. The work is finding an interesting way of telling the story. Common failure modes are getting too attached to one direction, not knowing the field well enough yet, and working in an environment that doesn’t support your interests. Talking to other researchers about their taste helps with all three.
De-risk your ideas #
Learning to de-risk ideas is one of the most useful research skills you can build. The idea is “kill fast, learn fast”: test the core assumptions quickly, before investing months.
One exercise: list the likely bottlenecks, prioritize them, commit a fixed amount of time to exploring each, then decide whether to continue or punt. For example, if you want to fine-tune a large model, first time a single batch of your largest model on fake data. Does it fit in memory? Multiply by the number of batches and a guess for epochs. Is the time to train one model reasonable?
Balancing multiple projects #
Once you’ve ramped up, I expect you to juggle two or three projects at a time. Research has natural ebbs and flows, whether you’re waiting on compute, collaborators, or paper reviews, and having more than one project helps fill those gaps. Still, there is real value in focusing on one thing at a time, and we try to walk that tightrope together.
Design docs #
Any time you’re about to spend more than a week on something, write a short design doc first covering what you’re building, why, the alternatives, and how you’ll know it worked. Writing it down helps clarify your plan and gives others a chance to spot issues early. Good introductions: Design docs at Google and Things I learned at Google: design docs.
The project spreadsheet #
We keep a project spreadsheet [TBD] with ideas, timelines, and who’s working on what. Seeing active and paused ideas in one place makes it easier to talk to people and find connections across projects.
Pivoting is normal #
Recycling, punting, and pivoting on projects is normal. Sometimes ideas don’t work, and what matters is diagnosing that as early as possible.
Sometimes deadlines get met by sprinting. In general it’s better to plan ahead, but sprints happen. If you’re going to do one, be focused about it so you can actually meet the deadline.
Reproducibility and data #
Our work should be reproducible, with good data management. I’m a big fan of Cookiecutter Data Science, an opinionated and sensible way of organizing projects, and our project template builds on it. A few of its opinions we follow:
- Raw data is immutable. Never edit or overwrite it: code should take raw data to final results.
- Data mostly stays out of git. GitHub rejects files over 100 MB, so keep data on lab or cloud storage.
- Use notebooks for exploration and communication, and move reusable logic into source files.
- Keep modeling organized: configs and hyperparameters in JSON or YAML, predictions and training logs saved alongside.
- Build from the environment up, with an environment file (we use Mamba).
Be transparent with your work and show it: put it in a GitHub repo and make commits. We usually won’t look at your commits in detail until we’re publishing something. What matters more is that you have a consistent way of keeping track of your work. When a project is ready to turn into a paper, see Writing & Publishing.