The 4 Areas to Focus on in Applied AI Research
Why Care About Prioritizing Time?
Let’s be clear —if you’re early in your career in AI, then this blog might not be worth caring about (for now).
That’s because this blog post is about the future. If you’re reading this, I’m going to guess that you’re probably interested in pursuing research at some point. In which case, that probably means you’re excited to see how your ideas could change the world, in one of the field where machine learning is being applied at the moment (e.g. from healthcare, autonomous vehicles, wearables, and the list goes on).
If that is the case, I have both good news and bad news. Let’s start with what’s bad:
As exciting as research can be, the reality is that creating machine learning models can be incredibly tedious work.
For example, when doing machine learning in industry you might just start a project and discover that:
- you have to write an ETL script all from scratch, as no pre-existing data sources exist at your lab/company
- your hardware accelerators are underpowered, and therefore you need to wait hours/days in between trainings before analyzing your experiments
- and if you opt to train models in the cloud, you might still need to set up AWS EC 2 instances all on your own, so that you can use faster hardware
This is because for many companies, machine learning is still relatively nascent. That means for you, me, and many practitioners I know in the field, we might have started in AI research because we wanted to be the next Geoff Hinton — but instead, we can become bogged down with being responsible for several functions beyond just doing research.
If this sounds like you, or it sounds like what you want to avoid, there’s good news too: many people have been in your shoes before. Most of us eventually do get to the point where we are able to minimize the effects of feeling overwhelmed in research (e.g. going into burnout, missing project deadlines, going over budget, etc.). We rarely fall into those pitfalls.
Goal of the Blog
My aim in this blog is to make that process shorter for you, by sharing what has (and hasn’t) worked for me. Because I’m speaking from experience, this should be most relevant to Applied AI researchers/engineers at smaller startups. However, anyone who’s at a company where AI is still “new” to the engineering culture might still benefit.
I’ll share the 4 areas that truly deserve your attention when you’re developing models for machine learning projects. My goal is to by the end of this blog, you will feel more empowered to:
- Brainstorm projects that let you dive deeper into any technical area you want to expand in,
- Increase job satisfaction by reducing distractions and (in turn) getting more done,
- Build a world-class support network in your company who will fuel your trajectory up the ladder.
Disclaimer
The goal of this blog is not to give you the right goals to prioritize upfront. Only you can do that. Please treat this instead as a starting point, to make sure you at least know where to look for valuable work in your own environment.
Also, there are several competencies that make a great researcher that I already assume you know, e.g. being able to read and self-study academic papers related to whatever domain you’re working in. But, in case you aren’t familiar with these skills, I would highly recommend the GradCoach Blog as a free resource to learn more.
If you have any questions, absolutely feel free to contact me; because we probably all could benefit from collaborating on developing better work-styles and goal setting strategies :)
4 Types of Priorities to Plan Work Around
So let’s say you sit down on Day 1 of your new project. You hop into a meeting with your manager about the new machine learning project (of which you’re leading). And they ask you “what are your goals for this?” — how do you reply?
The irony I find is, research inherently requires us to explore and think broadly. When setting goals however, alot of places typically look for a clear, specific outcome from our work. So how do we ensure we are making the most innovative work as researchers, while also delivering tangible value at the end of the day, so that our company succeeds?
Here are the four areas in which I try to look for valuable problems to solve:
I: Contributing to the “Sprint”
If you’re familar with agile development, you probably are familiar with what a sprint is. This is when you would be coding new features, fixing bugs, writing tests, and other things like being on-call for build issues (and of course, code review).
- How does this apply to research?
I get it— not everyone like agile. But one hidden advantage of working in sprints is you get to make small wins. I once went to a mentor when I first working on a computer graphics project for a WebVR startup called iQ3Connect. I had been stuck on a single problem for weeks, even though I was reading ~10 papers a week on the subject. I asked him what I could do to break out of the stagnation in my research project. His advice:
Break. Things. Down.
In my case, if I could go back I would definitely have spent more effort on solidifying my foundational knowledge, as opposed to always scanning the field for new developments.
For graphics specifically, this could be as simple as taking a quick break in the middle of the week to refresh yourself on Peter Shirely’s Ray Tracing in One Weekend (2020). For machine learning, if you’re working with a CNN, you might want to try and refresh on how to write a convolution function from scratch.
II: Contributing to the Team
If the first area applied to your current “sprint”, then this applies to how the team will handle future sprints.
Truth be told, I have only worked on small teams so far in industry. But the benefits of a strong team dynamic are probably always worth pursuing. That’s because in both research and engineering, we need collaborators.
In pure software development, your work here might look like improving the team wiki/docs (e.g. on how we write tests), having sprint reviews, reducing tech debt, finding/being a mentor to someone.
In AI research, you can still do alot of these same things! For example, take the time to explain the TensorFlow API to a newcomer over StackOverflow (here’s my own example), or refactor the variable names in the official GitHub implementation of your favorite paper.
- Pro-tip #1: one of the benefits I promised you above is being better at brainstorming new project ideas — one way I’ve done this is by setting up “informational interviews” with people around the org (or those you interact with on the platforms mentioned above) in the first few months on the job. An extra tip by Adam Braus here: start with managers who are not in charge of your team (because it’ll be more relaxed).
III: Contributing to the Org
To explain this area, let’s go back for a moment to when we were in school. When you did a machine learning project back then, did you ever need to worry about where the dataset came from? Or how you would deploy the model? Even, how you would re-deploy the model over time as needed?
All these operations — sometimes called “DataOps” or “MLOps” today — are a part of what separates effective teams working on machine learning, to those who are not. From those who can handle large, unstructured datasets, from those that aren’t.
Out of the four areas, I would argue this is the most similar to normal software engineering in industry — and again, it’s another high-value place to interesting problems. You might find yourself:
- creating Docker images to streamline environment setup for people who join your project
- (or if you’re into Anaconda, “environment.yml” also comes into play here),
- creating a mini-machine learning framework (my example here) for developers to implement common tasks faster (e.g. logging model metrics)
Whatever it looks like for you, be proud of what you do in this area— this kind of work is innovative because the impact of these activities is outside what we can see in a single sprint; instead, they can potentially be felt for many months/years into the future.
- Pro-tip #2: objective #2 is about preventing stagnation — one way that’s helped me do this better is keeping a daily journal of what I’m learning about the org — so I can learn how to unblock issues faster
IV: Contributing to the Industry
OK, so I actually don’t have many recommendations for this area. That’s because if you’re able to protect your time and set up systems properly, this is probably the area you’ll end up making the most impact in anyway, by getting to do your awesome research in the first place :)
Usually, what I’ve noticed is this usually takes the form of creating new components for your company’s infrastructure (perhaps more so if your research is in architecture) or all new fields of business entirely. I will go ahead and cite a few examples of this, as a source of inspiration when setting your own goals:
Examples:
- Chris Lattner (research example): interview clip speaking about the way Java “changed the world”
- Amazon (architecture example): after they restructured their APIs + started running on their own servers → led to a whole new branch of their business as a cloud provider, AWS, which so many of us use today
- Pro-tip #3 — objective #3 is about creating space to do what really interests you — the tip here (just like in pro-tip #1) might be again to meet new people and do informational interviews . BUT, the key difference is they should ideally be people closer to your level (e.g. people who are 1–2 yrs ahead of you, as opposed to managers). I’ve found this population tends to know more about opportunities like internal hackathons, OSS projects, etc. the company has, where you can scratch that creative itch :)
Conclusion
Let’s recap
Let’s be clear: this post doesn’t promise you that your research will succeed. If there’s anything I’ve learned, there doesn’t seem to be any hard guarantee of that.
I’ve yet to meet anyone who’s truly found a way to balance the goals they have for their AI project, with the changing demands that come from the actual work environment they go into (regardless of whether that be in the form of the existing infrastructure/data/people, or just that your final conclusions go against your hypothesis). Therefore, I thought this would be a worthwhile post to write.
Where to Now?
“How we spend our days is, of course, how we spend our lives” — Annie Dillard
Here’s why I love the quote above: at the end of the day, we — as AI researchers — tend to dream a lot about the future. We’re ambitious.
But perhaps just as important as ambition, is being able to reframe our definition of success in light of change. To be able to make use of our time, even if our projects don’t end up exactly reaching the destination we envisioned originally.
I’ve shared with you 4 areas where you can add value, that are adjacent to doing your actual AI research. They are:
I: Contributing to the “Sprint”
II: Contributing to the Team
III: Contributing to the Org
IV: Contributing to the Industry
My ask for the Reader
Now that you know about these areas, I invite you to see how they can supplement your own work in AI:
- Do your own reflection in response to this post:
- How well are you doing in prioritizing these areas? Do you feel you get enough done?
- What could your next steps be to create more value?
- What feedback do you have for the author (i.e. me?) What do you prioritize as a software engineer, and/or what could I have mentioned in addition to what’s above (or elaborated on clearly)?
