Stay on Top of SOTA: How an AI and Machine Learning PM Can Keep Track of Technical and Industry Advances
A question I often get is, as an AI/ML PM, how do I keep up with the fast-changing state-of-the-art (SOTA) for technical and product advances? How do I communicate my thoughts and ideas into the organization?
My framework: This is what I’ve come up with over the past few years - I hope it’s helpful, and am looking for ways to improve it.
Read lots of ML papers: Ideally 2-5 papers a week (“reading a paper” means knowing how to skim it properly - from the abstract and key diagrams to datasets and model architectures used, to the conclusions and caveats, to the objectives/tasks in the study, and then connect it to your web of knowledge in a space and specific problems you are thinking about and working on). LabML Weekly Papers, Yannic Kilcher on YouTube, or Jack Clark Import AI are good places to start, but if you just have a topic you’re interested in you can Google it and often find a paper written in the last 12-18 months (e.g. “ML study arxiv video user embeddings”).
One tip: Find a researcher who publishes in an area you care about (e.g. user sequence models for ads at Alibaba) and then look them up on Google Scholar and look through all their papers - often they publish more in the same area that is interesting. Same for the authors on the Google “Factory Floor of Ads” paper.
Play with AI-enhanced products: You have to use some AI-enhanced products by Meta and our competitors and have a sense of what the objective functions are and what the user experience (end user or advertiser). That could mean dogfooding ads manager as a tool with different people, or watching Tiktok vs FB Reels 2x a week to see what the organic and ad content is like, to form hypotheses on what is good or not. Play with the generative AI tools (ChatGPT, Perplexity, You, Neeva, Midjourney, Stability, DALL-E, etc) and come up with your ideas on how to use them better and how ads can be improved with new products.
Read some of the top newsletters and follow top podcasts: I usually read 1-2 newsletters a week and listen to 1-3 ML podcasts - cumulatively that adds up. See my list below. There are several "learning talks" that are given by engineers across the internet, via podcasts and YouTube. While the whole talk may sometimes be too dense, the first 10-15 mins are useful to gain a summary understanding of the why and what.
Use ML Twitter: Follow the top ML researchers on “ML Twitter”, where people post about interesting papers and results. Generally Twitter is a cesspool and a time sink, but you can set up an account to only follow ML researchers from industry and academia talking about what they published. I like to follow both top researchers, but also PhD and post-doc students in their labs who are publishing neat stuff. When on Twitter, you have to aggressively block or mute everything else the algo or system throws at you (their goal is to get you to engage on anything clickbaity, so if you go on, stay focused on ML people - you can see who I follow, or the people below follow, and get to 500-2000 researchers quickly).
Attend ML conferences: In any given year, there are like 15-25 worthwhile ML conferences. It’s worth attending at least one to see what papers are presented and to meet and talk with people - especially the product and practitioner ones like RecSys (versus the very theoretical ones like NEURips). After you attend, you should do a conference readout for your team. (Note that you may attend and work part time, so it’s hard to do, but worthwhile).
Keep learning via classes/seminars/coding: There are an endless number of ML courses to take, from the Andrew Ng Deep Learning ones on Coursera to Fast.AI on Deep Learning and Diffusion to academics putting materials on the web like Stanford classes on transformers, foundation models, ML Ops, etc. For some you watch videos, others you read notes and take your own, and others you code up toy systems. All are worthwhile - here is a starter list. I like to take 1-2 courses a year.
What PMs can Write and Communicate
Notes to engineers: Short email to relevant engineers and EMs on some papers you read, or interesting industry trends or specific products to highlight (eg Pinformer paper from Pinterest, or how Tiktok processes video ads, etc) - especially connecting them to key products or features at your company (and how they can increase value to customers and generate revenue, or reduce costs for your company)
State of the art in industry post: Do a high-level survey on the entire industry for one theme (e.g. native SFV ads, upstream embedding models, etc), or a deep dive on one company (e.g. Tiktok or YouTube Shorts).
Key application of new tech post: Go into a new architecture or technique that could have applications for ads (e.g. RL for LTV, or end to end candidate selection, etc)
Papers to Read and People to Follow
How to Read a CS Paper: Great basic intro, in case it’s been a while.
Star ML researcher and teacher Andrew Ng on how to read an ML/DL paper
Getting Started with Deep Learning Papers: Some good advice on getting started. I would take it and then tackle this roadmap: Deep Learning Paper Reading Roadmap and then check out these DL resources. While you’re at it, I would read these classics, the early Deep Learning Nature paper (not easy), and the maybe the DL overview by Schmiduber. Here is a fun application NEJM paper by Jeff Dean and others on Machine Learning in Medicine.
Most cited deep learning papers: this list is pretty good, but is light on SSL and RL and ends around 2017. Some newer lists: 2022, 2021, etc.
Year-end ML research reviews: The top labs in the world put out an annual, year-end research review that is worth reading, as are the top pick papers in conferences like ICML, NeurIPS, IJCAI, CVPR, ECCV, ACL, EMNLP, RecSys, ScaledML, etc. Check out 2019-2021 reviews at Google Research, Microsoft Research, FAIR, IDSIA, DAIR.AI, etc.
ML and Data Science Newsletters and Podcasts:
Jack Clark Import AI (from OpenAI)
Deeplearning.ai’s The Batch (from Andrew Ng’s team)
Lex Fridman Podcast (it’s fabulous!)
TWIML AI Podcast (more technical, more on MLops)
Yannic Kilcher on YouTube (dissects ML papers)
Academic “ML Twitter” is a great way to follow ML experts and research organizations:
Anima Anandkumar — see her great talks here.