Tulsee Doshi on Why AI PMs Matter Again

Tulsee Doshi runs product for Google's frontier models, the family that includes Gemini, and she opens with an odd comparison. Training a large model, she says, is

"The model isn't predictable. It's kind of like Snape's potions class. So you can put all the ingredients in, but you also have to expect that there will be a little bit of magic." — Tulsee Doshi [00:00]

That idea, that a product manager's real material is an unpredictable system rather than a spec document, runs through everything else she says. It explains why Google is racing to prove it still belongs at what people call the frontier, the leading edge of model capability. Doshi points to Gemini 4 Argon as evidence the company is not conceding that ground (06:45), and she rejects the idea that Google should settle for being merely efficient rather than state of the art. "I actually think we do want to win the frontier conversation," she says, adding that Google's advantage is building across the whole stack, from hardware to consumer apps used by what she estimates are fourteen separate Google products with more than a billion users each (07:34).

On safety, her answer is practical rather than philosophical. One concrete mitigation her teams use is monitoring a model's chain of thought, the step by step reasoning a model produces, so engineers can catch and intervene when a model's actions drift from what it was directed to do (11:08). She also points to a partnership with the cybersecurity company Wiz, now part of Google, and describes a case study from Wiz's Stand for Good program in which the company identified and patched a vulnerability inside a hospital system (12:06). Her bigger claim is that optimism about AI safety is not a feeling, it is a discipline: "The path to optimism here is through thinking about it, and being very careful and very thoughtful and very intentional on the path to get there" (13:12).

The Return of Taste

The most quotable part of the conversation is about product managers themselves. For a stretch of the generative AI boom, the industry joke was that PMs were obsolete, since anyone could prompt a model into building something. Doshi says that phase is over, and for a specific reason. When building is nearly free, the scarce skill becomes judgment: knowing what is worth building and what counts as genuinely good.

"When you can build anything, and you can build anything so easily, building something great actually is now a unique value add." — Tulsee Doshi [20:30]

That shift changes what she looks for when hiring. She wants PMs who have personally developed a feel for how models behave, what she calls "vibes," by using them constantly and forming opinions (21:38). But she pairs that with a harder technical bar: PMs need to be comfortable building their own evaluations, commonly shortened to evals, which are structured tests that check how well a model performs a specific task. Her advice for anyone intimidated by that word is to start absurdly small. A useful first eval, she says, can be as few as twenty prompts that stress test one capability well enough to see a difference between two models (23:49). The trick to the vibes side is similar: pick a task that actually matters to you, whether that is financial analysis, creative writing, or image generation, because that is where a new model's improvements show up most clearly (25:01).

She illustrates why evals need constant updating with a story from her six years leading Google's responsible AI team. Early AI products judged skin tone representation using the Fitzpatrick scale, a six-category system (27:22). The scale had two problems. It was built for dermatology, not for representing a global population, and it skewed toward lighter skin tones even within that narrower purpose. Google eventually adopted a ten-tone scale developed by Harvard researcher Professor Monk through years of anthropological and sociological fieldwork, and used it to rework evaluations behind products including the Pixel Camera (27:46). The broader lesson she draws is that an evaluation is only as good as its weakest link, including the quality of whatever model is being used to judge the output (28:45).

Building for a Target That Keeps Moving

Because models improve every few weeks, Doshi argues that traditional product wisdom, build a minimum viable product that works, is incomplete advice for this environment. Her alternative is what she calls the 60/20/20 rule: build the first 60 percent of a feature so it works reliably today, add a 20 percent layer that works well enough for the near future, and reserve a final 20 percent for a capability the model does not fully have yet but shows early signs of (33:26). The payoff is a product that works now and gets better automatically as the underlying model improves, instead of one that has to be rebuilt every time a new model ships.

The risk, she warns, runs the other way. Teams sometimes build rigid systems around a model's current limits, and when the model improves, that rigid scaffolding actually holds the product back rather than helping it (35:47). The scaffolding she means is a harness: the system instructions, available tools, context, and skills that surround a model and define what it can act on (37:11). Her recommendation is to keep retesting the harness rather than assuming last quarter's workarounds still make sense.

None of this, she says, comes free of strain. Her team is working long hours, and she is candid that she has not solved the sustainability question yet. But she distinguishes between two different sources of burnout. Long hours on work you care about are exhausting but tolerable. Burnout, in her view, comes from churn, chaos, and what she calls grunt work, tasks that feel wasteful rather than meaningful (45:40). She sees a narrow opening for AI to help here, not by reducing hours but by removing the frustrating, repetitive parts of a job so people spend more time on the parts they actually chose the work for. One practical version of this shows up in how her team handles documents: she wants AI to supply the data, pulling dashboards, metrics, and user feedback into a draft, while the PM supplies the insight and judgment layered on top (49:01). The net effect, she argues, is that the PM's output has shifted from writing requirements down to providing what she calls heft, demonstrated through prototypes, evals, and data rather than prose (50:32).

Doshi closes the conversation somewhere unexpected: classical Indian dance, which she has performed since she was three years old, and a cultural festival in Portland she has helped organize for years. She describes dance as one of the few activities that empties her mind of everything else, and she credits a related habit, actively seeking serendipity by putting herself in uncomfortable, unfamiliar situations, as the closest thing she has to a life philosophy (54:33). It is a fitting way to end a conversation that keeps circling back to the same idea: in a field defined by unpredictable models and moving targets, the people who do well are not the ones chasing certainty. They are the ones who have built the discipline to stay calm while reading whatever the system hands back to them.