Zapier's Automation Bench: AI Scores on Real Work

Wade Foster has a confession that cuts against the whole AI hype cycle. Foster is the co-founder and CEO of Zapier, the automation company that lets people wire different software tools together. Despite what he calls a smooth ramp in model capability over the past year, he says plainly: "Life is kind of the same" (36:14). He watches his friends and family go about their days almost unchanged, using ChatGPT for meal planning or vacation ideas, never touching what he calls astro-level model capabilities. That gap, between what AI can technically do and what most people actually do with it, is the real subject of this conversation, recorded about eighteen months after Foster's last appearance on the show. In that earlier conversation, four hundred thousand Zapier customers had already delegated more than one hundred million tasks to AI (00:00).

Foster's first big observation is about where work now happens. Most people, he says, have settled into a single daily AI tool, whether that is Claude Code, ChatGPT, or, in his own case, the coding tool Cursor (01:02). That has forced a strategic choice on every software company: try to pull users into your own walled product, or make your tools available inside whatever tool the user already lives in. Foster calls the second option "headless," meaning the tool has no interface of its own and instead plugs into someone else's daily driver through a connector called an MCP. "The last decade was SaaS," he says, "but now it feels like there's this almost explosion of headless tools that is happening" (11:42). Software-as-a-service, the rented, browser-based tools that defined the 2010s, is giving way to invisible plumbing that any AI assistant can call on demand.

The second, more sobering observation is about how good these assistants actually are. Zapier built its own test, called Automation Bench, made of roughly six hundred real knowledge-work tasks spanning marketing, sales, HR, and operations (01:39). A recent model Foster calls Astra set a new record on it. That record is a 40% success rate (01:39). Even the best available model, in other words, still fails on more than half of ordinary office tasks like routing a closed deal to the right team or reconciling currencies across a spreadsheet. That number is the quiet correction to a year of AI triumphalism: capability is real and improving, but it is not yet reliable enough to hand over most white-collar workflows unsupervised.

Why the Boring Code Still Wins

Foster's practical answer to that unreliability is almost anti-climactic. Roughly 80% of the tasks people currently ask AI agents to do, he argues, should really be handled by old-fashioned, deterministic code, meaning code that produces the same output every time given the same input (15:27). "You really only want the AI to reason over the things that you need it to reason for," he says (15:27). The trick is teaching an AI assistant to recognize which 20% actually calls for judgment, and to write ordinary code for the rest, because deterministic code is cheaper, faster, and more reliable than asking a model to reinvent the same logic every single time (15:03, 14:43). It is a useful discipline for anyone tempted to route every task through a chatbot: the return on carefully specifying and validating a workflow once, in code, still beats delegating it fresh to an AI each time.

"80% in fact, they probably should be using actually old-fashioned deterministic code. You really only want the AI to reason over the things that you need it to reason for." — Wade Foster [15:27]

When something does break, Zapier does not lean on a single model to diagnose the problem. Its internal support system spins up five independent AI agents to evaluate a failed workflow, and when at least four of the five agree on a diagnosis, that agreement is treated as a reliable signal (25:32). A human still reviews the outcome with a simple thumbs up or thumbs down, and the reason behind any rejection feeds back into the system to sharpen future attempts (25:58). It is a modest, almost bureaucratic process, five votes, a human referee, a feedback log, but it is the kind of unglamorous infrastructure that actually makes AI-assisted troubleshooting trustworthy at scale.

Foster is also using AI to solve a much older problem: most people do not know what to automate, because they are creatures of habit who never stop to ask if there is a better way. His own fix is a weekly workflow that reviews his activity across Gmail, Slack, his browser, and Cursor, then proposes specific tools he should build (01:39 chapter reference; described 37:45). He calls the barrier a "recommendations gap" and says the exercise has produced new automations almost every week since he started it at the beginning of the year (38:04). Zapier plans to turn that same self-audit into a customer product (40:56).

Rebuilding the Business Around Intelligence

These shifts are forcing changes to how software gets sold and how companies organize themselves. Foster argues that seat-based pricing, charging a flat fee per user per month, is dying because intelligence itself is now the product, and a fixed seat fee does not track the value of variable, usage-driven work (42:27). He expects most products to land on usage-based or outcome-based pricing instead, depending on how cleanly a company can define what success looks like for a given task (42:57).

Inside Zapier, Foster put the company's chief people officer in charge of AI transformation, not because people officers are the natural choice everywhere, but because the company's bottleneck had shifted from technical adoption (nearly all employees already use AI daily) to organizational questions: rewriting job descriptions, retraining teams, redesigning compensation (47:30). His advice to other leaders is not to copy the org chart but to find whoever is already best equipped for the bottleneck a company is actually facing (49:07).

Zapier has also pushed toward "public by default" internal communication, moving more discussion out of private Slack channels and DMs into shared, visible channels, partly as a values reminder and partly because AI agents perform far better with access to that broader context (49:30, 51:05). Sensitive matters, active security incidents or HR cases, still stay private while they are live (51:30). On the cost side, Foster has watched some individual engineers spend $30,000 a month on AI tokens (55:46). Rather than cap spending, Zapier, now an 800-person company, built dashboards so employees can see their own usage and compare cost against output, with the expectation that formal token budgets tied to demonstrated skill are coming eventually (56:37).

Foster's clearest personal rule concerns writing. He is comfortable with employees using AI to draft communications, but insists that whoever sends a message still owns it. "If you wrote it, you probably should be putting more time into authoring the thing than the reader is reading it," he says (58:34). That means being able to answer questions about a document's content, making requests explicit, and checking that an AI has not quietly passed along stale or hallucinated details (58:50, 59:14). The goal, as he puts it, is not banning AI-assisted writing but banning low-quality writing, whatever produced it (60:07).

Taken together, these threads describe a company trying to be honest about a technology that is powerful in patches and unreliable in aggregate. Zapier's own benchmark data, a 40% success rate on ordinary office tasks, is the anchor for everything else: the case for deterministic code, the multi-agent troubleshooting loop, the slow organizational retooling. Foster's closing image is of Zapier building "the factory that builds the products," chipping away, release by release, at the parts still too unreliable to hand off (62:02). It is a smaller, more patient story than the one the AI industry usually tells about itself, and that may be exactly why it is worth reading.