Erik Bernhardsson on AI Inference Economics

In March, Erik Bernhardsson spent a few days at Nvidia's annual GTC conference in San Jose, moving between roughly 500 cloud vendors, asking each one the same question: do you have GPUs? He has a name for what that felt like. "I would describe that as, like, an open-air drug market, but GPUs instead of drugs," he says (18:00). It is a vivid line, and also an honest one. Bernhardsson runs Modal, an infrastructure company that rents out computing power and tools for building AI applications. Modal just closed a $750 million round led by Accel, pricing the company at $15.75 billion, and crossed $1 billion in annual recurring revenue, the industry's shorthand for yearly subscription income (01:01). Getting there has meant learning to buy chips the way a trader buys anything in a shortage: fast, relentlessly, and often for more than the sticker price.

The shortage is not permanent, in Bernhardsson's view, just loud right now. He borrows a line from Nvidia CEO Jensen Huang: hardware prices are like gravity, they always come down eventually, but right now supply cannot keep up with demand, so prices have been pushed up instead (22:02). Modal manages this by forecasting three to four months ahead, the typical lead time for new GPU orders, using a straight-line projection of its own growth, which has been running 30 to 40 percent a month for the past year (20:46). The company then buys roughly two to three times its current capacity to stay ahead of the curve (20:46). That capacity gets pooled across thousands of customers, whose individual usage is spiky and unpredictable but whose combined demand smooths out into something manageable (19:14). The payoff for a customer is that they never have to lock into a fixed block of GPUs for three years. They pay per GPU-second and scale up or down as needed (20:07).

Where the Dollar Goes

The more interesting argument in this conversation is about money, not metal. A decade ago, Silicon Valley software companies routinely boasted gross margins above 90 percent, meaning that for every dollar of revenue, the cost of actually delivering the product was small. Bernhardsson says that era is over, and the reason is structural. "The 90% software margin businesses are gone," he says (16:24). A dollar spent on an AI application today has to pass through a stack of vendors: the application itself, the underlying model, the infrastructure layer, the cloud provider, and Nvidia's chips. At every layer, someone takes a cut of 20 to 30 percent (16:07). The pie is enormous, since inference, the process of running a trained AI model to generate answers, is shaping up to be as large a market as cloud computing itself, in his estimate. But it is sliced into more pieces than software margins used to allow. Bernhardsson thinks Modal can eventually reach 40 to 60 percent margins, high for an infrastructure company, but he is explicit that software-grade margins are not coming back (16:24).

This is also why he resists selling raw inference as a commodity. "If you're just selling, like, tokens for dollars, there's no switching cost, you know. People can just switch to a different provider. It's just a race to the bottom," he says (14:32). His answer has been to bundle inference with adjacent products: sandboxes, which are isolated environments where AI agents can safely run code, along with training, post-training, and reinforcement learning tools. The logic is that a customer who depends on five interlocking products is harder to poach than one buying a single service. It took four years for Modal's second product, sandboxes, to matter. Launched in 2023, it stayed a minor line item until the summer of 2025, when newer models got dramatically better at writing whole applications. Sandbox revenue then jumped to a $300 million run rate, about a third of Modal's total business (34:37). Company-wide revenue grew 15 times over the past twelve months (39:12).

On open source inference specifically, Harry Stebbings raises the idea that usage has already peaked, an argument he attributes to fellow podcaster Jason Lemkin, and Bernhardsson pushes back. He expects both open and closed models to grow by 5 to 20 times over the next year, calling the peaked narrative "a fixed pie size mentality" (38:17). Nvidia's own interest in propping up open models, through investments in companies like Reflection, reinforces this: he sees it as Nvidia defending its ecosystem rather than betting against open source (28:56). What he does concede is that CUDA, Nvidia's programming platform, is not really the moat people assume. "CUDA's actually kind of annoying to work with," he says, but the surrounding ecosystem, the optimized libraries and open-source engines built to run on Nvidia first, is what keeps everyone locked in (24:31).

The Jobs Argument

The conversation's other big thread is what happens to engineers as AI gets better at writing code. Bernhardsson's answer draws on thirty years of watching programming languages evolve, from assembly code and punch cards through COBOL, C++, and Python, and now to AI coding agents (43:41). His claim is simple: every time programmers get more efficient, demand for software goes up, not down, because cheaper software unlocks uses that were not worth building before.

"Every single time we make programmers more efficient, it just increases demand for software." — Erik Bernhardsson (43:22)

That optimism comes with a boundary, though. Bernhardsson thinks reinforcement learning, a training method that improves a model through trial, error, and feedback, works best on tasks with fast, checkable outcomes. Coding fits because code can be run and tested instantly, which is why he says many AI labs run their reinforcement learning workloads on Modal's sandboxes (41:41). Cybersecurity and math show similar promise, and he points to recent proof-solving results from OpenAI as evidence (42:05). Law is only partly tractable, since it allows for evaluation but lacks as clean a feedback signal (42:05). Longer-horizon engineering work, the kind that takes a human weeks or months to judge, sits outside what reinforcement learning can currently reach, in his view, though he adds that he has been wrong about this kind of prediction before (42:28).

Talent is the sharpest edge of this story. Modal spends about $500,000 a month on AI coding agents across a team of 150 engineers (44:15), and Bernhardsson says the company tries to pay in the 95th percentile of the market for new hires. Even that is not always enough. "We've lost new grads going to, you know, some of the labs for, like... talking, you know, 20-plus million. I mean, these are exceptional people," he says, describing compensation packages at frontier AI labs that can exceed $20 million (53:22). His counter is equity, not cash: Modal allows employees to sell up to roughly 10 percent of their shares in secondary sales tied to funding rounds, a way of proving the stock is real money rather than paper (52:44).

What ties the hour together is a kind of structural honesty. Bernhardsson is not promising a return to the fat margins of an earlier software era, and he is not promising that any one layer of the AI stack, including his own, is safe from specialization or competition. What he is betting on is scale: a market he believes is still being badly underestimated, growing into more sub-markets than anyone, himself included, predicted a few years ago. The open-air market he described at GTC may not stay chaotic forever. But by his own account, nobody, not Jensen Huang, not Bernhardsson, knows exactly when the chaos ends.