AI's Memory Wall: Thomas Sohmers on Inference Costs

A single In-N-Out burger restaurant, according to one guest on this week's 20VC, uses more water than the largest data center in the United States. It is the kind of fact that sounds too tidy to be true, which may be why Thomas Sohmers, co-founder and chairman of the AI chip startup Positron AI, leads with it (22:39). His point is not really about burgers or water. It is about how easily a narrative takes hold before anyone checks the numbers, and how that narrative, in his view, is doing real damage to America's ability to build the infrastructure an AI economy needs.

Positron builds hardware for AI inference, the stage where a trained model actually answers a question or writes a line of code. Sohmers has spent a decade in semiconductors, and his company just raised an $875 million round at a $5 billion valuation, backed by investors including Atreides' Gavin Baker. The conversation with host Harry Stebbings ranges from chip design to sovereign debt, but underneath it sits one idea that reframes almost everything else: the hardest problem in AI right now is not making models smarter. It is remembering things fast enough to keep up with them.

The Memory Wall

Training a large language model is what engineers call compute bound. The dataset already exists, so a system can chew through billions of examples in parallel, and the bottleneck is simply how many floating point operations per second, or flops, it can throw at the problem (04:45). Inference is different. When a model generates text, it produces one token, one small chunk of a word, at a time, and it cannot see ahead to what comes next. Every token requires the model to read through its full set of weights, the billions of learned parameters that define what it knows.

"Inference is heavily, heavily memory bound due to the fact that basically for every single token that's generated, every little bit of output, that requires going through the weights." (Thomas Sohmers, [05:47])

That distinction explains a strange gap in the hardware world. Between 2014 and 2024, Nvidia's GPUs got about 120 times faster at raw computation. Over the same decade, memory bandwidth, the rate at which a chip pulls data out of memory, improved only 17 times (07:59). Part of the reason is physical: SRAM, the fast on-chip memory that sits closest to the processor, is built from a six-transistor cell that has barely changed in three decades, even as other transistors kept shrinking under Moore's Law (09:00). Part of the reason is historical: through the 2010s the dominant AI workloads were convolutional neural networks, which are compute hungry but not especially memory hungry. It was the arrival of the transformer, the architecture behind GPT-3 and everything since, that turned memory into the real constraint.

Why Cheap Tokens Make Rich Companies

The clearest illustration of that constraint is KV caching, short for key-value caching. When a model processes a prompt, it generates two internal matrices for each token, called keys and values, that capture what it has already read. Early transformer systems recomputed those matrices from scratch for every new token in a conversation. Storing them instead costs memory, but it saves an enormous amount of repeated computation, because the compute cost of attention grows quadratically with the length of a conversation while the cost of storing a cache grows only in a straight line (35:01).

The economic consequence is large. Sohmers puts the savings at roughly a thousandfold: processing a token already sitting in the cache costs about one one-thousandth of what it costs to generate that token fresh (11:37). AI companies charge less for a cached token than for a new one, but the margin on that cached read is still enormous, which is part of why Anthropic is reported to run at 80 points of gross margin on its API business (12:02). Sohmers thinks that quietly undercuts a popular story, the idea that OpenAI and Anthropic are unprofitable ventures kept alive only by investor cash.

"It's absurd to me that the meme of OpenAI, Anthropic, et cetera, are just burning cash... if they stop training, they'd be massively profitable overnight." (Thomas Sohmers, [12:49])

Caching has trade-offs. In real agentic workloads, where a coding assistant runs through dozens of steps, the analytics firm SemiAnalysis found that about 96% of all tokens processed were pulled from cache rather than freshly generated (39:35). That is good for margins, but storing and retrieving those caches, some of which run larger than the model's own weights once a session gets long, becomes its own engineering problem, one Positron is building chips to solve by packing far more memory onto each device than Nvidia's current lineup allows (56:30).

Sohmers carries this technical argument into a political one. The same memory and energy constraints that shape chip design also shape the fight over data centers, and he thinks the public has been sold a false picture of the trade-offs. Data centers, he argues, mostly run closed loop liquid cooling systems that use far less water than critics claim, and new facilities typically bring their own generation capacity rather than draining power meant for homes (25:46). He goes further, framing the sudden bipartisan hostility to data center construction as something close to an engineered narrative.

"I think that is almost entirely a Chinese psyop. A single In-N-Out uses more water than the largest data centers in the United States." (Thomas Sohmers, [21:35])

Whether or not one accepts the psyop framing, his underlying worry is about competitiveness. China, he argues, faces none of the local political friction now slowing American projects, and can relocate people and reroute power to serve state priorities. His preferred fix is not less regulation everywhere, but more use of the country's federal desert land for data centers and the clean generation, solar, geothermal, even nuclear, that could power them without displacing anyone (24:39).

That same distrust of concentrated control shapes his view of AI safety debates. Sohmers says his biggest fear is not a runaway superintelligence but a world where the ability to run large models gets restricted to a handful of companies or governments, leaving everyone else in what he calls a form of serfdom (14:24). He is skeptical that calls from lab leaders like Anthropic CEO Dario Amodei to slow down, or "pace the frontier," are purely about safety, suggesting some of it doubles as a convenient way to justify lower training costs ahead of an IPO and to lock in the current leaders' position before rivals catch up (12:49).

He is equally unbothered by warnings about AI company debt loads. Asked about financing behind the AI buildout, Sohmers redirects the worry upward, toward the sovereign debt of the countries backing that buildout rather than the companies themselves. He says he trusts Oracle's ability to execute more than he trusts the United States government's balance sheet, since a government can print currency and use force to raise revenue in ways a company cannot (30:33). His real economic fear is a currency and bond market crisis, not a shortfall in AI company revenue.

By the end of the conversation, the technical and the speculative blur together. Sohmers describes an internal model he calls GPT-6 Astra as effectively AGI, artificial general intelligence, meaning a system with broad, human-level capability across tasks, comparing his first day with it to his first night with ChatGPT in November 2022 (48:51). He says it compressed a chip design task that would normally take two to three weeks into about 50 hours (53:06), and that it correctly finds a hidden piece of information buried in a long document 95% of the time, versus 70% for the model that preceded it weeks earlier (60:06). He also notes that the price of a million tokens has fallen from about $60 five years ago to under $1 today (60:33), but argues the price drop understates the real story: today's cheaper token, he says, is roughly a hundred times more capable than the sixty-dollar token of five years ago.

That is the thread running through the whole hour: the numbers people quote about AI, whether water usage, token prices, or company margins, rarely mean what they appear to mean on the surface. Sohmers spends much of the conversation trying to get underneath them, to the physical and economic mechanics that actually decide what gets built, who profits, and who is left arguing with a chart that never told the full story.