Fractile's Walter Goodwin on AI's Memory Wall

Ask a large language model a hard question and watch the words crawl out one at a time. The instinct is to think the model is straining to think. It isn't. According to Walter Goodwin, founder and CEO of chip startup Fractile, the slowdown has almost nothing to do with intelligence. It is a plumbing problem. "The reason that if you take a very large frontier model, it might generate at 80 tokens per second, 80 words per second, is going to be not because of the limited maths capability on the chip, but because you don't have nearly enough bandwidth to that memory," he says (18:14). Every word a model produces requires reloading its entire set of weights, the billions of numbers that define what it knows, from memory into the processor. That trip, not the arithmetic, is the holdup.

Fractile just raised a $720 million Series C, co-led by Thrive, Founders Fund, Lightspeed, and Redpoint, at a $7 billion valuation (00:23). The company makes chips, but Goodwin insists it is also, in effect, a frontier AI company, because designing hardware for models that don't exist yet requires guessing where the models are headed. That bet concentrates on one specific physics problem: the widening gap between how fast chips can calculate and how fast they can fetch the data needed to calculate with.

The Wall Nobody Can Wire Around

For 25 years, chipmakers rode Moore's Law, the long trend of packing more transistors onto silicon, to scale raw computing power by roughly a millionfold. Memory bandwidth, the rate at which data moves between storage and processor, scaled only 40-fold in the same stretch (17:24). That is a 25,000-times gap, and it is largely a function of physics rather than engineering cleverness. Data moves over copper wires at the edge of a chip, in a space engineers call the "beachfront." There are only so many wires that fit.

"We've scaled at about a millionfold in the last 25 years on these chips. Bandwidth, by contrast, is kind of much more intrinsically physics limited. We've scaled at about 40-fold in the same time." — Walter Goodwin [17:24]

There are two kinds of memory at play. SRAM sits directly on the same silicon as the processor, so it is fast, but it is also roughly ten times less dense than DRAM per unit of area (20:02), meaning a chip can only hold a small amount of it. DRAM is cheap and can hold far more data, but it lives on separate silicon and has to send information across that narrow beachfront, which caps its speed. Chipmakers have had to choose: fast and small, or big and slow.

Fractile's answer is to stop choosing. The company's approach stacks memory directly on top of the processing logic instead of placing it beside it (21:09). Goodwin's metaphor: instead of a few beachfront properties, imagine an archipelago where every unit has water access, with millions of connections running straight down through the chip rather than sideways across a crowded edge. The goal is a chip with the bandwidth of SRAM and the storage capacity of DRAM at once, something Goodwin calls a long-standing technical hinterland the Fractile team has worked on for years, now aimed squarely at a new problem.

Why does this matter beyond chip design trivia? Because the economics of AI have quietly flipped. Training a model, the expensive process of teaching it everything it knows, is a one-time cost. Inference, actually running the model to answer questions or solve problems, is where the value gets delivered or doesn't. "Training becomes a fixed cost. And the deployment, you know, inference is just the use of these models to do useful things. It's the only part that matters," Goodwin says (42:28). He compares it to an Olympic sprinter who trains for years and then skips the race.

That shift has accelerated because of what Goodwin calls test-time compute scaling: letting a model keep thinking, for minutes, hours, or days, rather than spitting out a single quick answer. One example he cites is a frontier model working on a Navier-Stokes fluid dynamics problem tied to a Millennium Prize challenge. The solution took roughly 80 hours of execution across a swarm of 10,000 parallel agents (43:26). That is a staggering amount of inference compute, and it means the chip that can run inference faster and cheaper, not just the one that trains the biggest model, increasingly decides what gets built. "Inference compute is the axis that we have... it's the knob that we have to turn to get greater capability," Goodwin says (45:23).

Chips, Clouds, and the Case Against Mixing Vendors

If inference is where the money is, who captures it? Goodwin draws a line between "neo clouds," the newer data-center operators that rent out GPU capacity, and chipmakers like Fractile. He is blunt about the neo cloud layer: "One neo cloud that is competent is much like another neo cloud that is competent. The reason it's so exciting today is we don't yet know which ones are competent" (10:46). In other words, that market is heading toward commoditization, which is why many neo clouds are racing to build their own inference software or even their own chips, to avoid becoming interchangeable.

Goodwin is more cautious about today's trend toward mixing different chip types inside one data center, a practice Nvidia itself is pursuing after acquiring the fast-inference chip company Groq. The idea is to split the work: a Groq-style chip with fast but limited memory handles the part of a model's workload shared across many users, while GPUs with more memory capacity handle the part unique to each user's context. The problem is that no one knows in advance how to size the mix. If context lengths grow or models get bigger, the ratio of chip types bought a year earlier can leave expensive capacity sitting idle. "That's unacceptable," Goodwin says of this stranded-capacity risk (57:09), because data centers are planned and paid for years in advance; you can't swap chip types partway through. He argues a single chip that combines high bandwidth and high capacity, which is what Fractile is trying to build, removes the need for that orchestration entirely.

Asked where he would place a bet across the AI stack, Goodwin says he would rather be a specialized inference provider, a company that runs other people's models efficiently, than a neo cloud, because inference providers retain the flexibility to adopt new chips and still carve out defensible niches (61:26). He is more skeptical of pure software orchestration layers that exist only to route workloads across many different chip types, arguing that need could collapse once better unified chips exist (54:16).

The conversation closes on a question that hangs over the entire AI buildout: is this 2000 all over again? Goodwin's answer is that the market has priced in too much fear of a repeat. He points out that Nvidia today trades at a lower forward earnings multiple than networking giant Cisco did after the dot-com crash (69:59), and unlike the fiber-optic overbuild of that era, today's GPUs are running at full utilization rather than sitting dark. He also notes that AI chips typically need replacing within six or seven years at most, and often sooner (70:23), which means the current spending wave isn't a one-time purchase but a recurring one. Whether that argument holds depends on whether demand for inference keeps climbing as fast as Goodwin expects. His own company's survival depends on betting that it will.