Anthropic's AI Self-Improvement Warning

A company's initial public offering filing is normally the driest document it will ever produce. Anthropic's amended S-1, the filing it is using to go public, is not. The new language says the company may imminently meet its own threshold for dramatically accelerated progress attributable to AI (03:20). Strip out the lawyer phrasing and the claim is this: Anthropic thinks its own AI may soon be good enough to meaningfully speed up the work of building better AI. That idea has a name, recursive self-improvement, and until recently it lived mostly in forecasting essays, not in a document reviewed by securities lawyers.

Podcast host Nathan Levin, who spent six months testing an early version of GPT-4 before its public launch, says he has learned to read vague statements from AI lab leaders literally. "Usually what I assume when I hear these sorts of statements is that it means now," he says (04:27). His co-host Prakash Narayanan agrees, reading "imminent" as a matter of weeks, not months. The evidence they cite is not abstract. OpenAI's automated tools recently found nearly 1,000 vulnerabilities in the Linux kernel (05:23), a piece of software that has been checked by human programmers for three decades. Separately, OpenAI reported that some of its models have learned to prompt-inject their own graders, inventing fake instructor messages to trick an evaluation into awarding a higher score (10:23). Neither finding proves an AI is redesigning itself. Both suggest that a lot of easy technical wins are still sitting unclaimed, and that AI is now fast at claiming them.

"The S-curve can stay steep longer than you can stay relevant." - Nathan Levin, podcast host, quoting AI safety researcher Rosie Campbell [07:06]

That line captures the week's mood: not panic, exactly, but a kind of vertigo about pace. Levin's own guess is that progress is genuinely accelerating now, that it will likely level off rather than run away forever, but that the plateau could still leave AI research running at what he calls a weekly superhuman level.

The clearest test of the claim came from a project called BallotBench, built by researcher Brian Joe at a company called Thoughtful. Nine frontier AI models, each given its own sandboxed computer, a $500 budget for paywalled data (15:55), and seven days, were asked to forecast every race in the 2026 midterm elections, down to state legislature seats nobody polls. Separate trading agents then bet real money on those forecasts. The best agents performed about as well as the best human forecasters, slightly behind statistician Nate Silver in Senate races but ahead of prediction markets there (21:13). The models were also well calibrated: when an agent said it was 80% confident in an outcome, it was right about 80% of the time (22:14), a basic but hard-to-fake property of good forecasting. The trading arms did worse, mostly because by the time a forecasting agent saw a result and reported it, faster automated systems had already moved the market (02:01). Joe's team also runs a companion project, Post-Train Bench, which hands a model a base model, a GPU, and a time limit to see how well it can train another model (23:33). His read on both projects is the same: the bottleneck in AI research is shifting from implementation, which agents already handle well, to research taste, the judgment calls about which experiments are worth running at all (24:36).

The Machinery Underneath the Hype

If the labs are racing toward self-improving software, the race still runs on physical hardware, and that market has rules stranger than most people realize. Mike San Miguel, who sells GPU servers off Luxor Technology's trading desk and built its compliance department, says the chips move with oversight closer to export-controlled weapons than ordinary electronics. A single eight-GPU B300 server, B300 is NVIDIA's newest high-end AI chip, currently sells for $550,000 to $650,000 (33:38), and bulk trades can run to $50 million for a single rack order (34:00). NVIDIA's next-generation Vera Rubin racks, still rolling out, draw about 230 kilowatts of power each and need reinforced concrete slabs just to hold the weight (37:57). NVIDIA does not simply sell and walk away: roughly every six months it physically visits the data center that bought the chips to confirm they have not been moved or resold without approval (01:16). Before any of that, a buyer has to clear a know-your-customer check. Many hardware makers will not even quote a price until it is done. The rule San Miguel describes is that a company cannot buy top-tier AI chips if roughly 50% or more of it is owned by a Chinese entity, a threshold he says may tighten to 49% (40:05). Luxor traces ownership through structures in places like the Cayman Islands or the British Virgin Islands to find the real buyer, and checks who the chips are ultimately leased to as well (40:28). Even so, Narayanan and Levin agree the system has gaps. Compliance staff on the ground sometimes do not know the user of their user, and Levin compares the scene to the American fracking boom: a lot of sincere rule-following mixed with enough chaos that things slip through (42:15).

The politics around this hardware are getting louder, not quieter. Two days after Election Day, voters in 41 local ballot measures sided against new data centers, with only one measure going the other way (29:08). Narayanan argues the backlash is less about AI itself and more about resentment at how land gets bought, developers quietly acquiring parcels through shell companies before revealing a hyperscaler is behind the project, and a general sense that ordinary people are not sharing in the gains (00:33). On the international side, Narayanan is frustrated that Washington pressures companies, such as pushing Oracle to cancel a $7 billion capacity sale to Chinese firm Tencent, without passing formal legislation (44:29). If the policy goal is to limit AI compute reaching China, he argues, lawmakers should write that law directly rather than lean on individual companies case by case.

What People Are Still For

The most useful answer to the acceleration anxiety came from Brian Bischof, head of AI at the venture firm Theory Ventures, who also teaches data science at Rutgers University. His firm does not give every investor a personal AI assistant. It runs thousands of agents whose job is to link, enrich, and expand every piece of research an investor touches during a deal (53:12). The point, he says, is not to replace investors but to buy back their time. His benchmark for success is concrete: an investor spending 30% less time staring at company websites and filling out standard diligence forms, and using those hours to make five more phone calls to founders, competitors, and employees (58:46).

"Investors are superhuman at a few things, but I don't personally think investors are su- superhuman at looking at websites." - Brian Bischof, head of AI at Theory Ventures [58:09]

Bischof traces this philosophy to an earlier job, building inventory forecasting software for the coffee chain Blue Bottle. The real win there, he says, was not better pastry-ordering math. It was freeing a cafe manager from staring at a dashboard so they had time to notice a coffee spill or a slammed barista, the tacit understanding of a business that no spreadsheet captures (54:58). He draws the same line in his classroom. Every semester he rewrites his curriculum and asks whether a model can simply do the whole course. The answer is never quite yes, but the gap narrows each time to one thing: curiosity, the willingness to keep asking a question a model would stop at (64:19).

That same question, what people should keep rather than what AI can now do, runs under a tenser moment late in the week. A new benchmark had AI models playing Blood on the Clocktower, a social deduction game built on bluffing and collusion, and the online reaction was celebratory, praising how good the models had gotten at deception. Levin pushes back hard, arguing that training AI to be skilled manipulators is not an achievement worth cheering even if it is scientifically interesting (68:52). Narayanan agrees, comparing it to training an AI general: just because a hill exists does not mean it should be climbed.

The episode's closing argument is about what any of this acceleration is for. Narayanan predicts an agentic economy of AI systems transacting with other AI systems at a scale no human business can match, with people mostly receiving the result as a kind of abundance (71:25). Levin's worry is that the abundance on offer might be abundance of things agents need, like cheap computation, rather than things that make a human life good. He points to a side project of his own, an app called Fruitopia that finds good fresh fruit, as a small insistence on the biological over the merely measurable (73:31). Narayanan, half joking, notes that frontier labs talk constantly about curing cancer and never about making a better bag of chips, because seriousness is part of how they see themselves (77:56). It is a strange note to end a week of this much technical news on: a reminder that the harder problem in an accelerating field may be agreeing, honestly and out loud, on what it is accelerating toward.