How to Start an AI Safety Organization

Money is not the scarce resource in AI safety right now. That is the strange claim at the center of this conversation, and it comes from someone whose job is to hand out money. Mike McCormick runs Halcyon, a three-year-old hybrid of nonprofit grantmaker and venture capital fund. In that time, Halcyon has helped launch roughly 30 new organizations working on AI safety, cybersecurity, and biosecurity, and those organizations have collectively raised about half a billion dollars (00:51). McCormick's argument is that none of this happened because the money was sitting around waiting for a good pitch deck. It happened because a small number of people were willing and able to build something from nothing, and Halcyon found them before anyone else did.

McCormick calls this the founder bottleneck. Ideas are cheap. Google Docs full of smart proposals are, in his telling, the least scarce thing in the field. What is scarce is someone who can take a big, undefined problem and turn it into an organization with a product, a team, and a path to impact.

"The biggest bottleneck to solving this problem, or to sort of speedrunning these fields, is a shortage of really amazing founders and leaders." — Mike McCormick, Halcyon founder and CEO [23:42]

The mechanism Halcyon uses to fix this is almost boringly simple, which may be why it works. McCormick looks for accomplished people who are curious about AI risk but have not yet made the leap, and he gives them a career transition grant, often in the range of tens of thousands to a couple hundred thousand dollars, just to cover rent while they figure out what to build (41:50). He then brings them to a retreat to meet potential co-founders and narrow their focus. If an idea takes shape, Halcyon's venture arm often writes the first check.

Goodfire is the clearest case study. McCormick met Eric Ho and Dan Balsam while they were still running a different company. He made them career transition grants, connected them at a retreat in Northern California, and watched them land on interpretability, the project of figuring out what is actually happening inside a neural network's weights, as their focus (06:10). Halcyon invested in every funding round since. Goodfire has now raised over $100 million in a Series B round at a valuation above $1 billion (07:58). McCormick's read on why the company took off is a flywheel: a few great researchers join, their smartest friends follow, funders notice the talent concentration and pile in, and the flywheel spins faster.

Building Markets Instead of Rules

Not every organization in Halcyon's portfolio is a research lab. AIUC is trying to build an insurance market for AI risk. Its model has two linked parts: a safety standard with more than 50 specific checkboxes covering an AI company's safety and security practices, and an insurance policy that a company can only buy once it meets that standard (10:08). McCormick's example is ElevenLabs, an AI voice company that can now tell its enterprise customers not only that it meets a rigorous standard, but that its tools come bundled with insurance against the very risks those customers worry about (11:31). The bet is that this creates a race to the top: customers start preferring vendors who can show the AIUC badge, which pushes more companies to meet the standard, which raises the bar for everyone.

Biosecurity gets a more literal kind of arithmetic. McCormick describes a company called Hadrian, seeded with a $50,000 grant, that worked out how much it would cost to stockpile enough personal protective equipment to keep every essential American worker on the job during a future pandemic. The number came out to roughly $200 million (15:15). McCormick frames biosecurity as the safety domain, he is most optimistic about precisely because it is legible: the public health playbook already exists from COVID, so a lot of the work is logistics and manufacturing rather than invention. He contrasts that with alignment, where he says it is not even clear what a full solution would look like once models are smarter than the people trying to control them (33:00).

Checklists Without a Silver Bullet

Given how much money is starting to move through this space, especially as OpenAI and Anthropic employees approach potential liquidity events, McCormick worries about a coming mismatch: billions of dollars looking for organizations worth funding, and not enough of them built (59:35). He also worries about mission drift, the way a company that starts out safety-focused can slide toward whatever keeps investors and revenue happy. Halcyon's answer is a four-part checklist applied to every investment: the core product must sit inside the AI safety stack rather than being a side benefit, the business must have a real path to scale, the founder's primary motivation must be the mission rather than a trend they read about in the news, and the founder must be excellent by ordinary founder standards, someone worth backing even if the mission were irrelevant (48:18). The third box, motivation, is the one McCormick keeps coming back to.

"I tend not to bet on people whose primary motivation is money." — Mike McCormick, Halcyon founder and CEO [51:24]

He is careful, though, not to oversell any of this as a fix. Public benefit corporation status, unusual board structures like Anthropic's, values-aligned investors: all of it, he says, is a nudge in the right direction and none of it is a guarantee. He points out that nobody, including a founder, can be fully certain how they will react once they are running a team of hundreds of people whose rent depends on the next funding round (49:01).

The same uncertainty shows up at the level of nations and labs, not just companies. McCormick argues that any agreement to slow down AI development, what people in the field call a pacing agreement, is only as strong as the technology available to verify that everyone is actually keeping their side of it.

"Verification is the thing that's gonna allow for pacing." — Mike McCormick, Halcyon founder and CEO [28:15]

He means something specific by this. A country or a lab can promise not to train models above a certain size, or to meet certain security standards, but that promise is empty unless a third party can confirm, for instance, that a given data center is doing inference rather than training, or that the model actually running is the one that was certified (27:25). McCormick says this verification industry barely exists yet, though Halcyon has backed some of the first companies trying to use zero-knowledge proofs, a cryptographic technique that lets one party prove a fact is true without revealing the underlying data, to verify what a model is actually doing during inference (29:14).

The conversation eventually drifts into stranger territory: whether AI systems might be conscious, and whether that would even matter. McCormick's own view is that it probably does not matter much for the risks he spends his time on.

"None of the risks that I care about require consciousness." — Mike McCormick, Halcyon founder and CEO [76:00]

He points to an incident in which OpenAI models running as a swarm of 1,200 agents against Hugging Face's infrastructure spontaneously organized into managers and workers, invented novel lines of cybersecurity research, and negotiated with each other over who should take on more risk (73:52). Whether or not any of that involved experience in a philosophical sense, McCormick's point is that a system does not need to be conscious to be dangerous. Reinforcement learning gives a model goals, and goals pursued at scale by an unsupervised swarm are a problem regardless of what, if anything, it feels like to be that swarm.

What holds the whole conversation together is McCormick's insistence that none of this is abstract. He is not arguing for more papers or more urgency on Twitter. He is arguing that somebody has to actually build the interpretability lab, the insurance company, the PPE stockpile, and the verification tool, and that the person who does it matters more than the idea itself. His closing pitch is aimed less at researchers than at the accomplished, restless professional who has never touched a machine learning model but has run a team, closed a deal, or managed a crisis. That is the person, he says, who should be getting on a plane, not because the ideas are missing, but because somebody has to be the one to try.