Harrison Chase on Why AI Agents Are So Hard to Build

Ask Harrison Chase, founder of the AI developer platform LangChain, what makes agents so hard to build, and he does not start with intelligence. He starts with behavior.

"The hard part about building agents is they don't behave how you want them to behave." — Harrison Chase [05:01]

An agent, in Chase's telling, is not a single question and answer. It is a large language model (LLM) running in a loop: calling tools, reading data, deciding what to do next, over and over, until a task is done (04:31). The model itself is a non-deterministic black box. Feed it two nearly identical prompts and it can produce two very different outputs (05:24). String a dozen of those calls together, as a real agent does, and small errors compound into what Chase calls cascading failures (05:24). The fix is not a smarter model. It is what Chase calls a "harness": the surrounding code that feeds the model data, calls its tools, stores its memory, and decides which model to use for which job (04:31). LangChain builds that harness layer, plus an observability product called LangSmith that tracks what agents actually do once deployed (04:31).

The harness has to survive the thing it was built for. Every time a frontier lab ships a better model, Chase says, LangChain has to delete parts of its own product, because a feature that took six months of engineering to hack together becomes a native model capability overnight (06:19). Early on, LangChain got models to call external tools by having them generate JSON and then parsing it by hand. Once OpenAI shipped native tool calling, that whole workaround became obsolete, and Chase says the team moved fast to drop it (07:45). The lesson he draws is that a harness should bet on what will always sit outside the model, like proprietary data and live API calls, and treat everything else as temporary scaffolding (08:20).

That is also why Chase insists a harness has to stay neutral between models. Over time an agent accumulates state: short-term and long-term memory, context from past interactions, learned facts about a user or a workflow (08:48). If that state is built in a way that only works with one model provider, a company gets locked in. It cannot adopt a better model when one appears, and it cannot threaten to leave when a provider raises prices.

"Being able to change models is very offensive and defensive." — Harrison Chase [09:48]

Why Narrow Beats General

Chase's clearest bet is on vertical agents: agents built for one domain, like law or sales or finance, rather than general-purpose assistants. His reasoning starts with verification. Someone has to judge whether an agent did a good job, and building that judgment, what the industry calls an "eval," is hard and domain-specific work (11:57). Chase thinks the people best positioned to write those evals are domain experts, which means the value in agents will flow to specialists who understand a field well enough to say what "good" looks like there (12:17).

"I'm massively bullish on vertical agents for different domains." — Harrison Chase [12:17]

His advice to someone building one of these agents is sequential: start with the context layer, meaning the data, memory, and tools an agent can access, because that is the cheapest lever to pull. Then tune the harness, the code that governs how the model uses that context. Only once there is real user volume does it make sense to post-train or fine-tune the underlying model itself (12:34). He points to Harvey, a legal AI company, and Ramp, a finance company, as examples of firms far enough along to be post-training open-weight models for their specific workloads, while being skeptical that many companies are training models entirely from scratch (13:13).

The Routing Fight: Harness or Gateway

One of the more technical disputes Chase wades into is where "model routing" should live: the decision, made on the fly, about which AI model should handle a given request. One camp builds this into a "gateway," a layer like Open Router that sits between an application and multiple models, letting any tool call any model through a single programming interface (16:46). The appeal is portability: a gateway can be dropped into any existing coding tool without changing its code.

Chase thinks the gateway approach is mostly fake intelligence dressed as routing. He argues real routing needs to know the task, the conversation history, and whether the AI's prompt cache, a mechanism that reduces cost by reusing previously processed text, is still warm, all information that lives in the harness rather than a thin API call (18:26).

"Routing, you have to be harness aware. You have to know what the task is." — Harrison Chase [18:44]

His concession is that gateways make sense for horizontal, general-purpose tasks, like plain chat or coding, where the model mix that works for everyone is roughly the same (20:21). But for a vertical, domain-specific agent, he says routing has to live inside the harness, because the right model for a legal research task and the right model for a sales email are simply different questions (20:56).

Memory is the piece Chase is least confident about. Storing facts in a database is not the hard part; most companies, he says, just dump memory into a single file (27:09). The hard part is the step before storage: deciding which observations from a user's behavior are worth remembering at all, a process he calls meta-reflection, and current models are still bad at it (27:33).

"The memory layer for agents is still massively unsolved." — note: paraphrased from [29:50]

LangChain tried to build a general-purpose memory product and killed it, because the team concluded memory is too domain-specific to solve once for everyone; what a personal assistant needs to remember looks nothing like what a legal agent needs to remember (31:03).

Underneath all of this sits a cost problem. LangChain spends roughly 30 percent of its engineering salary budget on coding tools, a number that has actually fallen in the last two months even as token usage rose (38:23). The team's fix is three steps: give engineers visibility into their own usage inside LangSmith, set spending budgets that require a conversation to raise, and then re-engineer the harness itself, for example by keeping the prompt cache warm, to cut costs structurally (38:38). The stakes of getting this wrong are concrete: one misconfigured plugin, repeatedly sending full webpage screenshots back to an agent, burned $40,000 in a single week (40:35).

Zoom out, and Chase says he has been wrong in one direction: he underestimated how big the demand for raw computing power to run AI models, known as inference, would get (46:50). Every provider he can think of, from Fireworks to AWS to smaller players, is finding buyers. On the venture side, he is equally direct about the value of a brand-name investor. He describes Benchmark, which backed LangChain's earliest rounds, as having functioned almost like a third co-founder, helping with recruiting, positioning, and company-building decisions well beyond writing a check, while describing Sequoia, which joined a few months later, as a tremendous partner with a different, more operational style, more like a machine that helps with everything from recruiting to launches (53:21). His advice to founders choosing between a slightly higher valuation and a top-tier firm is unambiguous: take the discount.

Asked what he knows now that he wishes he had known at the start in 2022, Chase does not mention a market size or a competitor. He says he wishes he had known what the harness layer would eventually look like, so LangChain could have built it sooner (57:24). It is a fitting last word from someone whose whole business rests on taming a technology that keeps outrunning the scaffolding built to contain it.