Parag Agrawal on AI Agent Web Search

Type a question into Google and something quiet happens: a wait of half a second, a page of ten blue links, a human eye scanning them in a rough zigzag before clicking one. That pattern, repeated billions of times a day, is the invisible architecture the entire web is built on. Parag Agrawal, the former CEO of Twitter and now founder of a search infrastructure company called Parallel, thinks that architecture is about to be obsolete. Not because people will stop searching, but because something else is about to start searching at a scale people never approached: agents, the semi-autonomous AI programs that book flights, monitor company registries, and write code on a person's behalf.

"Literally, the first genesis of the company was the statement that agents will use the web 1,000X more than humans," Agrawal says. "Hence, new tech is needed and new business models are needed" (04:16). It is worth sitting with that number. A thousand times more traffic does not mean scaling up the same servers. It means the entire stack, built over two decades for a slow, patient, curious human, has to be rebuilt for a customer that behaves nothing like one.

Consider how differently the two customers behave. A human types three underspecified keywords and tolerates up to a second of delay before getting impatient (05:24). An agent, by contrast, states a full sentence describing exactly what it needs, and its patience depends entirely on the job. A voice agent answering a spoken question needs a web search result inside 100 milliseconds, because a human is waiting on the other end of the call and will notice any pause (05:46). A background agent doing overnight research, on the other hand, is happy to wait five seconds if that buys a better answer (09:33). Building one search system for both is, in Agrawal's telling, a category error.

The deeper technical problem is compression. The web holds a few trillion documents. A model's usable attention, its context window, holds perhaps a thousand tokens (06:43). Getting from one number to the other requires a funnel: spend almost no computing power filtering out most of the web, a little more narrowing millions of pages down to ten thousand candidates, and progressively more computing power ranking those ten thousand down to the final one thousand tokens a model actually reads (07:09). "You're essentially allocating compute to web search in order to save compute on the model. That's roughly what's going on here," Agrawal explains (07:26). Spend too little on the search step and a cheap model gets a sloppy answer; spend too little ahead of an expensive model and it wastes its costlier reasoning cleaning up a bad result.

That funnel is not needed equally everywhere. Coding tools, it turns out, barely touch the open web: web search shows up in only about 5 percent of coding prompts, because a model writing software mostly reads the codebase already in front of it (10:19). Fields built on external facts behave completely differently. Law and insurance underwriting are, in Agrawal's words, "extremely web search-heavy," because a lawyer or underwriter has to comb through case law, company records, and public facts before being confident nothing was missed (11:12). The amount of web search a given industry needs, in other words, is not a technology question so much as a description of how that industry actually thinks.

The New Customer of the Web

If agents become the web's dominant visitors, the web's oldest business model runs into trouble. Online publishing has survived for twenty five years by showing ads to human eyeballs. An agent reading a recipe or a product page never sees the banner ad next to it, and never clicks it. "Ads don't work with agents in their current form," Agrawal says flatly (23:38). "You need to find a replacement to the ads business model" (24:24). Amazon already shows the direction of travel: its advertising business is now bigger than its e-commerce business, which is one reason, Agrawal notes, that Amazon has so far refused to let outside shopping agents like Muse operate inside its store (20:56, 23:16).

Parallel's answer is to try to pay publishers directly. The company calculates, for each search, how much a given source actually improved the final answer, then pays that source a variable fee approximating what an ad click on that page would have been worth (23:55, 28:02). The logic, Agrawal argues, is not charity but incentive alignment: a content owner that is not compensated when agents read its pages has every reason to block agents outright, the way Amazon has. He is betting this becomes less a story of a shrinking pie carved differently, as happened when streaming remade the music business, and more a story of a growing pie, since a thousand-fold increase in web usage means far more total value moving through the system even if the share going to any one page changes (25:33).

Underneath all of this sits a pricing anomaly Agrawal thinks the market has not noticed yet. Today's roughly ten dollars per thousand web searches is not a real cost of computing power; it descends from Google's historical advertising rates for human traffic (35:01). Parallel says it already delivers equivalent quality for about one dollar per thousand searches (36:13), and expects a further tenfold drop, to roughly ten cents, within three years (36:36). If that holds, the cost of letting an agent look something up approaches zero just as the volume of lookups increases by three orders of magnitude, a combination that reshapes who can afford to run agents at all.

From Pull to Push

The other shift Agrawal describes is stranger, and in some ways more consequential: the web moving from something people pull information out of to something that pushes information at them. Today's search, whether by a person or an agent, is triggered by someone asking a question. Harry Stebbings' own use of a personal assistant called Instinct includes an alert that scans every European company registry for newly founded companies with a founder under 25 (37:19). Doing that by asking an agent to wake up and search every six hours is wasteful, since almost nothing changes hour to hour. Parallel instead built what it calls a Monitor API: a system that crawls the web continuously and only spends meaningful compute when it detects an actual change, cutting the compute cost of that kind of standing question by ten to fifty times (38:19, 39:53).

That capability raises an obvious worry: agents that act on their own initiative, triggered by events rather than by a person's direct request, are also agents that can misbehave in ways nobody anticipated. Agrawal is careful here, distinguishing between a model still being trained through reinforcement learning, when most reported hacking incidents have occurred, and a model that has been fully aligned and shipped to users, which he says causes far fewer such incidents (41:50). He does not pretend the problem is solved. "These are stochastic systems," he says, and a stochastic system, one whose outputs vary rather than repeat identically, can never be guaranteed 100 percent safe (43:38). His view is that the responsibility sits with the labs building these models to close as many of those gaps as possible before release, and that treating a successful hack as a badge of honor obscures a more useful, more embarrassing lesson: the model was powerful enough to be dangerous, and the guardrails around it were not strong enough to stop it (44:03).

Agrawal served as Twitter's CEO through Elon Musk's acquisition of the company, working alongside Musk during the takeover. Asked what he took from that experience, he does not talk about chaos or conflict, though he says he disagreed with Musk often. He talks about expectations. "I think having unreasonable expectations of people is mostly a good thing," he says. "Most people don't understand what they're capable of" (49:47). It is a manager's lesson dressed as a philosophy of talent: most people, in his account, quietly lower the bar for themselves long before anyone else does it for them, and urgency, applied carefully, can be the thing that reveals what was possible all along.

It is also, in a way, the same argument he makes about the web itself. Search built for the patient, keyword-typing human of the last twenty five years was never going to survive an audience a thousand times larger and a thousand times less patient. The infrastructure has to be rebuilt, the business model has to be rebuilt, and the assumption that either can simply be scaled up rather than rethought is, on Agrawal's account, the one expectation worth abandoning first.