MongoDB AI Agent Memory & Retrieval

In June of 1970, an IBM researcher named E.F. Codd wrote a paper that would define how computers stored information for the next three decades. At the time, storage was the most expensive part of any computer system, more costly than memory or processing power. So Codd's design, which became SQL, split data into many small tables to avoid ever storing the same fact twice. A customer's address might live in one table, the customer's name in another, and a third table would link them, purely to save disk space (07:25). That single constraint, expensive storage, shaped software architecture for a generation.

Pete Johnson, MongoDB's field CTO of AI and a 30-year veteran of enterprise software, opens his account of database history there because he thinks the pattern still applies. By 2007, when the first commit to MongoDB was written, storage was cheap and disk was no longer the bottleneck. The internet, cloud computing, and the first iPhone had changed what applications needed to do. Suddenly they had to run continuously, not on a nine-to-five schedule, and developers had far less patience for wrestling with rigid table structures. As Johnson puts it, the scarce resource had moved.

"The scarce resource in that ecosystem became time." — Pete Johnson [08:49]

MongoDB's answer was to store data as JSON, a text format that mirrors how programming languages already represent objects, saved on disk in a binary form called BSON. Instead of splitting a customer's name and address across three tables, you keep them together in one document. You duplicate a bit of data, but you get it back in a single disk read instead of three (10:40). Johnson is careful to correct a common misunderstanding here: MongoDB is not schemaless. It is schema-flexible, meaning different documents in the same collection can have different shapes, and you can add new fields without the cascading rewrites that a SQL migration usually requires (21:36).

"It's not that we're schemaless, it's that we're schema-flexible, that you can change the schema more easily over time." — Pete Johnson [21:36]

Retrieval Is the Real Cost Center

That history matters now because AI has revived an old argument in a new form. When GPT-4-class models had tiny context windows, the space where a language model reads the text it is given, developers were forced to retrieve only the most relevant snippets of company data and feed them in, a technique called RAG, short for retrieval-augmented generation. When million-token context windows arrived, plenty of people declared RAG dead: why bother selecting anything if you can just paste in everything? Johnson says that phase ended badly. Uber reportedly burned through its entire 2026 token budget in just thirteen weeks (16:38), an outcome that became a cautionary tale across the industry. Feeding a model a million tokens every time is not just slow, it is expensive, and Johnson argues it does not even produce better answers, because the middle of a long context window tends to get ignored while the first and last roughly seven thousand tokens do the real work (56:54).

So retrieval quality, not sheer context size, is what Johnson calls the biggest lever on cost-adjusted agent performance (13:15). MongoDB's own vector search product grew in stages that trace this shift. In 2020 the company added lexical search, essentially keyword matching, because customers were already running separate search servers next to their databases (23:27). In 2023 it added vector search: an embedding model converts text, images, or audio into an array of numbers called a vector, and MongoDB simply stores that array as another attribute on the same document (25:33). Because the schema is flexible, developers can combine keyword search, vector search, and traditional filters, such as "only books published after 2000," in one query (27:17). This year MongoDB shipped Rank Fusion and Score Fusion, aggregation stages that let a single API call run both lexical and vector search and merge the results automatically, instead of requiring developers to make two calls and stitch the results together themselves (30:42).

Underneath all of this sits a genuinely hard engineering problem: chunking, the process of breaking a long document into pieces small enough to embed and search individually. Chunk too small, like a single sentence, and you lose the surrounding context. Chunk too large, like a whole chapter, and storage costs rise while retrieval precision falls, because the vector now represents too many different ideas at once (39:19). MongoDB's 2025 acquisition of embedding company Voyage AI, reportedly valued around 220 million dollars (77:03), was aimed partly at this problem. Voyage's contextualized chunking sends both a small target chunk and a second string carrying its surrounding context, and returns one vector that balances the two, which Johnson says lets developers get better retrieval quality even with smaller chunks, without running the usual trial-and-error cycle of testing chunk sizes by hand (41:23). A related feature, Matryoshka reasoning, named for Russian nesting dolls, lets developers cut a large embedding down to fewer dimensions, the individual numbers in the vector, without re-processing the entire dataset, because the values are ordered so the extra ones can simply be trimmed off the end (44:04).

These tools matter mostly at scale. Johnson pegs the point where vector database performance starts to require real engineering attention at around 100,000 vectors (49:01). Below that, most setups work fine regardless of vendor choice. He also pushes back on the idea that embedding models are interchangeable commodities: on the Hugging Face retrieval benchmark called RTab, Voyage AI's models reportedly outperform competitors by up to 14 percent (49:30), a gap he says is large enough in some cases to be the difference between a hallucination and a correct answer.

Memory Needs a Loop, and Forgetting Is the Hard Part

Retrieval solves the problem of injecting outside data into a model that was never trained on it. Memory solves a different problem: giving an agent continuity across many separate calls, since every call to a language model still starts with a technically blank context window. Early memory systems, Johnson says, simply crammed every message from a session, and then every session from the last few days, into that window. This produced the token-maxing costs already described, plus a quieter failure mode: useful information buried in the middle of a huge context window gets effectively ignored by the model (56:54). More advanced systems now practice what Johnson calls taxonomic memory, selectively injecting only the five or so vertical-specific terms relevant to the current task instead of a full glossary of a hundred (57:39).

Johnson frames the overall discipline as a four-step loop: write, change, recall, and forget (62:50). Writing and recalling are relatively well understood. Changing a memory as facts shift is harder. Forgetting, deciding what should decay or disappear, is the part nobody has solved yet, partly because so little time has passed to figure it out.

"We've been building databases for 60 years. We've been building agents for, like, 18 months, man." — Pete Johnson [60:07]

That humility carries into his advice on enterprise strategy. Johnson has visited seven countries this year talking to roughly a hundred companies about their AI plans, and he says the ones stuck in what he calls proof-of-concept purgatory usually picked the wrong problem, not the wrong tool. His filter: choose use cases where the company already has clean data and existing performance metrics, because a bad foundation gets worse, not better, once AI is layered on top.

"Things like bad data quality and bad security posture don't get solved by AI, they get amplified by AI." — Pete Johnson [68:54]

He also notes there is no equivalent yet of the LAMP stack, the standard set of tools that once made web development predictable, for building AI agents (71:12), which is why most Fortune 500 companies are still deploying AI on internal, employee-facing tasks with a human checking the output, rather than trusting it directly with customers.

Perhaps the most surprising finding from Johnson's travels has nothing to do with technology at all. He expected the United States to be furthest ahead. Instead, the two most sophisticated deployments he saw this year were in Mexico City and Sao Paulo (85:16), a reversal he attributes to infrastructure access spreading faster this cycle than in the cloud or mobile eras, when a missing data center or cell tower could leave an entire country behind. Fittingly, MongoDB's newest cost-saving trick works the same way, in miniature: its shared embedding spaces let a developer embed their full dataset with a large, paid Voyage model, then run all their development-time queries against a free, open-weight "Nano" version that shares the same embedding space, cutting token costs during the build phase to nearly zero (87:47). Access, in other words, keeps getting cheaper. What is expensive now, and worth arguing about, is judgment: which data to keep, which to trim, and when to let something go.