Bounded retrieval
Searching a lot of messages without showing the agent all of them.
This was a problem I ran into. I had a massive table of unstructured messages, and I wanted an agent to read through it and find something useful for whoever was asking. They could ask pretty much anything, so there was no fixed set of questions I could write queries for ahead of time. This is how I approached it, for anyone working on something similar.
The first thing I tried was the obvious one: give the agent direct access to the table, with every message as a raw row.
Honestly, it works. The best models are really good at this when they have real tools. With Python and a few other tools, a model will figure out how the threads join together and what people are talking about. I was using Opus 5 with thinking. The problem is that it's slow, and it's expensive because you need a model that smart. The agent does so much work just to understand the data that it takes forever to come back with an answer, and it burns through context the whole time. Even with subagents splitting up the work, it's a long, arduous process.
I tried subagents, and I tried cleaning up the data so the agent had more to work with. I also got really close to building a vector database.
Why not a vector database
A vector database would give you more, since it can search by meaning instead of exact words. But it takes time to build, and it opens up a whole new can of worms. I already had a flat table full of messages, which is pretty easy to work with. So the question I kept coming back to was whether I could get this working without one.
How would I search this?
At some point I stopped and asked myself: if I were the one searching this, how would I do it?
I wouldn't search it with regex. That's kind of stupid. I also wouldn't run SELECT * and read the whole thing, because that's useless to me as a person, and for an agent it's a lot of wasted context.
If you handed me a big flat table of Slack messages and SQL access, I'd start by checking how many messages even mention the thing I care about. Then I'd skim a few from different conversations. If one looked important, I'd pull up the whole thread, and if I thought I was missing something, I'd go read messages I hadn't seen yet. It's a needle in a haystack, and I'd never pick up the whole haystack to find it.
So I went piece by piece and turned each of those steps into a tool. The tools are just the way a person would approach the problem.
The tools
The project is an MCP server over a set of synthetic Slack-style messages (40,000 in the main test). It has five tools. One counts how many messages match and when they were sent, without returning any text. Another returns those counts plus a handful of excerpts, spread across different threads so the agent doesn't get eight copies of the same message. A third pulls messages the agent hasn't seen yet, in case the top results are missing something. A fourth opens the thread around a message the agent already found. The last one writes everything to a file for later, so it never has to go into the agent's context.
The tools don't have an order. The agent picks whichever one fits, the same way you would.
The server doesn't run a model. The agent decides what to search for, and the server runs the search, checks every match against the original text, and sends back only what fits in a budget, along with a note about what it left out. That's the core tradeoff: if you send too much, you drown the agent, and if you send too little, you throw away the answer. Every response is capped at 16 KiB, so the server can look at every message while the agent only sees what it needs for its next step.
What a person knows and the agent doesn't
One of my test questions asked what concerns clients had raised about OpenAI.
In one run, the agent searched for messages containing both "OpenAI" and "client," which found 39 messages. The search it actually needed was "OpenAI" in messages sent by clients, which found 105.
A person wouldn't make that mistake. We know clients don't say the word "client" when they talk, the same way we know a message that mentions OpenAI isn't automatically a complaint about OpenAI. An agent doesn't know that inherently, and it doesn't always notice that it should correct itself. When it does, it costs extra work. In that run, it searched the wrong way, pulled up a whole thread that didn't help, and only then fixed the search.
So I tried to give the agent what we know for free, in three places. The first is the tools themselves. "Client" is a filter on who sent the message, so it's an option the agent picks instead of a word it has to guess. The second is the results. Every response says what it searched (for example, 105 messages across 10 conversations, with 97 left out), so the agent can see the scope of what it's looking at. The third is a short set of instructions that tells the agent to turn the question into filters, to remember that a mention isn't a concern, and to change the search when the evidence doesn't answer the question.
With those instructions, the agent used the client filter on its first search. It made 4 tool calls instead of 6, got back about 27 KB of tool output instead of 39 KB, and finished in about 21 seconds instead of 43. That's one run each, so I wouldn't read too much into the exact numbers, but it's the difference I was hoping to see.
How it turned out
Sending every match for that question would have been about 98 KB. The answer came from about 12 KB of excerpts. On the full test set, returning every match for one search came to just over 1 MB, while counting them took about 4 KB.
Making search this efficient also means you can use a cheaper model, which was one of my goals from the start. Handing a smart model the raw table works, but you're paying it to do a lot of digging. When the tools make it quick to find what it's looking for, the model has much less work to do. The demo video runs on Luna instead of Sol or Opus, and it still finds the answer.
It's all synthetic data, so the real world will be messier. But the idea held up, and I haven't needed the vector database yet.