Retrieval for RAG-Resistant Docs: An agent that searches, outlines, and reads entire documents outperforms chunk-and-embed retrieval

Retrieving documents chunk by chunk works well with many types of text, but it can fail with documents in which the meaning of a given span — say, numbers in a table — depends on particulars found elsewhere on the page — say, the label of their units. Researchers assigned agents to keep all the necessary information in view. 

What’s new: Sagar Tamang, Ayush Vyas, and Tabarakul Hazarika at Indian Institute of Technology Patna and TwoSpoon, an independent tech lab, developed Reliable Embedding-free Agentic Document-search (READ). Their method replaces vector-based retrieval with an agentic harness and specialized search tools. 

Key insight: For RAG-style retrieval, a document is divided into chunks that are embedded, and a model retrieves chunks whose embeddings are similar to the embedding of the query. But this approach can fail with documents such as financial or scientific reports, where numbers in a table gain important meaning from units or dates that may lie many lines away, or legal briefs, in which words may have special meanings that are defined many paragraphs away. An agent with tools to navigate undivided documents can look beyond predefined chunks.

How it works: The authors used the READ method with Gemini 2.5 Pro.

• The system had access to three tools: (i) A keyword-search tool returned matching line numbers, (ii) an outline tool listed section headings with line numbers, and (iii) a text-reader tool fetched specific line ranges.
• Given a query like, “What is the mathematical difference between the Fiscal Deficit and the Revenue Surplus for the year 2024-2025?,” the READ agent activated a loop that generated a search term such as “Fiscal Deficit,” scanned a document for the term, checked matching line(s) against the document outline, and read the text from the table headers to the matching line(s).
• Within the loop, the model refined its query when it got too many hits, too few hits, or insufficient information to make sense of what it found. Each refinement repeated the loop with a new search term or a new range to be read.

Results: The authors used READ, RAG-style retrieval, and RAG-style retrieval in an agentic loop to answer 51 questions about a 780-page financial report published by the government of Gujarat, India. All systems were based on Gemini 2.5 Pro. READ answered 58.8 percent of the questions correctly, outperforming both RAG-style retrieval (15.7 percent) and RAG-style retrieval in an agentic loop (27.5 percent)

Why it matters: Chunk-and-embed retrieval methods tend not to work well with documents in which the meaning of any given passage is not self-contained, such as financial reports, contracts, and regulatory filings. In such cases, letting a model search the whole document is more effective.

We’re thinking: Chunk-and-embed has its place, but letting agents write search queries can be a better approach. Beyond READ and RLM (covered in last week’s issue of The Batch), Search-R1 trains LLMs to reason their way through multi-turn searches, and IRCoT uses an LLM’s reasoning traces as a search query.