Findings of EMNLP 2026

GRASP

GRanularity-Aware Search Policy for Agentic RAG

Teaching retrieval agents when to search semantically, when to match lexically, and when to read more.

Varun Gandhi1,*, Jaewook Lee1,*, Shantanu Todmal1, Franck Dernoncourt2, Ryan Rossi2, Andrew Lan1, Zichao Wang2

1University of Massachusetts Amherst    2Adobe Research

*Equal contribution

arXiv Code Models Explore trajectories
A multi-hop question about which Eminem album included vocals from the singer behind Unapologetic. The single-step panel hallucinates Encore despite having the right documents; the multi-step panel hallucinates Kelly Clarkson and also answers Encore.
Why controlled context matters for multi-hop reasoning. Single-step reasoning fails despite sufficient evidence, because irrelevant content distracts the model. Multi-step reasoning compounds errors: it retrieves non-relevant documents and hallucinates a misleading connection that derails every later turn. GRASP works this exact example correctly. Step through all three trajectories below.
Naive RAG
“Encore”  โœ•
Both gold documents were in the context window. It still answered the wrong album.
Agentic RAG ยท Search-R1
“Encore”  โœ•
Invented Kelly Clarkson as the bridge entity, then spent every later turn confirming it.
GRASP (ours)
“The Marshall Mathers LP 2”  โœ“
Declined the same distractor, verified the bridge entity with a paragraph read, then answered.
Abstract

Agentic retrieval-augmented generation (RAG) extends static RAG by allowing language models to iteratively reason, generate search queries, retrieve evidence, and predict answers. However, it remains challenging for models to decide when to retrieve, whether to use lexical matching or semantic similarity, and how to control context granularity to prevent irrelevant tokens from interfering with agent reasoning.

In this paper, we introduce GRASP, a reinforcement learning (RL) framework for training agents to adaptively coordinate complementary retrieval tools during multi-step reasoning. GRASP provides the agent with semantic search, keyword search, and paragraph-reading actions, enabling it to retrieve sentence-level evidence and expand further context only when needed. We train the policy with a reward that jointly accounts for answer accuracy, grounded reading, complementary search, and turn efficiency.

Experiments on multi-hop reasoning benchmarks across 3B and 7B models show that GRASP improves retrieval and downstream question answering performance compared with single-step retrieval and multi-step agentic RAG baselines. Qualitative and ablation analyses show that the learned policy develops interpretable skimming and scanning behavior: semantic search for broad exploration, paragraph reading for local verification, and keyword search for entity-specific evidence.

0.90
Retrieval recall on 2Wiki
(best; +0.13 over IRCoT)
+0.08
EM over best 3B baseline
on HotpotQA
3
Retrieval actions the
policy learns to coordinate
0.26 โ†’ 1.22
Total reward over
270 GRPO steps
The problem

Retrieval succeeded. The answer is still wrong.

Agentic RAG lets a model retrieve repeatedly, but repetition alone does not fix the three decisions that actually determine whether a multi-hop chain holds together.

1

Complementary retrieval signals

A fixed hybrid retriever is enough for single-step RAG. Across a reasoning trajectory it is not: some steps need exact lexical matching to pin an entity name, others need semantic similarity to connect conceptually related evidence. The right signal changes step to step.

2

Context granularity

Large chunks obscure the one sentence that matters. Retrieving whole paragraphs fills the context window with entities and relations the agent did not ask for, weakening the signal for which clue should drive the next search.

3

Learning the policy

Deciding when to retrieve, which signal to use, and how much context to expose is a sequential decision problem. Existing methods fix a single retriever and a single granularity, so there is nothing to adapt.

The failure compounds. Because agentic reasoning proceeds incrementally, one bad retrieval or one hallucinated intermediate conclusion propagates into every subsequent query. In the example above, Search-R1's first-hop search returns five non-relevant documents; the agent invents Kelly Clarkson, and from that point on it is searching for evidence about the wrong person.
Method

GRASP

We formulate agentic RAG as a finite-horizon MDP and learn the retrieval policy with GRPO, using a trajectory-level reward that scores how the agent gathers evidence, not just whether it answers correctly.

GRASP framework: a query goes to a rollout module with keyword, semantic, and read tools plus a policy LLM; rollouts are scored by a reward model against a reference LLM with KL penalty; group computation produces advantages that update the policy via GRPO.
Overview of the GRASP framework. Given a query q, the policy LLM generates multiple rollouts. For example, trajectory ξ1 follows reason → semantic search → read parent paragraph → reason → keyword search. Each rollout is scored by a reward model, with a reference LLM providing a KL-regularization penalty. The policy is updated via GRPO using advantages computed from group-wise rewards.

Action space

Both search actions return the top-K sentences with their parent-paragraph ids. The agent expands to full context only when it decides it needs to.

τs

Semantic search

<semantic_search>

Dense retrieval with Qwen3-0.6B. Recovers evidence under lexical mismatch: the explore step that surfaces a candidate topic or entity when the agent has no anchor yet.

τk

Keyword search

<keyword_search>

Lexical matching with BM25. Favors exact or near-exact overlap: the scan step, used once a specific entity name must be matched in the corpus.

τr

Read paragraph

<read_chunk>

Expands a retrieved sentence into its parent paragraph. The verify step: confirm a bridge entity in context before committing the next hop to it.

Reward design

Answer accuracy stays unweighted and the auxiliary terms are capped at 1.0 in aggregate, so the policy cannot trade correctness for easier intermediate signals.

RAAnswer accuracy

weight 1.0

Token-level F1 between the predicted and reference answer.

RA = F1(ลท, y) โˆˆ [0, 1]

RRGrounded reading

α = 0.7

Rewards reads of gold-document evidence and penalizes reads of distractors.

RR = F1(๐’Ÿr, ๐’Ÿg)

RSComplementary search

β = 0.15

Fires only when both search tools retrieve gold evidence, which discourages repeated calls to a single retriever.

RS = ๐Ÿ™[๐’Ÿs โˆฉ ๐’Ÿg โ‰  โˆ… โˆง ๐’Ÿk โˆฉ ๐’Ÿg โ‰  โˆ…]

RETurn efficiency

γ = 0.15

Gated on RA > 0.5, so the agent cannot earn efficiency credit by guessing early.

RE = (T โˆ’ Tcur)/T ยท ๐Ÿ™[RA > 0.5]
R = RA + ฮฑยทRR + ฮฒยทRS + ฮณยทRE
Answer accuracy Grounded reading Complementary search Turn efficiency

Training. Policy learned with GRPO on the HotpotQA distractor split (90,447 examples; 270 optimization steps ≈ 19% of an epoch, at which point validation reward plateaued). Tool-observation tokens are masked from the policy loss, so the model is trained only on text it generates itself. Corpus is indexed at the sentence level with parent-paragraph and title metadata retained.

Case study explorer

Same question. Same evidence. Different chains.

Every trajectory below is verbatim from the paper's appendix. Step through them together to see exactly where each method's reasoning diverges.

step 1 / 1
How to read a column. Steps are tinted by action: semantic search, keyword search, paragraph read. Retrieved passages carry a gold badge when they are one of the question’s supporting documents and a bridge badge for the linking entity, so you can see at a glance that the failing methods did retrieve the evidence. In the reasoning text, red marks an entity the model committed to that the evidence does not support; green marks an entity on the correct chain. Press โ† / โ†’ to step, or hit Play.
Experimental results

Retrieval and QA across three multi-hop benchmarks

HotpotQA, 2WikiMultiHopQA and MuSiQue. Trained on HotpotQA only, so 2Wiki and MuSiQue measure cross-dataset generalization. 500 questions sampled per dataset (seed 42).

Emergent behavior

The policy learns to skim, verify, then scan

Modeling trajectories as a first-order Markov chain over actions reveals a consistent cycle that nobody supervised. It closely resembles how human readers forage for information.

τs
Skim

Semantic search

Begin broad: locate a relevant topic or entity in a space the agent has no anchor in yet.

τr
Verify

Read paragraph

Expand a promising sentence into its parent paragraph to confirm local context and extract the bridge entity.

τk
Scan

Keyword search

Use that grounded entity as an exact lexical query, shifting from broad exploration to targeted retrieval.

When keyword search comes up short

If τk returns too little, usually because the query was too narrow or too fine-grained, the agent returns to τs with a broader semantic query rather than pressing on. Keyword search thus plays two roles: a precision lookup, or a signal that renewed exploration is needed.

When it finds something promising

The agent invokes τr to check the fact in its full paragraph rather than trusting a search-result snippet. This verify-before-advance habit is what prevents the premature commitment that sinks the baselines in both case studies.

Why this is not just prompting. The Base row in the 3B results uses GRASP's exact prompt and all three tools with no RL, and scores 0.17 EM on HotpotQA, below every single-step baseline. Access to the tools is not the mechanism; the learned policy over them is.
Citation

BibTeX

@inproceedings{gandhi2026grasp,
  title     = {GRASP: Granularity-Aware Search Policy for Agentic RAG},
  author    = {Gandhi, Varun and Lee, Jaewook and Todmal, Shantanu and
               Dernoncourt, Franck and Rossi, Ryan and Lan, Andrew and
               Wang, Zichao},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026},
  year      = {2026},
  eprint    = {2607.10463},
  archivePrefix = {arXiv}
}