BRIDGE: Bilevel Retrieval-Credit-Aware Agentic Reinforcement Learning

Introduction

Better reasoning begins with better evidence.

Agentic reinforcement learning (RL) trains LLMs to interleave reasoning with search by learning from sparse yet verifiable outcome rewards. However, most agentic RL methods optimize only model-generated reasoning, search, and answer tokens. In retrieval-augmented generation (RAG), retrieved information is instead treated as an environment observation and masked from the RL objective. Consequently, the retriever receives no corrective feedback when missing or misleading evidence derails a rollout.

This creates an information-credit gap: failures induced by missing or misleading evidence are attributed entirely to the LLM policy, while the retriever that supplied the information receives no corrective signal. The problem is particularly acute in multi-hop QA, where successful reasoning depends on relevant evidence distributed across multiple documents, and no policy update can compensate for a missing supporting document.

Consider a question about brown tumors: one retrieved passage links them to hyperparathyroidism, while another discusses unrelated finger lesions. If the agent answers incorrectly, the final reward signals failure but does not identify which passage was useful. RAG likelihood provides a document-specific learning signal, favoring passages under which the current LLM assigns higher probability to the reference answer—even when the sampled rollout fails.

RAG signalExample: useful evidence in a failed rollout

What causes brown tumors?

Reference answer: Hyperparathyroidism

Candidate passages

A · Relevant evidence

Brown tumors are associated with hyperparathyroidism.

B · Distractor

Other finger lesions can arise from local injury or malignancy.

RL outcome reward

Sampled answerMultiple myeloma

Incorrect0

One outcome reward does not identify which passage helped.

RAG likelihood

A · Higher likelihoodSupports the reference answer

B · Lower likelihoodDoes not support the reference answer

A failed rollout can still contain useful evidence worth rewarding.

Conversely, RL connects retrieval learning to the outcome of the complete trajectory. A passage can be useful because it enables the next search, even without stating the final answer. In the paper’s Norman Reedus example, the episode passage identifies the actor; a follow-up search is needed to find the company, Prada.

RL outcomeExample: evidence that enables the next search

Which company’s advertisements featured the actor playing Daryl in “Cherokee Rose”?

Reference answer: Prada

First retrieval · both checkpointsDaryl Dixon is played by Norman Reedus.

Step 50Early checkpoint

No follow-up searchAdvertising evidence remains missing.
Final answerLifeIncorrect

Step 250Later checkpoint

Follow-up searchNorman Reedus’s advertising work
Second retrievalHis modeling history names Prada.
Final answerPradaCorrect

The later trace uses the actor clue to retrieve the missing advertising evidence.

Motivated by the complementary roles of RL and RAG signals, we argue that alleviating the information-credit gap requires optimizing both the retriever and the LLM policy. However, the order of optimization matters. We find that updating the retriever first provides the LLM with higher-quality evidence, thereby increasing its propensity to search. In a controlled sequential comparison, RAG-then-RL attains a multi-hop average of 23.8, compared with 23.3 for RL-then-RAG and 21.7 for the top-performing baseline.

Accuracy gains from RAG-then-RL and RL-then-RAG over VT-Search across seven QA benchmarks. — **Retrieval first motivates the bilevel hierarchy.** Accuracy gains over VT-Search across seven QA benchmarks. This comparison motivates adapting retrieval at the lower level before optimizing the LLM policy through the upper-level RL objective.

To preserve this ordering throughout training, we introduce BRIDGE, a memory-efficient first-order bilevel method for retrieval-credit-aware agentic reinforcement learning, with nested retriever and LLM-policy updates. At the lower level, for each LLM policy, a query-side retriever LoRA learns to select relevant documents by optimizing RAG answer likelihood and trajectory-level reward at retrieval turns. At the upper level, the LLM policy is then optimized on the same rollouts using the verifiable outcome reward.

Across seven open-domain QA benchmarks, BRIDGE achieves the highest overall and multi-hop accuracy among the compared methods with Qwen2.5-3B-Instruct and Qwen2.5-7B-base, with 9.6- and 3.4-point multi-hop gains over the strongest baselines. With Qwen3-14B, it reaches 50.7 overall and 45.5 multi-hop accuracy. On five medical QA benchmarks, BRIDGE achieves the highest macro-averaged answer accuracy and reasoning quality. At comparable wall-clock time, the 3B-base model retains an 8.0-point multi-hop advantage over VT-Search.

Tool-use rollout

01QuestionWhat evidence is needed?

02RetrievalExpose useful information

03ReasoningLearn from retrieved evidence

04AnswerReturn credit to both levels

Preview of BRIDGE’s learning hierarchy

Upper level

LLM policy

RL onlyTask reward

RAGAnswer likelihood

RLTrajectory-level reward

Lower level

Query-side retriever

Update retrieval before policy

Method

1

Rollout generation

Generate fresh rollouts

Generate tool-use trajectories with the current LLM policy and retriever, then compute rewards and advantages.

2

Lower level

Adapt the retriever

Update the query-side LoRA using retrieval-turn RL credit and RAG answer likelihood from those rollouts.

3

Upper level

Update the policy

After the retriever update, apply GRPO to the LLM-generated action tokens in the same rollouts.

BRIDGE formalizes retrieval-first training as a bilevel problem:

\min_{\theta,\eta}\ \mathcal{J}_{\mathrm{RL}}(\theta,\eta) \quad \mathrm{s.t.}\quad \eta \in \eta^*(\theta) := \arg\min_{\eta'} \mathcal{L}_{\mathrm{RAG}}(\theta,\eta').

For each policy $\theta$ , the lower level selects a retriever that minimizes the RAG loss. The upper level then minimizes the RL objective subject to that lower-level solution. This formulation optimizes retrieval for the current policy before updating that policy, while allowing the retriever to adapt as the policy evolves.

BRIDGE couples an LLM policy with a query-side retriever LoRA using reinforcement-learning and retrieval-augmented generation signals. — **(a) BRIDGE framework.** Retriever and policy updates alternate within one retrieval-first loop.

A multi-hop question trace leading into BRIDGE's upper-level policy objective and lower-level retrieval objective. — **(b) Bilevel optimization formulation.** The lower-level RAG objective defines the retriever used by the upper-level RL objective.

Technical details

Solving this problem is difficult because the lower-level optimum $\eta^*(\theta)$ depends implicitly on $\theta$ . Exact differentiation through this dependence introduces inverse-Hessian and Jacobian terms that are infeasible at LLM–retriever scale. Existing first-order penalty methods avoid these computationally heavy second-order operations but maintain both an unpenalized and a penalized retriever state:

\eta^*(\theta) = \arg\min_{\eta'} \mathcal{L}_{\mathrm{RAG}}(\theta,\eta') \qquad \eta_{\gamma}^*(\theta) = \arg\min_{\eta} \left[\mathcal{J}_{\mathrm{RL}}(\theta,\eta) + \gamma\mathcal{L}_{\mathrm{RAG}}(\theta,\eta)\right].

The unpenalized state is optimized only for RAG, whereas the penalized state combines trajectory-level RL credit with the weighted RAG loss. Maintaining both states requires separate retriever parameters, optimizer states, and retrieval logs. The corresponding fully first-order hypergradient is

\nabla \mathcal{J}_{\mathrm{RL}}\!\left(\theta,\eta^*(\theta)\right) \approx \nabla_\theta \mathcal{J}_{\mathrm{RL}}\!\left(\theta,\eta_\gamma^*(\theta)\right) + \gamma\!\left[ \nabla_\theta \mathcal{L}_{\mathrm{RAG}}\!\left(\theta,\eta_\gamma^*(\theta)\right) - \nabla_\theta \mathcal{L}_{\mathrm{RAG}}\!\left(\theta,\eta^*(\theta)\right) \right].

The last two terms compare two gradients of the RAG loss with respect to the policy: one evaluated at the penalized retriever and one at the unpenalized retriever. Computing this difference requires both states. To alleviate the memory cost, BRIDGE avoids this second retriever state by a careful examination of RAG and RL objective landscapes.

Landscape analysis

The RAG policy gradient cannot be omitted individually; instead, BRIDGE neglects the difference between the two RAG gradients evaluated at the nearby retriever states $\eta_\gamma^*(\theta)$ and $\eta^*(\theta)$ .

Estimator geometry

Joint RAG loss landscape over the retriever parameter eta and policy parameter theta. — **(a) Joint RAG, middle checkpoint.** The RAG loss varies strongly with the policy across most retriever states, so $\nabla_\theta \mathcal{L}_{\mathrm{RAG}}$ cannot be set to zero at the unpenalized retriever alone.

Joint reinforcement-learning objective landscape over the retriever parameter eta and policy parameter theta. — **(b) Joint RL, initial checkpoint.** The RL loss shows weak variation along the retriever direction across a broad range of policy perturbations, suggesting that $\eta_\gamma^*(\theta)$ and $\eta^*(\theta)$ remain close.

Policy training dynamics

Initial reinforcement-learning objective landscape projected onto two policy directions. — **(c) Policy-only RL, initial checkpoint.** At initialization, the projected RL surface around the policy is sharp and irregular.

Final reinforcement-learning objective landscape projected onto two policy directions. — **(d) Policy-only RL, final checkpoint.** At the final checkpoint, the projected RL surface around the learned policy is substantially smoother.

Qwen2.5-3B-base with an E5 retriever. $\eta$ and $\theta$ denote retriever and policy directions; $\alpha$ and $\beta$ denote two policy-space projection directions.

Algorithmic implication. Panel (a) rules out the shortcut $\nabla_\theta \mathcal{L}_{\mathrm{RAG}}(\theta,\eta^*(\theta)) \approx 0$ : the RAG policy gradient at the unpenalized retriever cannot be dropped by itself. Panel (b) instead suggests $\eta_\gamma^*(\theta) \approx \eta^*(\theta)$ , so the last two RAG policy-gradient terms are evaluated at nearby retriever states. BRIDGE therefore neglects their difference as a pair—not the RAG gradient at $\eta^*(\theta)$ alone—and uses $\nabla_\theta \mathcal{J}_{\mathrm{RL}}(\theta,\eta_\gamma^*(\theta))$ as the policy-gradient estimator. Panels (c)–(d) further show that training moves the policy into a smoother RL region, consistent with improved generalization.

Case Study

The following nine examples cover comparison, compositional retrieval, query repair, and medical QA. Choose a question, then compare BRIDGE with VT-Search through selected reasoning, search, and retrieval excerpts from the paper’s logged trajectories.

Case 1 · Battle durationCase 2 · Zhou founderCase 3 · Birth yearsCase 4 · Species countsCase 5 · Hang-OnCase 6 · ChimeraCase 7 · Brown tumorsCase 8 · Drug-induced dystoniaCase 9 · Canine rest design

Correct trace: BRIDGE

1Reasoning<think>

To find out which battle lasted longer, I need to gather information about the duration of both the Battle of Hürtgen Forest and the Battle of Pusan Perimeter.

2Search<search>

Which battle lasted longer, Battle of Hürtgen Forest or Battle of Pusan Perimeter?

3Retrieved information<information>

Doc 1(Title: "Battle of Pusan Perimeter") Battle of Pusan Perimeter was a large-scale battle between United Nations and North Korean forces lasting from August 4 to September 18, 1950 ... It was one of the first major engagements of the Korean War ... UN forces made a final stand at the Pusan Perimeter ...

Doc 2(Title: "Battle of Pusan Perimeter") North Korea refused to surrender ... the battle page again states August 4 to September 18, 1950 and describes the UN troops being rallied after repeated defeats by the advancing North Koreans ...

Doc 3(Title: "Battle of Hürtgen Forest") Battle of Hürtgen Forest was fought from 19 September to 16 December 1944 between American and German forces ... It was the longest battle on German ground during World War II and the longest single battle the U.S. Army has ever fought ...

AnswerBattle of Hürtgen Forest

Incorrect trace: VT-Search

1Search<search>

which battle lasted longer, Battle of Hürtgen Forest or Battle of Pusan Perimeter

2Retrieved information<information>

Doc 1(Title: "Battle of Hürtgen Forest") Battle of Hürtgen Forest was a series of fierce battles fought from 19 September to 16 December 1944 between American and German forces on the Western Front during World War II in the Hürtgen Forest ... It was the longest battle on German ground during World War II, and is the longest single battle the U.S. Army has ever fought ...

Doc 2(Title: "Battle of Pusan Perimeter") The Battle of Pusan Perimeter was a large-scale battle between United Nations and North Korean forces lasting from August 4 to September 18, 1950 ... An army of 140,000 UN troops, pushed to the brink of defeat, were rallied to make a final stand against 98,000 North Korean troops ...

Doc 3(Title: "Battle of Pusan Perimeter") North Korea refused to surrender ... Online sources repeat that the battle lasted from August 4 to September 18, 1950 and describe UN forces being forced back to a defensive line around the southeastern tip of the Korean peninsula ...

AnswerBattle of Pusan Perimeter

BRIDGE retrieves the duration of both battles and identifies Hürtgen Forest as the longer engagement. VT-Search retrieves the relevant dates but answers with the shorter Pusan Perimeter.

Correct trace: BRIDGE

1Reasoning<think>

To find out who was the first king of the longest Chinese dynasty, I need to identify the longest Chinese dynasty and its founding monarch.

2Search<search>

Who was the first king of the longest Chinese dynasty?

3Retrieved information<information>

Doc 1(Title: "King Nan of Zhou") King Nan of Zhou (? - 256 BC), born Ji Yan, was the 37th and last king of the Chinese Zhou dynasty ... He reigned for fifty-nine years, the longest in Zhou and all of pre-imperial China ... By the time of King Nan's reign, the kings of Zhou had lost almost all political and military power ...

Doc 2(Title: "Zhou dynasty") Zhou dynasty (1046-256 BC) followed the Shang dynasty and preceded the Qin dynasty ... The Zhou dynasty lasted longer than any other dynasty in Chinese history ... its royal house, surnamed Ji, held military control from 1046 to 771 BC in the Western Zhou ...

Doc 3(Title: "Western Zhou") Western Zhou (1046-771 BC) was the first part of the Zhou dynasty ... It was established by King Wu of Zhou, who overthrew the Shang dynasty ... the Zhou later moved east after King You was overthrown in 771 BC ...

AnswerKing Wu of Zhou

Incorrect trace: VT-Search

1Search<search>

who was the first king of the longest Chinese dynasty

2Retrieved information<information>

Doc 1(Title: "King Nan of Zhou") King Nan of Zhou (? - 256 BC), born Ji Yan, was the 37th and last king of the Chinese Zhou dynasty ... He was king for fifty-nine years, the longest in the Zhou Dynasty and all of pre-imperial China ... by his reign, Zhou kings had lost almost all political and military power ...

Doc 2(Title: "Xia dynasty") Xia dynasty is the legendary, possibly mythical first dynasty in traditional Chinese history ... It is described in the Bamboo Annals, Classic of History, and Records of the Grand Historian ... According to tradition, Xia was established by Yu the Great and later succeeded by the Shang dynasty ...

Doc 3(Title: "Tang of Shang") Tang or Cheng Tang, recorded on oracle bones as Da Yi, was the first king of the Shang dynasty ... He overthrew Jie, the last ruler of Xia ... Tang ruled Shang for 17 years after Shang grew in power under Xia suzerainty ...

AnswerTang of Shang

BRIDGE retrieves both links in the reasoning chain: the Zhou dynasty lasted longest, and King Wu established Western Zhou. VT-Search retrieves the founder of Shang without resolving the first link.

Correct trace: BRIDGE

1Reasoning<think>

To determine who was born first between Andreas P. Nielsen and Lydia Flood Jackson, I need to gather information about their birth dates.

2Search<search>

Was Andreas P. Nielsen or Lydia Flood Jackson born first?

3Retrieved information<information>

Doc 1(Title: "Lydia Flood Jackson") Lydia Flood Jackson (June 6, 1862 - July 8, 1963) was an American businesswoman, suffragist, and clubwoman ... Lydia Flood was born in Oakland, California ... her father Isaac Flood joined the California Colored Convention Movement to fight segregation in California schools ...

Doc 2(Title: "Fredrikke Nielsen") Fredrikke Louise Nielsen was born Fredrikke Louise Jensen July 5, 1837 in Haugesund ... She was a Norwegian actress and was married to actor Harald Nielsen ...

Doc 3(Title: "Andreas P. Nielsen") Andreas P. Nielsen (1943-2016) was a Danish politician ... He was born on February 12, 1943 in Copenhagen ... He represented a Danish constituency and served in national politics ...

AnswerLydia Flood Jackson

Incorrect trace: VT-Search

1Search<search>

who was born first, Andreas P. Nielsen or Lydia Flood Jackson

2Retrieved information<information>

Doc 1(Title: "Constance Jackson") Constance L. Jackson is a business start-up professional in health technology and filmmaking ... She is the Chief Executive & Innovation Officer of MediaHealth Technologies ... In 2008, Jackson published a book about abolitionist Lydia Maria Child, "Over the River... Life of Lydia Maria Child, Abolitionist for Freedom" ...

Doc 2(Title: "Lydia Flood Jackson") Lydia Flood Jackson (June 6, 1862 - July 8, 1963) was an American businesswoman, suffragist, and clubwoman ... Lydia Flood was born in Oakland, California ... her mother opened schools for black children in Sacramento and Oakland ...

Doc 3(Title: "Constance Jackson") Constance L. Jackson is a business start-up professional in health technology and filmmaking ... She has spoken at national and international conferences ... She is the writer, director, and producer of feature-length documentary films ...

AnswerAndreas P. Nielsen

BRIDGE retrieves birth years for both candidates and correctly selects Lydia Flood Jackson. VT-Search finds only her date and lacks the evidence needed for a valid comparison.

Correct trace: BRIDGE

1Reasoning<think>

To determine which genus, Erodium or Cymbidium, includes more species, I need the number of species in each genus.

2Search<search>

Does Erodium or Cymbidium include more species?

3Retrieved information<information>

Doc 1(Title: Erodium) Erodium is a genus of flowering plants in Geraniaceae ... The genus includes about 60 species, native to North Africa, Indomalaya, the Middle East, and Australia ... flowers strongly resemble the better-known Geranium ...

Doc 2(Title: "Cymbidium eburneum") Cymbidium eburneum, the ivory-colored cymbidium, is a species of orchid ... It belongs to the genus Cymbidium and is found in the eastern Himalayas, Assam, and southern China ...

Doc 3(Title: Cymbidium) Cymbidium is a genus of orchids ... There are approximately 50 species in the genus ... the species are distributed in tropical and subtropical Asia and northern Australia ...

AnswerErodium

Incorrect trace: VT-Search

1Search<search>

does Erodium or Cymbidium include more species

2Retrieved information<information>

Doc 1(Title: Erodium) Erodium is a genus of flowering plants in the botanical family Geraniaceae ... The genus includes about 60 species, native to North Africa, Indomalaya, the Middle East, and Australia ... They are perennials, annuals, or subshrubs with five-petalled flowers in white, pink, and purple ...

Doc 2(Title: Erodium) Erodium includes about 60 species ... American species are known as filarees or heron's bill, whereas Eurasian ones are called storksbills in English ... some species are used as food plants by Lepidoptera larvae ...

Doc 3(Title: "Cymbidium suave") Cymbidium suave, or the snake orchid, is an Australian orchid species ... It is part of the genus Cymbidium ... The genus consists of 52 species and this species has flattened, yellowish green pseudobulbs ...

AnswerCymbidium

BRIDGE compares genus-level counts for Erodium and Cymbidium and selects Erodium. VT-Search retrieves compatible counts but returns an answer that contradicts them.

Correct trace: BRIDGE

1Reasoning<think>

To find out the genre of "Hang-On," I need to gather information about the game "Hang-On."

2Search<search>

What genre is Hang-On?

3Retrieved information<information>

Doc 1(Title: "Hang on the Box") Hang on the Box, also known as HOTB, is a punk band based in Beijing ... They were China's first all-female punk band ... the lineup includes Wang Yue on vocals, Yilina on bass, Li Yan Fan on guitar, and Shenjing on drums ...

Doc 2(Title: "Hang On (Weezer song)") "Hang On" is a song by the band Weezer ... It impacted radio on February 15, 2011 and is the seventh track and second single from the album "Hurley" ...

Doc 3(Title: "The Hang Ups") The Hang Ups are an indie pop rock band from Minneapolis, Minnesota, formed in 1990 ... Members include Brian Tighe, Jeff Kearns, Stephen Ittner, and John Crozier ...

4Search<search>

What genre is Hang On (video game)?

5Retrieved information<information>

Doc 1(Title: "Super Hang-On") Super Hang-On is a motorcycle racing video game by Sega ... it is the sequel to Hang-On and uses arcade-style racing across multiple tracks ...

Doc 2(Title: Hang-On) Hang-On uses a behind-the-motorcycle perspective ... the player races a linear race track divided into stages within a limited time ... reaching checkpoints extends the time limit ...

Doc 3(Title: Hang-On) Hang-On was released as an arcade game with motorcycle-style controls ... The title is derived from the biker having to "hang on" to the bike while leaning ... the Master System version was also built into some consoles ...

Answerracing game

Incorrect trace: VT-Search

1Search<search>

what genre is Hang-On

2Retrieved information<information>

Doc 1(Title: "Hang on the Box") Hang on the Box, also known as HOTB, is a punk band based in Beijing ... They were China's first all-female punk band ... The band usually sings about sex and relationship issues in a forward political manner ...

Doc 2(Title: "The Hang Ups") The Hang Ups are an indie pop rock band from Minneapolis, Minnesota, formed in 1990 and fronted by vocalist and guitarist Brian Tighe ... They released their debut EP "Comin' Through" in 1993 and later released four studio albums ...

Doc 3(Title: "Hang on the Box") Hang on the Box were featured on the local edition cover of Newsweek six months later ... Their music could be described as cheerful and hilarious, deadly serious, and personally political ... The page again identifies HOTB as a punk band ...

Answerpunk

BRIDGE repairs the ambiguous query by searching for the video game and correctly answers “racing game.” VT-Search follows music results and answers “punk.”

Correct trace: BRIDGE

1Reasoning<think>

To find out who the father of Chimera is, I need to gather information about Chimera and her parentage.

2Search<search>

Who is the father of Chimera?

3Retrieved information<information>

Doc 1(Title: "Chimera (short story)") Chimera is a comical short story by Lee Yeongdo based on Dragon Raja ... The king of Caria, Amisodarus, raised the Chimera to terrorize the surrounding region ...

Doc 2(Title: "King Chimera") King Chimera is a fictional superhero in the DC Comics Universe ... The character first appeared in Justice Society of America and claims to be the son of "King" Standish ...

Doc 3(Title: "King Chimera") King Chimera is a fictional superhero in the DC Comics Universe ... He is a member of the Justice Society of America and first appeared in "Justice Society of America" (vol. 3) #24 ...

4Search<search>

Who is the father of Chimera in Greek mythology?

5Retrieved information<information>

Doc 1(Title: "Chimera (mythology)") Chimera is a monstrous hybrid creature in Greek mythology ... commonly described as fire-breathing and as one of the offspring of Typhon and Echidna ...

Doc 2(Title: "Chimera (mythology)") The mythology page discusses variant genealogies ... most traditions attribute the Chimera and related monsters to Typhon and Echidna ...

Doc 3(Title: "Chimera of Arezzo") The Chimera of Arezzo depicts the mythological Chimera in bronze ... the object is identified with the creature described as offspring of Typhon and Echidna ...

AnswerTyphon

Incorrect trace: VT-Search

1Search<search>

who is the father of Chimera

2Retrieved information<information>

Doc 1(Title: "King Chimera") King Chimera is a fictional superhero in the DC Comics Universe, a member of the Justice Society of America ... He first appeared in "Justice Society of America" (vol. 3) #24 in April 2009 ...

Doc 2(Title: "King Chimera") King Chimera keeps a mysterious identity and very little is known about his past ... the page says he claims to be the son of "King" Standish ...

Doc 3(Title: "Chimera (short story)") Chimera is a comical short story by Korean author Lee Yeongdo based on Dragon Raja ... The king of Caria, Amisodarus, raised the Chimera to terrorize the surrounding region ...

AnswerAmisodarus

BRIDGE recognizes the ambiguous entity and issues a second search for Greek mythology, which retrieves Typhon and Echidna. VT-Search stops after results for unrelated entities.

Appendix case study 7 · Medical QA · MedQA

Question

What causes finger pain, confusion, and brown tumors with bone loss in a 50-year-old woman?Show full questionHide full question

A 50-year-old woman is brought to the office by her daughter with a chief complaint of finger pain in her left hand. She is slightly confused and can’t remember when the pain started. Her daughter states that the pain has been present for approximately one month. She expresses concern that her mother has been progressively more confused during the past week. She reports that her mother also has had worsening nausea for several months. The patient’s vital signs are heart rate 92/min, respiratory rate 13/min, temperature 37.1°C (98.8°F), and blood pressure 120/86 mm Hg. Upon physical examination, the patient reports pain when the fingers of the left hand are palpated, and small palpable tumors are visible on the fingers. An X-ray image of the left-hand shows thin bones with brown tumors and osteolysis of the distal phalanges. What is the most likely cause for this patient’s condition?

A: Hyperparathyroidism
B: Hypocalcemia
C: Osteosarcoma
D: Multiple myeloma

Ground truthA: Hyperparathyroidism

BRIDGE VT-Search

Correct trace: BRIDGE

1Reasoning<think>

To answer this question, I need to analyze the symptoms and the X-ray findings provided. The patient has finger pain, small palpable tumors on the fingers, and an X-ray showing thin bones with brown tumors and osteolysis of the distal phalanges. …

2Search<search>

symptoms of finger pain, small palpable tumors on the fingers, thin bones with brown tumors and osteolysis of the distal phalanges

3Retrieved information<information>

Doc 1(Title: "Osteochondroma") Signs and symptoms

Doc 2(Title: "Brown tumour in phalanx of the ring finger due to chronic kidney failure. A case report.") Brown tumours are highly vascular lytic bone lesions found in primary and secondary hyperparathyroidism. The brown term is given due to the red-brown colour of the tissue, which is due to the accumulation of hemosiderin. …

Doc 3(Title: "Benign tumor") Signs and symptoms

4ContinuationUntagged model output

From the search results, I found that the symptoms described are consistent with hyperparathyroidism, which is a condition where the parathyroid glands produce too much parathyroid hormone (PTH). This hormone regulates the levels of calcium and phosphorus in the body, and when it is overproduced, it can lead to bone resorption and the formation of brown tumors.

5Reasoning<think>

Based on the symptoms and the X-ray findings, the most likely cause for this patient's condition is hyperparathyroidism. The presence of brown tumors and osteolysis of the distal phalanges is characteristic of this condition.

AnswerHyperparathyroidism

Incorrect trace: VT-Search

1Reasoning<think>

To answer this question, I need to analyze the given information and use it to determine the most likely cause of the patient's condition. The patient is a 50-year-old woman with finger pain in her left hand, confusion, and small palpable tumors on her fingers. …

2Search<search>

What is the most likely cause for a 50-year-old woman with finger pain, confusion, palpable tumors, and osteolysis of the distal phalanges?

3Retrieved information<information>

Doc 1(Title: "An unusual presentation of a ring injury.") A case of chronic erosion of the soft tissues and bone of the proximal phalanx of the ring finger in a 62-year-old woman is presented. The patient's rings had eroded the skin, nerves, and tendons of the finger as well as the palmar aspect of the proximal phalanx; however, circulation was adequate. …

Doc 2(Title: "Metastatic phalangeal osteolysis as an initial presentation of carcinoma colon.") Metastatic malignancies of the hand are rare and usually develop from lung, breast, or kidney tumors. Metastases from tumors and sarcomas of the gastrointestinal system are even more uncommon. …

Doc 3(Title: "Fibro-osseous pseudotumour of the digit-a diagnostic challenge.") A 60-year-old right-hand dominant woman presented to the emergency department with a 3-week history of pain and swelling in her right thumb. The X-ray showed evidence of erosion at the tip of the distal phalanx. …

4ContinuationUntagged model output

Based on the information provided and the symptoms described, the most likely cause for this patient's condition is multiple myeloma.

AnswerMultiple myeloma

BRIDGE retrieves evidence linking brown tumors to hyperparathyroidism. VT-Search retrieves other causes of finger lesions and retains multiple myeloma.

Appendix case study 8 · Medical QA · MedQA

Question

What treats painful neck stiffness and rotation after haloperidol in a 31-year-old man?Show full questionHide full question

A 31-year-old man, who was hospitalized for substance-induced psychosis two days prior, has had episodic neck stiffness and pain for the past 8 hours. These episodes last for approximately 25 minutes and are accompanied by his neck rotating to the right. During the last episode, he was able to relieve the stiffness by lightly touching his jaw. He has received six doses of haloperidol for auditory hallucinations since his admission. He appears anxious. His temperature is 37.3°C (99.1°F), pulse is 108/min, and blood pressure is 128/86 mm Hg. Examination shows a flexed neck rotated to the right. The neck is rigid with limited range of motion. Which of the following is the most appropriate therapy for this patient's symptoms?

A: Benztropine
B: Physical therapy
C: Dantrolene
D: Baclofen

Ground truthA: Benztropine

BRIDGE VT-Search

Correct trace: BRIDGE

1Reasoning<think>

To answer this question, I need to identify the cause of the patient's symptoms and then determine the most appropriate therapy. The patient has a history of substance-induced psychosis and is currently experiencing episodic neck stiffness and pain, which is accompanied by neck rotation to the right. …

2Search<search>

treatment for neck dystonia caused by haloperidol

3Retrieved information<information>

Doc 1(Title: "A patient with haloperidol induced laryngeal dystonia.") We discus the case of a forty-nine year old patient with haloperidol induced laryngeal dystonia (LD). Laryngeal dystonia is a life threatening, very rare medical condition which is difficult to diagnose. …

Doc 2(Title: "Haloperidol-Induced Dystonia due to Sedation for Upper Gastrointestinal Endoscopy: A Pediatric Case Report.") Dystonia is a movement disorder characterized by sustained muscle tone. Antipsychotic agents sometimes cause acute dystonia that can rapidly worsen within a few hours or days. …

Doc 3(Title: "Treatment of dystonia and tics.") Treatment of dystonia and tics continues to evolve. In dystonia, while oral agents such as benzodiazepines, baclofen and anticholinergics remain in use, botulinum toxin (BoNT) continues to be regarded as the treatment of choice for focal and segmental dystonia, but new preparations are being studied. …

4ContinuationUntagged model output

Based on the search results and the patient's history of haloperidol use, the most appropriate therapy for this patient's symptoms is Benztropine, which is an anticholinergic medication used to treat dystonia caused by antipsychotic medications like haloperidol.

AnswerBenztropine

Incorrect trace: VT-Search

1Reasoning<think>

To answer this question, I need to determine the most appropriate therapy for the patient's symptoms of episodic neck stiffness and pain, which are accompanied by neck rotation and limited range of motion. First, let's analyze the patient's condition: - The patient is a 31-year-old man with a history of substance-induced psychosis. …

2Search<search>

What is the most appropriate therapy for neck muscle spasms or stiffness in patients with substance-induced psychosis or those taking haloperidol?

3Retrieved information<information>

Doc 1(Title: "[On a muscular dystony case from manganism. Therapeutic clinic considerations (author's transl)].") It is described a muscular dystony case of manganism, treated with good results in two following phases with haloperidol and diazepam and then with L-Dopa + benserazide. On the basis of clinical and laboratory results, it makes way the hypothese that the haloperidol had developed his therapeutic action with chelation mechanism.

Doc 2(Title: "Is haloperidol or a benzodiazepine the safest treatment for acute psychosis in the critically ill patient?") A best evidence topic in cardiac surgery was written according to a structured protocol. The question addressed was whether haloperidol or a benzodiazepine is the safest treatment for acute psychosis in the critically ill patient. …

Doc 3(Title: "InternalMed_Harrison") Optimal treatment of stiff-person syndrome requires therapy of the underlying tumor, glucocorticoids, and symptomatic use of drugs that enhance GABA-ergic transmission (diazepam, baclofen, sodium valproate, tiagabine, vigabatrin). IVIg and plasma exchange are transiently effective in some patients.

4ContinuationUntagged model output

Based on the information, it seems that muscle relaxants like dantrolene and baclofen are effective in treating muscle spasms and stiffness. Since the patient is already taking haloperidol, which is an antipsychotic medication, it is important to find a therapy that addresses the neck muscle stiffness without causing additional side effects. …

AnswerBaclofen

BRIDGE targets haloperidol-induced dystonia in its search and selects benztropine. VT-Search uses a broader query about muscle spasms and selects baclofen.

Correct trace: BRIDGE

1Reasoning<think>

To answer this question, I need to analyze each option and determine which one provides the most reasonable explanation for why a properly designed rest on the lingual surface of a canine is preferred over a rest on the incisal surface. I'll consider the biomechanical, anatomical, and aesthetic factors involved in dental restorations.

2Search<search>

Why is a properly designed rest on the lingual surface of a canine preferred to a rest on the incisal surface?

3Retrieved information<information>

Doc 1(Title: "The marginal-ridge rest seat.") Natural canine crowns are preferred as abutments for removable partial dentures because of their root morphology and bony support. However, preparing the rest seat on the lingual surface of a mandibular canine risks perforating the enamel. …

Doc 2(Title: "[Evaluation of stresses in different cingulum rest seats].") Both lingual and incisal rest seats are used on anterior teeth. Lingual rest seats are preferred since they are more acceptable esthetically and located closer to the centre of rotation of the tooth. …

Doc 3(Title: "Evaluation of prefabricated lingual rest seats for removable partial dentures.") An investigation into the feasibility of using prefabricated lingual rest seats bonded to mandibular canines and incisors with an orthodontic enamel bonding system was undertaken. 1. …

4ContinuationUntagged model output

From the search results, it is clear that the lingual surface of a canine is preferred for a rest because it is more esthetically acceptable and located closer to the center of rotation of the tooth. This reduces the leverage exerted against the tooth by the rest.

5Reasoning<think>

Based on the information gathered, the correct answer is: A: Less Leverage is exerted against the tooth by Lingual rest

AnswerA

Incorrect trace: VT-Search

1Reasoning<think>

To answer this question, I need to analyze the given options and determine which one provides the most reasonable explanation for why a properly designed rest on the lingual surface of a canine is preferred over a rest on the incisal surface. Option A: Less Leverage is exerted against the tooth by Lingual rest Option B: Enamel is thicker on lingual surface Option C: Visibility and access is better on lingual …

2Search<search>

A properly designed rest on lingual surface of a canine is preferred to a rest on incisal surface

3Retrieved information<information>

Doc 1(Title: "The marginal-ridge rest seat.") Natural canine crowns are preferred as abutments for removable partial dentures because of their root morphology and bony support. However, preparing the rest seat on the lingual surface of a mandibular canine risks perforating the enamel. …

Doc 2(Title: "Evaluation of prefabricated lingual rest seats for removable partial dentures.") An investigation into the feasibility of using prefabricated lingual rest seats bonded to mandibular canines and incisors with an orthodontic enamel bonding system was undertaken. 1. …

Doc 3(Title: "[Evaluation of stresses in different cingulum rest seats].") Both lingual and incisal rest seats are used on anterior teeth. Lingual rest seats are preferred since they are more acceptable esthetically and located closer to the centre of rotation of the tooth. …

4ContinuationUntagged model output

Based on the information, it seems that the correct answer is related to the distribution of stress and the location of the rest seat. The lingual surface is preferred because it is closer to the center of rotation of the tooth and provides a more uniform stress distribution. …

AnswerD

Both methods retrieve the same three passages in different orders. BRIDGE connects the rotation-center evidence to reduced leverage and selects A; VT-Search acknowledges that evidence but chooses the cingulum distractor, D.

Experimental Results

We evaluate BRIDGE on seven open-domain and five medical QA benchmarks, across six LLM backbones, including Base and Instruct variants at both 3B and 7B.

Main Results

BRIDGE reaches 41.5% overall and 34.5% multi-hop exact match with Qwen2.5-3B-Instruct, and 48.2% and 43.2% with Qwen2.5-7B-base. These exceed the second-best overall averages by 7.1 and 2.3 points, and the second-best multi-hop averages by 9.6 and 3.4 points, respectively.

Table 1 from the current BRIDGE paper, including Agentic-R with a 3B backbone: 19.5 multi-hop and 31.9 overall exact match, alongside all compared methods with Qwen2.5-3B and Qwen2.5-7B. — **Table 1. Evaluation on open-domain QA.** Bold and underlined entries denote the best and second-best results within each Qwen model block. Large-model references use direct inference without retrieval. M-Avg. averages HotpotQA, 2Wiki, MuSiQue, and Bamboogle; Avg. averages all seven benchmarks. †/∗ mark in-domain/out-of-domain benchmarks; ‡ marks published baseline results.BRIDGE achieves the best overall and multi-hop averages among the compared methods at both 3B and 7B.

BRIDGE with Qwen2.5-7B-base gains +24.3 multi-hop points and +20.6 overall points over GPT-5.6 Luna direct inference.

Table 2. BRIDGE, Search-R1, and the Agentic-R joint-training baseline across backbone and retriever configurations, with large-model direct-inference references. Exact match, percent.
Model / configuration	General QA			Multi-hop QA				M-Avg.	Avg.
Model / configuration	NQ	TriviaQA	PopQA	HotpotQA	2Wiki	MuSiQue	Bamboogle	M-Avg.	Avg.
Large-model references · direct inference without retrieval
GPT-4o	22.3	66.5	36.9	22.8	19.8	6.2	26.4	18.8	28.7
GPT-5.6 Luna	19.0	65.2	33.4	22.2	19.9	7.1	26.4	18.9	27.6
Qwen3-235B-A22B	23.0	58.4	28.9	23.2	29.1	4.9	20.8	19.5	26.9
Search-R1 · RL-only with fixed retrieval
Qwen2.5-3B-base	42.1	58.3	41.3	29.7	27.4	6.6	12.8	19.1	31.2
Qwen2.5-3B-Instruct	39.7	56.5	39.1	33.1	31.0	12.4	23.2	24.9	33.6
Qwen2.5-7B-base	48.0	63.8	45.7	43.3	38.2	19.6	43.2	36.1	43.1
Qwen2.5-7B-Instruct	42.9	62.3	42.7	38.6	34.6	16.2	40.0	32.4	39.6
Agentic-R · joint retriever–policy training baseline
Qwen2.5-3B-InstructE5 retriever	43.1	60.0	42.3	31.0	27.2	8.4	11.2	19.5	31.9
Qwen2.5-7BPublished results	42.4	69.0	44.1	45.8	45.3	20.3	48.0	39.8	45.0
BRIDGE · jointly trained retriever and policy
Qwen2.5-3B-base	47.3	63.4	46.0	40.2	40.1	14.3	30.4	31.3+12.4	40.3+12.7
Qwen2.5-3B-Instruct	46.3	61.6	44.7	42.3	41.4	16.8	37.6	34.5+15.6	41.5+13.9
Qwen2.5-7B-base	50.1	66.3	48.4	48.9	49.9	22.7	51.2	43.2+24.3	48.2+20.6
Qwen2.5-7B-Instruct	49.7	66.2	46.9	46.8	48.4	20.6	49.6	41.4+22.5	46.9+19.3
Qwen2.5-7B-baseQwen3-Embedding-8B retriever	48.9	67.3	50.0	48.6	47.0	22.9	52.0	42.6+23.7	48.1+20.5
DeepSeek-R1-Distill-Llama-8B	49.4	65.1	49.2	46.6	47.7	20.6	49.6	41.1+22.2	46.9+19.3
Qwen3-14B	51.8	70.0	51.0	49.5	52.5	23.3	56.8	45.5+26.6	50.7+23.1

Table 2. Backbones, retrievers, and large-model references. Scores are exact match (%); M-Avg. covers the four multi-hop datasets and Avg. covers all seven. Gains beneath M-Avg. and Avg. are percentage points over GPT-5.6 Luna, calculated from the displayed scores.Even with a 3B backbone, BRIDGE surpasses all 3 direct-inference references on 6 of 7 benchmarks, including all four multi-hop datasets.

Table 3: Medical QA results for Direct Inference, IRCoT (Instruct), RAG, s3, MedRAG, VT-Search, and BRIDGE on MedQA, MedMCQA, PubMedQA, BioASQ, and MMLU-Medical, with macro and micro averages. — **Table 3. Medical QA with Qwen2.5-7B-base.** Cells report GPT-5.6-Luna-judge reasoning quality with exact match in parentheses (both 0–100). Macro and micro averages weight benchmarks and questions equally. Bold and underlined values mark the best and second-best scores per metric.BRIDGE leads reasoning quality on all five medical QA benchmarks and exact match on three.

In-Depth Analysis

We first isolate what drives the accuracy gains, then examine estimator fidelity, retriever-update cost, and training efficiency.

Component ablation: BRIDGE reaches 31.3 multi-hop and 40.3 overall EM, ahead of RAG-then-RL, Decoupled-RL, Decoupled-RAG, VT-Search, and Search-R1. — Component ablationBRIDGE gains 7.5 multi-hop points over sequential RAG-then-RL.
Qwen2.5-3B-base; G = 8.

Detailed ablation results

Table 4: Retriever objective and update coupling ablation comparing VT-Search, Decoupled-RL, Decoupled-RAG, RAG-then-RL, and BRIDGE. — **Table 4. Retriever objectives and update coupling.** All variants use Qwen2.5-3B-base and eight rollouts. These controlled ablations use the Base model, distinct from the 3B-Instruct result in Table 1. M-Avg. and Avg. denote multi-hop and overall QA averages; arrows show point gains over the VT-Search row in this table (21.7 multi-hop, 34.7 overall).Combined retrieval credit and online co-adaptation deliver the largest multi-hop gain: 9.6 points over VT-Search.

Figure 4(a): Cosine-similarity distributions for BRIDGE and Drop relative to the full two-state gradient estimator. — Figure 4(a,b) · Estimator fidelityOne retriever state closely matches the full estimator.
At γ = 0.02, BRIDGE achieves 0.998 cosine similarity and 0.048 relative error against the full two-state estimator. Dropping only the value-function term (Drop) yields 0.183 and 14.756, respectively.

Figure 4(b): Relative gradient error across penalty weights, showing BRIDGE below Drop by more than two orders of magnitude. — Figure 4(a,b) · Estimator fidelityOne retriever state closely matches the full estimator.
At γ = 0.02, BRIDGE achieves 0.998 cosine similarity and 0.048 relative error against the full two-state estimator. Dropping only the value-function term (Drop) yields 0.183 and 14.756, respectively.

Retriever update cost: BRIDGE uses 31.3 GiB peak allocated memory and 46.2 seconds, versus 62.3 GiB and 104.5 seconds for the full two-state bilevel estimator. — Retriever-update cost49.7% less peak memory and 55.8% less time per retriever update.
Relative to the full two-state estimator: 62.3 to 31.3 GiB and 104.5 to 46.2 seconds.

Raw, unsmoothed training rewards over steps 1–150 for BRIDGE and the official Search-R1 Qwen2.5-7B-base runs. — Training dynamicsBRIDGE reaches higher training rewards in fewer steps than Search-R1.

Multi-hop QA accuracy for BRIDGE and VT-Search under different rollout budgets. — Compute allocationRetriever adaptation beats more rollouts.
On 4 × A100 GPUs, with Qwen2.5-3B-base and the same 250-step budget, BRIDGE with eight rollouts reaches 31.3 M-Avg. in 79.4 hours; VT-Search with sixteen rollouts reaches 23.3 in 82.4 hours.

Takeaway. BRIDGE is especially effective on retrieval-intensive tasks and when reasoning capabilities are limited. Controlled ablations show that the result depends on combining outcome credit with RAG likelihood and updating the retriever online—not simply adding more rollouts or tuning either component in isolation.

Citation

Build on BRIDGE.

Read the paperCode is coming soon

BibTeX

arXiv:2609.36505

@article{xiao2026bridge,
  title={{BRIDGE}: Bilevel Retrieval-Credit-Aware Agentic Reinforcement Learning},
  author={Xiao, Quan and Liu, Mingda and Liu, Gaowen and Fujisawa, Katsuki and Chen, Tianyi},
  journal={arXiv preprint arXiv:2609.36505},
  year={2026}
}

BRIDGE

Bilevel Retrieval-Credit-Aware Agentic Reinforcement Learning

TL;DR

Introduction

What causes brown tumors?

Candidate passages

RL outcome reward

RAG likelihood

Which company’s advertisements featured the actor playing Daryl in “Cherokee Rose”?

Step 50Early checkpoint

Step 250Later checkpoint

Tool-use rollout

Preview of BRIDGE’s learning hierarchy

LLM policy

Query-side retriever

Method

Generate fresh rollouts

Adapt the retriever

Update the policy

Estimator geometry

Policy training dynamics

Case Study

Correct trace: BRIDGE

Incorrect trace: VT-Search

Correct trace: BRIDGE

Incorrect trace: VT-Search

Correct trace: BRIDGE

Incorrect trace: VT-Search

Correct trace: BRIDGE

Incorrect trace: VT-Search

Correct trace: BRIDGE

Incorrect trace: VT-Search

Correct trace: BRIDGE

Incorrect trace: VT-Search

Correct trace: BRIDGE

Incorrect trace: VT-Search

Correct trace: BRIDGE

Incorrect trace: VT-Search

Correct trace: BRIDGE

Incorrect trace: VT-Search

Experimental Results

Main Results

In-Depth Analysis

Citation

Bilevel Retrieval-Credit-Aware
Agentic Reinforcement Learning