Editorial illustration for Expert Rubrics Guide RL Rewards for Non-Verifiable Retrieval
Expert Rubrics Boost LLM Retrieval by 6.7% in Finance
Expert Rubrics Guide RL Rewards for Non-Verifiable Retrieval
A new paper puts a number on a problem that's been nagging at anyone trying to train LLM agents for financial services work: Recall@3 improves by up to 6% and MRR@3 by 6.7% once researchers fix how reward signals get built for retrieval tasks that don't have a clean right answer.
The setting is financial retrieval, where an agent has to read what a user actually wants, rewrite the query, and pull back results under constraints specific to that domain. Reinforcement learning has mostly proven itself on tasks where a program can check the answer and hand out a binary reward. Financial query rewriting doesn't work that way. Judging whether a query is good or bad takes a subject-matter expert, and that expertise is both expensive and limited in supply.
The researchers built their training signal around Group Relative Policy Optimization, a method increasingly used for agent training. GRPO has a known failure mode in sparse-reward setups, which the authors ran into directly while working on retrieval quality in a domain where expert judgment can't be automated away. Their fix reframes how far a small amount of expert input can stretch across a training run.
In non-verifiable retrieval tasks, assessing query quality requires scarce subject-matter expert (SME) judgment that is inherently graded rather than binary. With Group Relative Policy Optimization (GRPO), sparse rewards yield no useful learning signal when all sampled outputs within a group receive the same outcome, causing optimization to stall.
Why this matters GRPO's blind spot is practical, not theoretical: when every sampled response in a group scores the same, the method learns nothing, and that's exactly what happens in domains like financial retrieval where there's no ground-truth answer to check against. Turning scarce SME judgment into dense, graded rubrics is a workaround, not a solved problem. It still depends on experts who are expensive and hard to scale, which means teams building agents for compliance-heavy or judgment-heavy fields should expect rubric design and maintenance to become its own engineering discipline, not a one-time setup cost.
For founders pitching RL-trained agents outside math and code benchmarks, this is a reminder that "reinforcement learning" doesn't automatically mean cheap or automated. Someone still has to define what "good" looks like, in writing, repeatedly, as the task drifts. We'd watch for how well these rubrics transfer across adjacent non-verifiable tasks. If every new domain needs its own expert-built scoring scheme from scratch, the approach solves today's sparsity problem while creating tomorrow's labeling bottleneck.
Related Reading
Industry Applications Westinghouse teams with Google Cloud to build AI platform for nuclear power
Industry Applications NVIDIA NeMo powers telco reasoning model for autonomous network workflows
Industry Applications Month-1 Agent Adds Holistic Observability with Trace IDs and Token Tracking
From Archives Teaching AI about symmetry could make prenatal MRI diagnosis more reliable
From Archives Cars.com and Amtrak Block AI Agents as Websites Quietly Close Doors
Common Questions Answered
How much does Recall@3 and MRR@3 improve when using expert rubrics for non-verifiable retrieval tasks?
According to the research, Recall@3 improves by up to 6% and MRR@3 by 6.7% once researchers properly construct reward signals for retrieval tasks without clean right answers. These improvements demonstrate the significant impact of using expert-guided rubrics in financial retrieval scenarios where traditional binary evaluation methods fail.
What is the main limitation of Group Relative Policy Optimization (GRPO) in non-verifiable retrieval tasks?
GRPO's primary limitation is that sparse rewards yield no useful learning signal when all sampled outputs within a group receive the same outcome, causing optimization to stall. This blind spot is particularly problematic in domains like financial retrieval where there is no ground-truth answer to check against, making it difficult for the reinforcement learning model to improve.
Why is subject-matter expert (SME) judgment necessary for training LLM agents in financial services retrieval?
In non-verifiable retrieval tasks, assessing query quality requires scarce subject-matter expert judgment because there is no binary right or wrong answer. Experts provide graded, nuanced evaluations that can be converted into dense reward signals, enabling reinforcement learning models to learn effectively in compliance-heavy domains where traditional verification methods don't apply.
What are the scalability challenges of using expert-guided rubrics for financial retrieval agents?
While expert-guided rubrics effectively convert scarce SME judgment into dense reward signals, this approach still depends on expensive and hard-to-scale expert resources. This dependency creates a practical bottleneck for teams building agents for judgment-heavy or compliance-heavy domains, as the method cannot be easily scaled without proportionally increasing expert involvement.
Further Reading
- Finance Agent Benchmark: Benchmarking LLMs on Real-world Financial Research Tasks - arXiv
- FinAgentBench: A Benchmark Dataset for Agentic Retrieval in Financial Question Answering - arXiv
- Reinforcement Learning with Robust Rubric Rewards - arXiv
- From Holistic Evaluation to Structured Criteria: Rubrics Across the RL Pipeline - arXiv
- FinanceHarness: Autonomous Financial Deep Research - arXiv
