Editorial illustration for NanoJev Handles Changing Candidate Lists, But Choosing Well Is Separate
NanoJev Outpaces LLMs on Dynamic Candidate Routing
NanoJev Handles Changing Candidate Lists, But Choosing Well Is Separate
Seven decision models just went up against a generative LLM baseline, and the gap in both speed and accuracy is wide enough to notice. The Jev repo team ran Qwen3-0.6B against models like Kev and Verdict on two tasks: routing requests across fixed options and handling candidate lists that change between requests. Kev hit 98% accuracy on both setups.
Verdict reached 89% on the changing-candidate panel, well ahead of the LLM baseline's 66% on the same test. Latency told a similar story. Kev and Verdict produced zero generated answer tokens per request, compared to roughly 10 for Qwen's JSON generation.
Median request times came in at 125-130 ms for Kev and as low as 17-18 ms for Verdict, against 331-357 ms for the LLM baseline.
The comparison grew out of a question raised in an earlier piece in this series: why ask a language model to generate an answer one token at a time when the software only needs to pick from a small set of options. A banking support case, like a customer reporting a duplicate charge, is exactly that kind of bounded choice. The numbers above show what happens when seven decision models take that job instead of a generative model.
Kev led on accuracy; Verdict also outperformed the LLM baseline. Kev reached 98% on both fixed routes and changing candidates. Verdict reached 89% on changing candidates, compared with 66% for LLM (baseline) Qwen3-0.6B, which reached 83% on fixed routes.
Why this matters
The NanoJev results are a useful reminder that architecture and competence are different problems. Swapping a generative decoder for a scoring model that returns a typed decision solves a real engineering headache: no more parsing free text to figure out what an agent meant to choose. That's worth something on its own.
But 17% accuracy on banking requests, from a checkpoint trained on games, tells us the interface change didn't come with any transfer of judgment. NanoJev can accept a new candidate list without retraining. It just doesn't know what to do with it.
For anyone building agent systems, the lesson is to stop conflating "can handle dynamic inputs" with "makes good choices on them." Those are separate benchmarks, and vendors (or repos) that blur that line deserve a second look. If you're evaluating decision models for routing or tool selection, demand accuracy numbers on your actual domain, not just architecture diagrams. The format is promising. The judgment still has to be earned, likely through domain-specific training, not assumed from a clever interface.
Related Reading
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google to End Free Access to Gemini AI for Non-Subscribers
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
From Archives Apple Silicon Mac Studio Cluster Benchmarked for Distributed LLM Inference
From Archives Cracking AI's 'Random' Number: Why One Guess Is Not Enough, Toy Case Shows
Common Questions Answered
How did Kev and Verdict perform compared to the Qwen3-0.6B LLM baseline on changing candidate lists?
Kev achieved 98% accuracy on both fixed routes and changing candidates, significantly outperforming the LLM baseline. Verdict reached 89% accuracy on the changing-candidate panel, which was substantially ahead of the LLM baseline's 66% accuracy on the same test, demonstrating the superior performance of decision models in handling dynamic candidate scenarios.
What is the key difference between using a generative decoder versus a scoring model in NanoJev?
Swapping a generative decoder for a scoring model that returns a typed decision eliminates the engineering challenge of parsing free text to determine what an agent intended to choose. This architectural change provides a real engineering improvement by removing ambiguity in decision outputs, though it does not automatically transfer judgment capabilities from the model.
Why does the article emphasize that architecture and competence are different problems in the NanoJev results?
The article demonstrates that while NanoJev solved the architectural problem of interface clarity, it did not solve the competence problem of making accurate decisions. For example, a checkpoint trained on games showed only 17% accuracy on banking requests, proving that changing the interface structure alone does not guarantee transfer of judgment or domain expertise to new tasks.
What were the two main tasks that the decision models were tested on in the Jev repo comparison?
The decision models were tested on routing requests across fixed options and handling candidate lists that change between requests. These two tasks allowed researchers to evaluate how well the models performed in both stable scenarios with predetermined choices and dynamic scenarios where available options varied across different requests.
Further Reading
- DecisionBench and Bosun v3.1: building and testing decision models - Hugging Face
- NanoJev: A nano replica of Jev - GitHub
- NanoJev Review: A 0.6B Open Jev Replica - MrJev
- Jev 深度解析：不生成文字的模型，凭什么爆火 - GitHub
- Lev alternatives: NanoJev and other decision models - GitHub
