Editorial illustration for Metric-Dependent Annotation Saturation for Learning from Label Distributions
Metric-Dependent Annotation Saturation for Learning from...
We treat annotator disagreement like static to filter out. Maybe we're filtering out the point. A new study asks how many people you actually need to pin that signal down.
The answer isn't one number. It depends entirely on what you're measuring.
Researchers fine-tuned NLI models using subsampled judgments from ChaosNLI, a dataset with 100 independent annotations per item. They found a metric-dependent pattern. If you want a model to identify which items spark disagreement, measured by entropy correlation, you need 20 to 50 annotators for the signal to stabilize.
If you just want the overall distribution of labels to match, measured by KL divergence, you can stop at about 10. That smaller pool captures 87 to 95 percent of the total improvement. This builds on the fact that real soft labels carry item-specific information.
Popular techniques like label smoothing can't replicate that.
Metric-Dependent Annotation Saturation for Learning from Label Distributions AuthorsGuneet Kohli Metric-Dependent Annotation Saturation for Learning from Label Distributions AuthorsGuneet Kohli When annotators disagree on a label, the disagreement itself carries signalâand the number of annotators needed to capture it depends on the evaluation metric. We fine-tune NLI models on label distributions subsampled from ChaosNLI, a dataset providing 100 independent annotator judgments per item, and identify metric-dependent saturation. In our 3-class NLI setting, entropy correlationâwhether the model identifies which items elicit disagreementârequires N â 20â50 annotators to converge, while distributional match (KL divergence) saturates by N â 10 (87â95% of improvement across five model seeds). This finding rests on a prior observation: soft labels carry item-specific signal that label smoothing cannot replicate.
This changes the budget conversation. Annotation isn't a one-size-fits-all line item. It's a variable cost tuned to your evaluation metric.
Want to capture the shape of human uncertainty? That's expensive. You're paying for fifty judgments to map ambiguity.
Just need the crowd's average vote? Ten people will do.
Label smoothing offers a cheap, synthetic substitute for that uncertainty. It applies the same bland blur to every item. Real disagreement is textured and specific.
You cannot smooth your way into it. The takeaway is practical. Define your metric first.
Then buy just enough human judgment to satisfy it. Anything more is wasted money. Anything less is a broken model.
Related Reading
Research & Benchmarks Google's FACTS benchmark shows 70% factuality ceiling across four tests
Research & Benchmarks Databricks finds multi-step agents beat single-turn RAG by 21% to 38% on STaRK
Research & Benchmarks Nvidia's DLSS 4.5 beta adds 6x Multi Frame Generation for RTX 50 GPUs
From Archives DFlash speculative decoding boosts NVIDIA Blackwell inference up to 15×
From Archives NVIDIA BioNeMo Toolkit Enables AI Scientist to Align, Fold, and Dock Molecules
Common Questions Answered
What is metric-dependent annotation saturation and why does it matter for NLI models?
Metric-dependent annotation saturation refers to the finding that the number of annotations needed depends entirely on what evaluation metric you're using, rather than being a fixed number. Researchers fine-tuned NLI models using subsampled judgments from ChaosNLI and discovered this pattern, showing that annotation requirements vary significantly based on your specific measurement goals.
How many annotators do you need if you want to capture human disagreement versus just get the crowd's average vote?
If you want to capture the shape of human uncertainty and identify which items spark disagreement, you need approximately fifty independent judgments per item to map ambiguity accurately. However, if you only need the crowd's average vote for a simpler metric, ten annotators will suffice, making it a much more cost-effective approach.
What is the difference between using real annotator disagreement and label smoothing as a substitute?
Real disagreement from multiple annotators is textured and specific to individual items, capturing the nuanced uncertainty in the data. Label smoothing, by contrast, applies the same bland blur to every item synthetically, which is cheaper but fails to capture the genuine patterns of human disagreement that vary across different examples.
Why should annotation be treated as a variable cost rather than a fixed line item in machine learning budgets?
Annotation should be treated as a variable cost because the required number of judgments depends entirely on your evaluation metric and what you're trying to measure. This means organizations can optimize their annotation budgets by using fewer annotators for simpler metrics while investing more in annotations when capturing complex human uncertainty is essential.
Further Reading
- Metric-Dependent Annotation Saturation for Learning from Label Distributions — Semantics Scholar
- [Literature Review] Metric-Dependent Annotation Saturation for Learning from Label Distributions — The Moonlight
- Label Distribution Learning with Biased Annotations Assisted by Multi-Hot Degeneration — IJCAI 2025
- A Unimodal-Weighted Label Distribution Learning Approach — IEEE Transactions
- Label Distribution Learning on Auxiliary Label Space Graphs for Enhanced Annotation — YouTube (Research Presentation)
