Editorial illustration for Watermark Alters AI's Internal Probability Numbers
Watermarks Change How AI Models Calculate Probabilities
Watermark Alters AI's Internal Probability Numbers
Every large language model runs on a bet. At each step it calculates a probability distribution over the next possible token, and most of the time that bet is safe. Ask for the capital of France and "Paris" claims nearly all the probability mass.
There's no real decision happening. But ask why a character left a party, and the model faces a spread of equally plausible answers, forcing it to pick one almost at random. That spread has a name: entropy.
Regulators have built two separate tools around this same mechanic. The EU AI Act's Article 50 and China's Cyberspace Administration both require watermarking, a hidden signature baked into generated text so its AI origin can be proven later. Separately, hallucination detection tools try to flag the moments a model is guessing rather than reporting.
Both systems, it turns out, watch the exact same signal inside the model: how uncertain it is at any given step. Push on one, and you're pushing on the other. That's the collision nobody designing these rules seems to have accounted for.
AI safety has a friendly-fire problem. The technology that proves where AI text came from is weakening the technology that checks whether it is true. Watermarking, or embedding a hidden signature in generated text so that its origin can be proven later, is used to increase transparency on AI applications (Europe’s EU AI Act Article 50
Why this matters
The industry has been treating watermarking and hallucination detection as two separate compliance boxes to check, one for provenance, one for accuracy. This research shows they're fighting over the same real estate: the token-by-token moments where a model is least certain. Nudge those probability numbers to embed a signature, and you've also nudged the exact signal that uncertainty-based safety checks rely on to flag a bad answer. That's a design collision, not a bug you patch with a software update.
For teams shipping under the EU AI Act's Article 50 disclosure rules, this means watermarking and hallucination detection can't be bolted on independently and assumed to coexist quietly. Anyone building or auditing detection pipelines needs to ask which layer runs first, and what it does to the numbers the next layer depends on. If your hallucination checker reads logits that a watermarking scheme already altered, you're not measuring what you think you're measuring. That's worth testing before regulators or customers find it for you.
Related Reading
Policy & Regulation After ditching AI fitness apps and a Fitbit, I return to Peloton classes
Policy & Regulation Grammarly faces class-action suit over AI ‘Expert Review’ feature
Policy & Regulation Akamai Proposes SAFE Cybersecurity Guidelines for AI Threats
From Archives Meta AI Glasses Sales Surge Raises Covert Recording Concerns
From Archives Amazon Destroys Rare Books for AI Training, Tracking Device Shows
Common Questions Answered
How does watermarking interfere with hallucination detection in large language models?
Watermarking embeds hidden signatures into generated text by altering the model's internal probability numbers at each token step. This process directly conflicts with hallucination detection systems that rely on the same probability signals to identify uncertain or false outputs, creating a design collision where improving one safety measure weakens the other.
What is entropy in the context of large language model decision-making?
Entropy refers to the spread of equally plausible answers when a language model faces multiple valid options for the next token. For example, when asked why a character left a party, the model encounters high entropy because many answers are equally probable, forcing it to pick one almost at random rather than making a clear decision.
Why does the EU AI Act Article 50 require watermarking in AI applications?
Watermarking is used to increase transparency and prove the origin of AI-generated text by embedding a hidden signature that can be verified later. This requirement helps regulators and users identify content that was created by AI systems, supporting compliance with transparency standards outlined in the EU AI Act.
What is the fundamental problem with treating watermarking and hallucination detection as separate compliance requirements?
The research reveals that watermarking and hallucination detection are not independent systems but rather compete for the same resource: the token-by-token moments where a model is least certain. Adjusting probability numbers to embed a watermark signature simultaneously weakens the uncertainty-based signals that safety checks depend on to flag inaccurate or false answers.
Further Reading
- A Watermark for Large Language Models - arXiv
- Watermarking Large Language Models: An Unbiased and Low-risk ... - ACL Anthology
- Watermarking Makes Language Models Radioactive - NeurIPS Proceedings
- A Statistical Framework of Watermarks for Large Language Models - arXiv
- Undetectable Watermarks for Language Models - IACR ePrint
