Editorial illustration for Cracking AI's 'Random' Number: Why One Guess Is Not Enough, Toy Case Shows
AI's 'Random' Numbers Aren't Random at All
Cracking AI's 'Random' Number: Why One Guess Is Not Enough, Toy Case Shows
Ask a chatbot for a random number between 1 and 100, and the answer won't be random at all. One researcher decided to find out exactly how predictable it really is, not by running a model a thousand times and tallying results, but by reading the raw probability the model assigns to every possible answer, straight from its logits. No sampling noise, no guesswork about frequency, just the actual odds baked into the model before it ever generates a token.
That distinction matters because the usual write-ups on this topic count outputs after the fact. This approach flips the question into something closer to a security audit: if a model's "random" pick is standing in for a secret, how many guesses would it take an attacker to crack it? The researcher ran this test across base models and their instruction-tuned counterparts, comparing the two to see whether the bias toward certain numbers comes from pretraining itself or gets baked in during fine-tuning. The results, converted into bits of entropy and expected guesses-to-crack, show just how far these models are from anything resembling a fair hundred-sided die.
The extreme case is Qwen2.5-1.5B-Instruct with the raw prompt. It puts 74% of its probability on 47 and 94% on its top three numbers. An attacker who guesses 47 first is right three times in four and needs 1.9 guesses on average, against 50.5 for a fair pick, and the model behaves like a fair pick among two or three numbers.
Why this matters
For anyone building on top of these models, averages lie. Shannon entropy told us the toy model's output looked like 20 fair coin flips worth of randomness. Min-entropy told us the truth: one guess, 73, wins half the time.
That gap matters whenever a model's "randomness" feeds into something security-relevant, a password seed, a CAPTCHA-style challenge, a game mechanic meant to be unpredictable. If we're only checking entropy on average, we're measuring the wrong thing and getting fooled by it.
The fix here isn't exotic. Read the logits, not just the sampled text. A model that hands you its actual probability distribution over answers is giving you the min-entropy for free, no guesswork required.
Teams fine-tuning instruction models for anything touching randomness should be running this kind of check before deployment, not after someone exploits the skew toward 73 or whatever number their model happens to favor. A base model and its instruction-tuned sibling can also diverge heavily here, which is its own worthwhile thing to watch as fine-tuning pipelines multiply.
Related Reading
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google to End Free Access to Gemini AI for Non-Subscribers
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
From Archives YouTube adds "Made with AI" labels to altered or fully generated songs
From Archives Google moves AI teams from DeepMind as focus shifts from London to California
Common Questions Answered
Why is examining raw logits more accurate than sampling multiple times when testing AI randomness?
Examining raw logits reveals the actual probability distribution the model assigns to each possible answer without sampling noise or frequency estimation errors. By reading the probabilities directly from the model before token generation, researchers get the true odds baked into the model rather than approximations based on repeated sampling attempts.
What does the Qwen2.5-1.5B-Instruct model's behavior reveal about AI number generation predictability?
The Qwen2.5-1.5B-Instruct model puts 74% of its probability on the number 47 when asked for a random number between 1 and 100, meaning an attacker guessing 47 first would be correct 75% of the time. This extreme bias demonstrates that the model's output is far from random, requiring only 1.9 guesses on average compared to 50.5 for a truly fair selection.
How does Shannon entropy differ from Min-entropy in measuring AI randomness security?
Shannon entropy measures average randomness across all possible outcomes and may suggest a model's output has adequate randomness, while Min-entropy reveals the worst-case scenario by identifying the most likely outcome. In the toy model example, Shannon entropy indicated 20 fair coin flips worth of randomness, but Min-entropy showed that one guess wins half the time, exposing the actual security vulnerability.
Why is AI randomness predictability a concern for security-relevant applications?
When a model's "randomness" feeds into security-critical functions like password seeds, CAPTCHA challenges, or game mechanics meant to be unpredictable, biased probability distributions create exploitable vulnerabilities. If developers only check average entropy rather than worst-case probability, they measure the wrong metric and fail to detect predictable patterns that attackers can exploit.
Further Reading
- Deterministic or probabilistic? The psychology of LLMs as random number generators - arXiv
- AIs have a favorite number, and it's not 42 - The Register
- Base Models Beat Aligned Models at Randomness and Truthfulness - arXiv
- Evaluating the Quality of Randomness and Entropy in Tasks Performed by Large Language Models - arXiv
- Large Language Models Are Bad Dice Players: LLMs Struggle to Generate Random Numbers from Statistical Distributions - alphaXiv
