Editorial illustration for Estonian institute benchmarks AI models' vulnerability to Russian propaganda
Estonian institute benchmarks AI models' vulnerability...
The Institute of the Estonian Language just graded sixty AI models on a critical new subject: Kremlin propaganda. Their benchmark, published today, hits models from global giants and open-source projects with 75 questions in Estonian, Russian, and English. Each query tests one of 14 common Russian narratives, phrased neutrally, with bias, or with outright manipulation.
Every answer got a score from 1 to 5—a 1 means the model parroted the line. To ensure those scores were sound, the institute used a calibrated Claude Opus 4.5 for evaluation, a method then validated by analysts at the Estonian watchdog Propastop. The full rankings and detailed methodology are now live on the institute's site.
Related Reading
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google tests visual 'magazine-style' UI for Gemini 3 Pro users
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
From Archives Study quantifies AI agent trust formation, breakage, recovery in survival game
From Archives UP‑NRPA Allows Dynamic Customization of Dialogue Strategies Without Offline RL
Common Questions Answered
How many AI models did the Institute of the Estonian Language evaluate in their propaganda benchmark?
The Institute of the Estonian Language benchmarked sixty AI models from both global giants and open-source projects. This comprehensive evaluation tested each model's vulnerability to Kremlin propaganda across multiple languages and narrative types.
What languages and types of Russian narratives were included in the Estonian institute's propaganda benchmark?
The benchmark consisted of 75 questions presented in Estonian, Russian, and English, with each query testing one of 14 common Russian narratives. These narratives were phrased in three different ways: neutrally, with bias, and with outright manipulation to comprehensively assess model vulnerabilities.
How did the Institute of the Estonian Language score AI model responses to propaganda questions?
Each answer received a score from 1 to 5, where a score of 1 indicated the model simply repeated the propaganda line without critical analysis. To ensure scoring accuracy, the institute used a calibrated Claude Opus 4.5 for evaluation, with results then validated by analysts at the Estonian watchdog organization Propastop.
Where can researchers access the full rankings and methodology from the Estonian propaganda benchmark study?
The full rankings and detailed methodology from the Institute of the Estonian Language's propaganda benchmark are now publicly available on the institute's website. This transparency allows researchers and organizations to understand how different AI models performed and the specific evaluation criteria used.
Further Reading
- Estonian study finds AI models still vulnerable to propaganda prompts — ERR News
- EKI and Propastop Studied AI Resistance to Propaganda — Propastop
- These LLMs are the best at resisting Russian propaganda — Ars Technica
- The Estonian government has released a benchmark to determine ... — GIGAZINE
- Europe's AI champion Mistral vulnerable to Russian disinformation, study finds — Financial Times
