Editorial illustration for OpenAI, Google AI models easily tricked into making nudes, researchers say
AI Models Tricked Into Generating Nude Images
OpenAI, Google AI models easily tricked into making nudes, researchers say
Google's own researchers spent part of this summer surveying internet users in the United States, Australia and the United Kingdom about AI-generated explicit images. The results, published in July, pointed to a problem that mostly festers out of sight: non-consensual content traded over text, slipped into direct messages, or buried in private cloud storage where nobody reports it. Google's conclusion was blunt. The "most critical intervention point" isn't moderation after the fact, it's the generation tools themselves.
That finding lands awkwardly given what's happening on the other side of the industry. As OpenAI and Google push their image generators to be more capable, outside testers say the guardrails meant to block explicit content are getting weaker, not stronger. The scrutiny comes from White Circle, a public benefit corporation backed by people with ties to Anthropic, DeepMind, Hugging Face and OpenAI.
Its CEO, Denis Shilov, built his reputation by posting one of the most widely cited jailbreaks of a major AI model back in October 2024. White Circle now spends much of its time trying to break models before bad actors do, with text-to-image filters as a particular focus. Shilov says what his team just found is "horrifying" enough to put in front of the public.
In a little-noticed preprint published last month, White Circle researchers exposed flaws in the guardrails of DALL·E 3, Google’s Nano Banana 2, and GPT-Image-2. In a press release, they explained that “using just off-the-shelf tools, available online for anyone to download, and by sending API calls, White Circle was able to use the novel attack method to bypass the safety filters” and generate explicit content about 13 percent of the time.
Why this matters
White Circle's testing shows a gap most safety marketing glosses over: generation-time filters are not the same as deployment-time guarantees. OpenAI and Google have both built refusal systems for explicit content, and both get bypassed with what researchers describe as minimal effort. For developers building on these models, that's a direct warning.
A safety claim in a model card is not a safety property you can inherit by default. You need your own red-teaming before you ship anything that touches image generation, especially if your product lets users upload reference photos or chain prompts.
The distribution angle matters just as much. Google's own July study found this content mostly moves through DMs and private cloud storage, not public feeds, which is exactly where moderation systems are weakest and regulators have the least visibility. If the fix has to happen before generation rather than after sharing, that puts real pressure on model providers rather than platforms.
Founders pitching "safe by design" image tools should expect buyers and regulators to start asking for evidence, not assurances. This is a testing problem, not a policy press release problem.
Related Reading
Browse all OpenAI news →
Research & Benchmarks Microsoft AI Chief: "If It Isn't Safe We Shouldn't Build It
Research & Benchmarks Google's FACTS benchmark shows 70% factuality ceiling across four tests
Research & Benchmarks Nvidia's DLSS 4.5 beta adds 6x Multi Frame Generation for RTX 50 GPUs
Related Topic OpenAI researcher quits, citing distrust over ad‑driven engagement metrics
LLMs & Generative AI
Related Topic OpenAI launches GPT-Image 1.5 with precise editing for enterprise visuals
LLMs & Generative AI
From Archives Harvard: men more optimistic and confident about AI than women
From Archives Gemini 4 Argon review: GraphWalks F1, DeepSWE, Vibe Code Bench
From Archives OpenAI employee allegedly used chatbot to plant recording device at office
AI Tools & Apps
From Archives OpenAI employee arrested for allegedly planting bug in conference room
Common Questions Answered
What specific AI models were found vulnerable to generating explicit content by White Circle researchers?
White Circle researchers exposed flaws in the guardrails of DALL·E 3, Google's Nano Banana 2, and GPT-Image-2. Using off-the-shelf tools and API calls, they were able to bypass the safety filters and generate explicit content approximately 13 percent of the time.
What did Google's researchers conclude about preventing non-consensual AI-generated explicit images?
Google's researchers surveyed internet users in the United States, Australia, and the United Kingdom about AI-generated explicit images and concluded that the 'most critical intervention point' isn't moderation after the fact, but rather prevention at the generation stage. This finding highlighted that post-generation moderation is insufficient for addressing the problem of non-consensual content.
Why is there a gap between AI safety marketing claims and actual deployment-time guarantees?
White Circle's testing demonstrates that generation-time filters are not the same as deployment-time guarantees, meaning safety claims in model cards are not automatically inherited by default. Developers building on these models need to conduct their own red-teaming before shipping products, as OpenAI and Google's refusal systems for explicit content can be bypassed with minimal effort.
How were researchers able to bypass the safety filters of these AI image generation models?
Researchers used just off-the-shelf tools available online for anyone to download and sent API calls using a novel attack method to bypass the safety filters. This accessibility demonstrates that the vulnerability to generating explicit content does not require sophisticated hacking techniques or specialized knowledge.
Further Reading
- OpenAI, Google pushing new AI models easily tricked into making nudes, researchers say - Ars Technica
- Low-Effort Jailbreak Attacks Against Text-to-Image Safety Filters - arXiv
- Study finds 7 of 9 Hugging Face image models made deepfake nudes - Dataconomy
- A Multi-Region Survey on AI-Generated Image-Based Sexual Abuse Victimization - Google Research
- Child Predators Turn to Open-Source AI to Make Unlimited Illegal Images - Bloomberg
