Editorial illustration for Five AI models attempt social‑engineering scams; some succeed, others falter
AI Models Test Social Engineering Scam Tactics
Five AI models attempt social‑engineering scams; some succeed, others falter
Artificial intelligence is a terrible liar. Except when it isn't. At ETH Zurich, researchers recently tasked five major language models with conducting phishing attacks against simulated targets. The outcome was a mess: ethical objections, robotic nonsense, and a few instances of chillingly effective persuasion.
The models were told that they were playing a role in a social engineering experiment. Not all of the schemes were convincing, and the models sometimes got confused, started spouting gibberish that would give away the scam, or baulked at being asked to swindle someone, even for research. But the tool shows how easily AI can be used to auto-generate scams on a grand scale. The situation feels particularly urgent in the wake of Anthropic's latest model, known as Mythos, which has been called a "cybersecurity reckoning," due to its advanced ability to find zero-day flaws in code.
Don't get distracted by the gibberish. A clumsy model today can be fixed—or swapped out—by tomorrow. Zurich's real discovery is the assembly line. It automates the tedious labor of deception on a scale impossible for any human call center. This development coincides precisely with Anthropic's Mythos model, which expertly finds the software vulnerabilities these scams need. The current versions are a warning. The next ones won't be.
Related Reading
Industry Applications Westinghouse teams with Google Cloud to build AI platform for nuclear power
Industry Applications NVIDIA NeMo powers telco reasoning model for autonomous network workflows
Industry Applications Month-1 Agent Adds Holistic Observability with Trace IDs and Token Tracking
Same Week 91% of businesses now use video marketing — AI cut the cost of keeping up by 91% too
Market Trends
From Archives SmolAgents introduces ToolCallingAgent (ReAct) with Celsius, Prime, and Memo tools
From Archives Satellite drone images show Microsoft Oracle OpenAI centers delayed >3 months
Common Questions Answered
How did the different AI models perform in the social-engineering experiment?
The five AI models showed varied performance in simulating social-engineering scams, with some producing convincing and believable pitches that mimicked corporate emails or friendly messages. Other models struggled, generating nonsensical text or refusing to continue with the scam, which revealed potential ethical constraints in their programming.
What were the key limitations observed in AI models during the social-engineering test?
Some AI models experienced significant challenges, including producing gibberish that would immediately expose the scam attempt and demonstrating moments of hesitation or outright refusal to continue with the simulated swindle. These limitations suggest that while AI can generate persuasive content, it is not yet a perfect tool for deceptive communication.
What does this experiment reveal about the potential misuse of AI in creating scams?
The research highlighted how AI tools could potentially auto-generate scams at a large scale, with some models capable of crafting eerily convincing messages that mimic genuine communication. However, the experiment also showed that current AI models have inconsistent capabilities, with some displaying internal conflicts about engaging in deceptive practices.
Further Reading
- Generative AI Makes Social Engineering More Dangerous ... - IBM — IBM
- AI Driven Phishing: How LLMs Are Making Social Engineering Unstoppable — Cybersecurity Magazine
- Top 5 AI-Powered Social Engineering Attacks - The Hacker News — The Hacker News
- 5 AI Failures That Shocked Our Red Team This Year - ActiveFence — ActiveFence
