250 Toxic Docs Expose Critical LLM Hacking Vulnerability

Anthropic, working with the UK’s AI Security Institute and the Alan Turing Institute, has discovered that as few as 250 poisoned documents are enough to insert a backdoor into large language models - regardless of model size. The team trained models ranging from 600 million to 13 billion parameters and found that the number of poisoned documents required stayed constant, even though larger models were trained on far more clean data. The findings challenge the long-held assumption that attackers need to control a specific percentage of training data to compromise a model.

In this case, the poisoned samples made up only 0.00016 percent of the entire dataset - yet they were enough to sabotage the model’s behavior. Currently low risk The researchers tested a "denial-of-service" style backdoor that causes the model to output gibberish when it encounters a specific trigger word. In their experiments, that trigger was "SUDO." Each poisoned document contained normal text, followed by the trigger word and then a sequence of random, meaningless words.