Editorial illustration for Meta's JiTTests score failing tests via LLM-as-Judge before risky code lands
Meta's JiTTests Catches Privacy Bugs Before Deploy
Meta's JiTTests score failing tests via LLM-as-Judge before risky code lands
Meta's engineers merged 74 privacy bug fixes before those bugs ever shipped, catching them at the diff stage during the first half of 2026. The tool behind that number is called JiTTests, short for Catching Just-in-Time Tests, and Meta has now moved it out of research and into production infrastructure that runs across a codebase measured in hundreds of millions of lines.
The system works differently from most AI code review tools on the market. Instead of reading a diff and leaving a comment, the way a reviewer might, JiTTests writes an actual executable test aimed at the specific change, a test built to fail if that change breaks something. A failing test is hard evidence. It's not a suggestion or a guess about risk, it's proof that behavior changed.
Meta described the approach in a paper for FSE 2026, "Just-in-Time Catching Test Generation at Meta," which analyzed 22,126 generated tests and used judge models including Llama3.3-70B, Gemini 3 Pro, and Claude Sonnet 4 to score results. The company first laid out the underlying concept, splitting tests into "hardening" and "catching" categories, in a 2025 paper called "Harden and Catch."
Meta JiTTests is an LLM-based system that writes executable tests designed to fail on a risky code change. A failing test is concrete evidence that a diff changed behavior. Meta team reports that JiTTests caught 74 privacy SEVs at the diff stage in the first half of 2026.
Why this matters
JiTTests flips the usual pitch for AI coding tools. Instead of asking an LLM to write code that works, Meta asks it to write tests that break, then layers a judge ensemble and a rule-based filter, RubFake, on top to decide if the failure is real. That's a more honest use of these models: generation paired with adversarial checking, not generation alone. The 74 privacy SEVs caught at diff stage in the first half of 2026 is a concrete number worth watching, because it's the kind of claim that invites scrutiny: how many false positives did RubFake kill, and how does that ratio hold up outside Meta's codebase and review culture.
For developers and founders building with LLMs, the lesson isn't "use AI to catch bugs." It's that catching bugs with AI requires a second AI checking the first one's work, plus deterministic rules underneath both. Researchers should read the FSE 2026 paper closely for how the judge ensemble handles disagreement among models, since that's usually where these systems quietly fail in ways a single accuracy number won't show.
Common Questions Answered
How does Meta's JiTTests differ from traditional AI code review tools?
Instead of reading a diff and leaving comments like most AI code review tools, JiTTests writes executable tests specifically designed to fail on risky code changes. A failing test provides concrete evidence that a diff has altered behavior, making it a more adversarial approach to catching bugs before they ship.
What is the main purpose of the LLM-as-Judge system in JiTTests?
The LLM-as-Judge system evaluates whether test failures are genuine issues or false positives by layering a judge ensemble with a rule-based filter called RubFake. This combination helps determine if a failing test represents a real behavioral change that needs attention.
How many privacy bugs did JiTTests catch at the diff stage during the first half of 2026?
Meta's engineers merged 74 privacy bug fixes before those bugs ever shipped to production, all caught by JiTTests during the diff stage in the first half of 2026. This concrete number demonstrates the tool's effectiveness in preventing privacy security vulnerabilities from reaching users.
What scale of codebase does JiTTests now operate on in Meta's production infrastructure?
JiTTests has been moved from research into production infrastructure that runs across a codebase measured in hundreds of millions of lines. This demonstrates Meta's confidence in the tool's ability to scale across their massive engineering organization.
Why does Meta consider JiTTests a more honest use of LLMs compared to other AI coding tools?
JiTTests pairs LLM generation with adversarial checking rather than relying on generation alone, which is a more honest approach to using these models. By asking the LLM to write tests that break instead of code that works, Meta combines generation with verification to ensure reliability and catch real behavioral changes.
Further Reading
- Catching Bugs Before They Land: JiTTesting’s Human-on-the-Loop Approach at Full Scale - Meta Engineering
- Meta says its AI-written failing tests now catch serious bugs before code ships - Sulat
- Meta's JiTTests Catches 74 Privacy Bugs Before They Hit Production - AlphaSignal
- Meta AI Updates: October 31, 2026 - My2Cents
- Catching Bugs Before They Land: JiTTesting’s Human-on-the-Loop Approach at Full Scale - DevFeed