Editorial illustration for Rubrik's AI judges every agent move, but accuracy remains unmeasured
Rubrik's AI Agents Run Unsupervised, Accuracy Unknown
Rubrik's AI judges every agent move, but accuracy remains unmeasured
Rubrik has a policy problem, and it's not the one most CISOs worry about. The data security company, which trades publicly and stores what its own AI chief calls the most important data in the world, has started letting its AI agents run in what it calls YOLO mode: no permission prompt, no human clicking approve before an action executes. Dev Rishi, GM of AI at Rubrik, says the push came from the company's own founder and CTO, who wants agents acting without a person in the loop.
To make that workable, Rubrik built a second AI whose only job is to judge every move the first agent makes, checking it against internal policy in real time. Rishi laid out the setup at a VB Transform 2026 fireside chat in Menlo Park, days after a CISO roundtable hosted by Anthropic's chief information security officer surfaced a familiar gap: every security leader in the room had written AI governance policies, but almost none had a real way to enforce them. Rubrik's judge-AI is its answer to that gap.
Whether the judge itself gets things right is a separate question, and one Rishi admits nobody has actually tested.
YOLO mode strips the permission prompt out of agent workflows and lets the agent act on its own. In Rubrik's version, a second AI judges every action in real time against policy in place of a human clicking approve.
Why this matters
Rubrik built a judge for its AI agents before it built a way to check the judge's homework. That's not a small gap. It's the same gap VentureBeat's own survey found across the industry: 66% of enterprises are pushing toward production with policies nobody enforces, let alone audits.
Rishi's roundtable story is telling precisely because the room full of security leaders laughed instead of answering. Written governance policies are cheap. Enforcement is where the actual work lives, and right now most companies are grading their AI's homework with an AI that nobody has graded.
For developers and founders building agentic systems, the lesson isn't "add an AI judge." It's that a judge without a validated accuracy rate is theater dressed up as oversight, and boards will eventually ask for the number. Researchers should treat judge-model calibration as its own open problem, not an afterthought bolted onto agent architecture. Rishi calls unenforced governance a drag on ROI. We'd go further: shipping unverified judges at scale is a liability waiting for its first bad headline.
Common Questions Answered
What is YOLO mode in Rubrik's AI agent workflows?
YOLO mode removes the permission prompt requirement from agent workflows, allowing AI agents to execute actions without waiting for human approval. Instead of a human clicking to approve each action, a second AI judges every move in real time against the company's established policies to ensure compliance and security.
Why did Rubrik's founder push for AI agents to operate without human approval loops?
Rubrik's founder and CTO wanted agents to act without a person in the loop to improve operational efficiency and speed. This shift toward autonomous AI agent operation represents a move away from traditional approval workflows that require manual human intervention before each action executes.
What is the main gap Rubrik faces with its AI judge system?
Rubrik built an AI judge to oversee agent actions against policy, but has not yet established a way to verify whether the judge itself is making accurate decisions. This accountability gap means there is no measurement system in place to audit the AI judge's own performance and correctness.
What did VentureBeat's survey reveal about enterprise AI policy enforcement?
VentureBeat's survey found that 66% of enterprises are moving toward production environments with AI policies that nobody actively enforces or audits. This industry-wide pattern mirrors Rubrik's situation, highlighting that written governance policies are relatively easy to create, but actual enforcement and monitoring require significantly more work and resources.
Further Reading
- AdaRubric: Task-Adaptive Rubrics for LLM Agent Evaluation - arXiv
- Agent-as-a-Judge: Multi-Turn Rubric-Guided ... - OpenReview
- Agent Observability: How to Monitor AI Agents - Rubrik
- AI agents make mistakes; Rubrik has figured out a way to reverse them - Computerworld
- Inside Rubrik AI: The Lessons We Learned Architecting an AI Agent from Scratch - Rubrik