Editorial illustration for Anthropic Adds Six Grader Types to Claude's Plugin Evaluation System
Claude Gets 6 New Plugin Evaluation Graders
Anthropic Adds Six Grader Types to Claude's Plugin Evaluation System
Anthropic shipped a new command for Claude Code this week that lets plugin developers actually test whether their work does anything. The tool, invoked as claude plugin eval, requires Claude Code v2.1.269 or later and runs against any directory carrying a plugin.json manifest, a .claude-plugin/plugin.json file, or a skills-directory structure. It takes a plugin, feeds it realistic prompts, grades the output, then runs the same prompts with the plugin stripped out to see if the difference was worth building in the first place.
That last part matters more than it sounds. Plugin authors have had no clean way to answer three basic questions: whether a skill actually triggers when it should, whether it keeps working after an edit or a model swap, and whether it beats Claude with nothing loaded at all. Anthropic's framework answers all three in one run, and it's a real deployable feature, not a preview. Every eval and every judge call is a live model request billed to the developer's plan or API account, so testing has a real cost attached.
The structure underneath the command, how a test case is built and how graders are configured, is where the actual mechanics live.
Anthropic has published a new plugin evals workflow for Claude Code. The claude plugin eval command runs a plugin against realistic prompts, grades what Claude produced, and compares the result with a run where the plugin is not loaded. It answers 3 questions plugin developers could not previously measure: does the skill trigger, does it survive an edit or a new model, and does it beat a bare model.
Why this matters
Plugin developers have been shipping skills on vibes: does it feel like it works, does the demo look good. The Δ metric changes that calculus. If your plugin scores the same with or without it loaded, you've built decoration, not a tool, and now there's a command that says so in plain numbers rather than gut feel.
The CI gate matters more than the six grader types themselves. Skills silently rot when Claude Code updates or someone tweaks a prompt three months later, and nobody notices until a user complains. Wiring `claude plugin eval` into CI means that regression shows up in a pull request, not in production. For teams building on Claude Code, that's the difference between a plugin ecosystem you can trust and one you have to babysit.
The cost detail is worth flagging, too. Four graders are free; `llm` and `baseline` bill a judge model. That's a reasonable tradeoff, but it means comprehensive eval coverage isn't free, and teams should budget for it rather than discovering the bill after scaling up test suites. Worth watching whether Anthropic extends this pattern to other agent frameworks, or keeps it Claude Code-specific.
Related Reading
Browse all Anthropic news →
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google tests visual 'magazine-style' UI for Gemini 3 Pro users
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
Related Topic Claude gains shared context in Excel, PowerPoint; Microsoft adds Copilot Cowork
AI Tools & Apps
Related Topic Trump cracks down on Anthropic after Amazon tip; staff largely foreign
Open Source
From Archives ChatGPT Now Shares Image Creation Prompts for Reuse
From Archives CapQuiz: New Method Tests Video Caption Quality with Information Fidelity
From Archives Anthropic's Cybersecurity AI Model Called Most Likely to Cause Severe Harm
Research & Benchmarks
From Archives Chinese AI labs used 5,380 fake accounts to relay 300,000 user queries to Anthropic
Common Questions Answered
What is the claude plugin eval command and how do plugin developers use it?
The claude plugin eval command is a new tool for Claude Code v2.1.269 or later that allows plugin developers to test their work against realistic prompts. It works with any directory containing a plugin.json manifest, .claude-plugin/plugin.json file, or skills-directory structure, and automatically grades the output to measure plugin effectiveness.
What three key questions does the plugin evaluation system answer for developers?
The evaluation system answers whether the skill triggers properly, whether it survives edits or new model updates, and whether it actually outperforms Claude without the plugin loaded. These questions help developers determine if their plugin provides real value or is merely decorative.
How does the Δ metric change plugin development practices?
The Δ metric compares plugin performance against a baseline run without the plugin loaded, replacing subjective assessments with objective numerical measurements. This shift from gut-feel evaluation to data-driven metrics prevents developers from shipping plugins that don't actually improve Claude's capabilities.
Why is the CI gate important for long-term plugin maintenance?
The CI gate prevents skills from silently degrading when Claude Code updates or prompts are modified over time. By continuously running evaluations, developers can catch performance regressions and ensure their plugins remain functional across model versions and code changes.
What are the six grader types included in Anthropic's plugin evaluation system?
While the article mentions that Anthropic added six grader types to the evaluation system, the specific details of each grader type are not provided in the excerpt. The graders work together to comprehensively assess plugin functionality and performance against realistic use cases.
Further Reading
- Claude Code Plugin Eval: Catch a Skill That Never Activates - Matthew S. Wong
- agents/docs/plugin-eval.md at main - GitHub
- How to build a Claude plugin marketplace with evals - AI Builders Blog
- build-eval - ClaudePluginHub
- Eval — Claude skill / plugin install guide - ClaudeCowork
