Editorial illustration for Supabase Launches Evals to Benchmark Claude, Codex, and OpenCode on Real Tasks
Supabase Evals Benchmark Claude, Codex, OpenCode
Supabase pushed its evaluation framework to GitHub this week under an Apache-2.0 license, giving developers a way to check whether AI coding agents actually know how to build with its platform. The repository, supabase/evals, runs locally through pnpm and pits Claude Code, Codex, and OpenCode against tasks pulled from real support tickets and GitHub issues: writing a database schema, tracking down a failed Edge Function, patching a broken RLS policy.
The scoring feeds two places. A public leaderboard lives at supabase.com/evals, and an internal regression suite checks for backsliding every day. Getting it running takes a Docker daemon, provider API keys, and ports 54321 through 54329 free, since the harness spins up a hosted-like stack alongside a local CLI project in containers rather than faking the environment.
That setup matters most in places where an agent's mistake isn't cosmetic. Fintech and healthcare backends built on Supabase can't tolerate a coding agent quietly writing a wrong access policy, and the framework is built to catch exactly that kind of failure before it ships.
Supabase has open sourced Supabase Evals, its benchmark and framework for testing how well AI agents build using Supabase. It runs coding agents including Claude Code, Codex, and OpenCode against real tasks, such as building a schema, debugging a failed Edge Function, or fixing a broken RLS policy, then scores the result.
Why this matters For anyone building on Supabase with an AI agent, this is the first time you get an actual scorecard instead of vendor promises. Claude Code, Codex, and OpenCode now get graded on the same nine products and nine topics, from RLS policies to declarative schemas, rather than judged on marketing demos. That's useful for founders picking a coding agent and for researchers who want a fixed rubric instead of anecdotes about which model "feels" better at SQL.
The Apache-2.0 license matters more than the leaderboard itself. Anyone can clone supabase/evals, add their own tasks, or run the suite against a new model the day it ships, which is exactly what an internal regression suite monitored daily is built for. Worth watching whether other platform vendors, Vercel, Cloudflare, whoever, follow with their own task-specific evals rather than leaning on generic benchmarks that say little about how an agent handles a broken Edge Function or a bad migration at 2 a.m.
Related Reading
Browse all Anthropic news →
Open Source D&B rebuilds 642 million‑business database after AI agents hit limits
Open Source OpenAI's GPT Transcribe Cuts Error Rate to 3.31%, Improving on GPT-4o
Open Source Study: 6.7% of Deepfake Requests on Hugging Face Targeted Children
Related Topic Anthropic says Claude writes >90% of its code; AI pause button urged
LLMs & Generative AI
Related Topic Claude Mythos highlights EU AI safety gaps, says researcher Caroli
Policy & Regulation
From Archives OpenAI to Publish Report on AI Solving Ten Unsolved Math Problems
From Archives Sources: More OpenAI Agents Reportedly Escaped Sandboxes
From Archives Anthropic says Claude AI hacked companies during safety test
LLMs & Generative AI
From Archives Anthropic says its AI models breached three companies in security tests
Common Questions Answered
What specific AI coding agents does Supabase Evals benchmark?
Supabase Evals benchmarks three AI coding agents: Claude Code, Codex, and OpenCode. These agents are tested against real-world tasks to evaluate their performance on the Supabase platform.
What types of real tasks are used to evaluate the AI agents in Supabase Evals?
The evaluation framework tests AI agents on real tasks pulled from support tickets and GitHub issues, including writing database schemas, tracking down failed Edge Functions, and patching broken RLS policies. These tasks represent actual challenges developers face when building with Supabase.
How does Supabase Evals differ from traditional vendor benchmarks?
Unlike vendor marketing demos and anecdotal comparisons, Supabase Evals provides an objective scorecard grading all three AI agents on the same nine products and nine topics using a fixed rubric. This standardized approach gives founders and researchers reliable data instead of subjective claims about which model performs better.
What license does Supabase use for its open-source Evals framework?
Supabase released its Evals framework under an Apache-2.0 license and pushed it to GitHub as the supabase/evals repository. This open-source approach allows developers to run the evaluation framework locally through pnpm and contribute to its development.
Why is Supabase Evals important for developers choosing a coding agent?
Supabase Evals is the first benchmark that provides an actual scorecard for AI agents building on Supabase, rather than relying on vendor promises or marketing claims. This gives founders and developers concrete performance data to make informed decisions about which coding agent best suits their needs.
Further Reading
- Supabase opens source Evals to benchmark Claude Code, Codex, and OpenCode on real Supabase tasks - Supabase on X
- Today we're open-sourcing Supabase Evals - Supabase
- Supabase Agent Skills: early benchmark results for Claude Code and Codex on Supabase workflows - Supabase Blog
- Supabase is now an official Claude connector - Supabase Blog
- AI Agents Know About Supabase. They Don't Always Use It ... - Supabase Blog
