ESQ-Bench Reveals NL2SQL Model Weaknesses

Spider and BIRD, the two benchmarks most NL2SQL papers lean on for bragging rights, let top models clear 89 percent execution accuracy. That number looks reassuring until you ask what the benchmarks actually contain: academic schemas, open-source SQL dialects, and query patterns far simpler than anything running inside a bank or an insurance company. A new benchmark called ESQ-Bench takes aim at that gap directly, building its test set from enterprise Oracle schemas rather than textbook databases, and it structures queries into difficulty tiers meant to mimic how real analysts escalate from simple lookups to convoluted joins.

The design choice that matters most is what ESQ-Bench measures alongside execution success. A query can run without errors and still return the wrong answer, a failure mode invisible to benchmarks that only check whether SQL executes. ESQ-Bench tracks exact-match accuracy separately from execution accuracy, and it flags cases where a model's output passes execution but diverges semantically from the correct result. That distinction, largely absent from Spider and BIRD, is where the benchmark's findings start to diverge sharply from the polished numbers vendors publish.