Editorial illustration for Baidu ERNIE Evolve Self-Reports 33.7% Throughput Gain Over NanoChat, NanoGPT
Baidu ERNIE Evolve Achieves 33.7% Throughput Boost
Baidu says it has built an automated research system that can rewrite its own training and inference code and then prove the rewrite actually works. The system, called ERNIE Evolve, takes a repository, an objective, and a set of evaluation criteria, then picks what to investigate, edits the code, and runs the experiment itself. Nothing sticks unless it clears a hidden evaluator and a separate reviewer, a gate meant to filter out changes that look good on paper but fall apart under scrutiny.
Baidu ran the system against three public benchmarks, using a single NVIDIA B200 for NanoChat and SOL-ExecBench and an 8xH100 setup for Speedrun Track 1. It then turned the system loose on something with real stakes: inference optimization for ERNIE-5.5-Flash, a model still in development, tested on Kunlunxin P800 hardware and GPUs Baidu hasn't named. Baidu frames this as a step toward recursive self-improvement, where AI systems take on the job of improving the software that builds and runs AI. The company announced the work on X, alongside benchmark numbers it reported itself.
Baidu has just released ERNIE Evolve, an automated research system that improves AI training and inference code. Given a repository, an objective and evaluation criteria, it picks research directions, edits code and runs experiments. Only changes that pass a hidden evaluator and a separate reviewer are kept.
Why this matters
Baidu is asking us to take its word for a 33.7% throughput gain and a 14.9% decode speed bump, measured by Baidu's own evaluator on a model that isn't public, ERNIE-5.5-Flash. That's the whole catch. An automated system that edits its own training and inference code, then grades itself against a reviewer it also built, is interesting engineering, but it's not independent verification.
We've seen plenty of self-reported benchmarks evaporate once outside labs get access. The public-benchmark numbers against NanoChat and NanoGPT are reportedly modest, which tells us more about where the real gains came from: a closed, in-house model Baidu controls end to end. For developers and researchers, the lesson isn't "AI agents can now optimize AI systems," it's "wait for the repo, the weights, or at least a third-party replication." Founders pitching automated R&D pipelines should expect the same question investors will ask Baidu: who checked this besides you?
Until ERNIE Evolve or its benchmarks are reproducible outside Baidu's walls, treat the numbers as a press release with code comments attached.
Related Reading
Research & Benchmarks Microsoft AI Chief: "If It Isn't Safe We Shouldn't Build It
Research & Benchmarks Google's FACTS benchmark shows 70% factuality ceiling across four tests
Research & Benchmarks Nvidia's DLSS 4.5 beta adds 6x Multi Frame Generation for RTX 50 GPUs
Related Topic Baidu launches ERNIE 5.0 preview 1022, beating GPT-5 on charts and document tasks
Business & Startups
Related Topic Tencent, Baidu launch AI companion apps; women dominate market, says Sun Zhaozhi
LLMs & Generative AI
From Archives NVIDIA Isaac Lab Lets Robots Rehearse Delicate Work in Simulation at CoRL
From Archives India Office Workforce to Reach 48 Mn by 2030 as AI, GCCs Drive Demand
From Archives Baidu Releases Open-Source ERNIE 5.5 Flash Base With 280 Billion Parameters
Open Source
From Archives Baidu Open-Sources Unlimited OCR 2 for Hundred-Page Document Parsing
Open Source
Common Questions Answered
How does ERNIE Evolve automatically improve AI training and inference code?
ERNIE Evolve takes a code repository, an objective, and evaluation criteria, then autonomously selects research directions, edits the code, and runs experiments to test improvements. The system implements a dual-gate verification process where changes must pass both a hidden evaluator and a separate reviewer before being accepted, ensuring that modifications work in practice rather than just on paper.
What are the reported performance improvements of ERNIE Evolve over NanoChat and NanoGPT?
According to Baidu's claims, ERNIE Evolve achieved a 33.7% throughput gain and a 14.9% decode speed improvement over NanoChat and NanoGPT. However, these benchmarks were measured using Baidu's own evaluator on the non-public ERNIE-5.5-Flash model, which raises questions about independent verification.
Why is independent verification important for ERNIE Evolve's reported benchmarks?
Baidu's self-reported benchmarks lack independent verification since the system grades itself using an evaluator that Baidu also built, creating potential bias in the results. History shows that self-reported AI benchmarks often fail to hold up when outside laboratories gain access to the models and conduct their own testing.
What is the dual-reviewer gate system in ERNIE Evolve designed to prevent?
The dual-reviewer gate system, consisting of a hidden evaluator and a separate reviewer, is designed to filter out code changes that appear promising on paper but fail when subjected to actual scrutiny and real-world testing. This mechanism aims to ensure that only genuinely effective improvements are retained in the system.
Further Reading
- ERNIE Evolve: Toward Automated Research for Real-World AI R&D - Baidu ERNIE Research
- Baidu's ERNIE Evolve Automated Research System Beats NanoChat and NanoGPT Records - MarkTechPost
- Baidu's ERNIE Evolve Rewrites Its Own AI Code and Boosts Speed 6x - AlphaSignal
- Baidu Details ERNIE Evolve, an Automated Research System, and Reports Gains on Public AI R&D Benchmarks - Deyron Labs
