Editorial illustration for Waymo Shifts Focus to AI Evaluations, Testing Models Post-Launch
Waymo Tests AI Models After Launch, Shifts Strategy
Waymo Shifts Focus to AI Evaluations, Testing Models Post-Launch
Waymo has logged more than 220 million fully autonomous miles on public roads, and the company says its cars produce 17 times fewer serious crash injuries than human drivers over the same distance. Getting there required rethinking how the company builds and ships its AI models in the first place, according to Manasi Joshi, Waymo's director of engineering for systems intelligence and machine learning.
Speaking at VB Transform 2026, Joshi laid out the internal system Waymo uses to decide when an AI project is actually ready for the road. Unlike a chatbot that generates the wrong answer or a back-office tool that misfires quietly, a self-driving car's mistakes happen in traffic, in real time, with real consequences. Waymo's answer has been to treat evaluation not as a final gate before launch but as a discipline that runs through every stage of development, from early model training to post-deployment monitoring. Joshi calls this "eval-forced" or "eval-centric" development, and she said it now shapes how engineering teams judge whether a project is mature enough to move forward, regardless of how well the model performs in isolated tests.
In practice, Waymo assesses a project’s readiness partly by examining the maturity of the tests surrounding it. That approach has clear implications for enterprises building customer service agents, coding assistants, financial systems or other AI applications: If a company cannot reliably measure a system’s performance, it may not be ready to place that system into production.
Why this matters
Waymo's approach is a useful corrective for anyone building AI products right now. It's tempting to treat evaluation as a gate you pass through once, right before shipping, then move on to the next feature. Joshi's team doesn't get that luxury: a self-driving car that fails silently on a rainy Tuesday isn't a bug ticket, it's a liability.
So evals run before training, after training, and in simulation loops that never really stop. For developers and founders building with large language models, the lesson isn't "add more tests." It's that evaluation needs to be treated as infrastructure, not a checkbox, especially once a model is making decisions with real consequences instead of just generating text. Most teams still ship, watch metrics dip, and patch reactively.
Waymo's model suggests building the feedback loop into the product from day one, because retrofitting rigor after a failure is a much worse position than starting with it. The stakes in chatbots aren't the same as in traffic, but the discipline scales down fine.
Related Reading
Business & Startups Nordic pilot adds Gemini for Education, NotebookLM to boost AI literacy
Business & Startups Kling launches Video O1, all-in-one model with MVL bridge using transformer
Business & Startups DeepSeek Seeks More Capital Weeks After USD 7B Funding Round
From Archives Raindrop AI CTO: Agents Need Audit Logs to Fix Critical Issues
From Archives Target SVP: AI Competitive Advantage Lies Beyond the Models
Common Questions Answered
How does Waymo's autonomous vehicle safety record compare to human drivers?
Waymo's cars produce 17 times fewer serious crash injuries than human drivers over the same distance traveled. The company has logged more than 220 million fully autonomous miles on public roads to achieve and validate this safety performance.
What is Waymo's approach to assessing AI project readiness before deployment?
Waymo assesses a project's readiness by examining the maturity of the tests surrounding it, according to director of engineering Manasi Joshi. This means that if a company cannot reliably measure a system's performance through comprehensive testing, the system may not be ready for production deployment.
When does Waymo conduct AI model evaluations throughout the development lifecycle?
Waymo runs evaluations before training, after training, and in continuous simulation loops that never really stop. This ongoing evaluation approach is necessary because autonomous vehicles operating in real-world conditions like rain cannot afford silent failures, making continuous testing a critical liability management practice.
What are the implications of Waymo's evaluation methodology for enterprises building AI applications?
Waymo's approach serves as a corrective for companies building AI products like customer service agents, coding assistants, and financial systems. The key implication is that evaluation should not be treated as a one-time gate before shipping, but rather as a continuous process integrated throughout development to ensure reliable system performance.
Further Reading
- Waymo says it built a better benchmark for comparing robotaxis to humans - TechCrunch
- Demonstrably Safe AI For Autonomous Driving - Waymo Blog
- Challenges for the evaluation of automated driving systems using on-road and simulation data - Waymo Research
- Collision Avoidance Testing of the Waymo Automated Driving System - arXiv
- Safe to Deploy: How We Know The Waymo Driver Is Ready to Scale - Waymo Blog
