Editorial illustration for AT&T cuts AI orchestration costs 90% after handling 8 B tokens daily
AI Agent Routing Slashes Enterprise Token Costs 90%
AT&T cuts AI orchestration costs 90% after handling 8 B tokens daily
AT&T’s AI pipeline was processing 8 billion tokens every single day. That’s a staggering volume, enough to dwarf most enterprise deployments. The challenge wasn’t just scale; it was complexity.
The company needed to swap components in and out, test rigorously, and avoid the trap of over-engineering. So they rebuilt their orchestration from the ground up. The result?
A 90% cost cut. No gimmicks. Just disciplined evaluation, a core framework of LangChain, fine-tuned RAG, and a close partnership with Microsoft.
But here’s the crucial insight: not every tool needs to be agentic. Sometimes the smartest move is knowing what to leave out.
"We need to be able to pilot, plug in and plug out different components." They do "really rigorous" evaluations of available options as well as their own; for instance, their Ask Data with Relational Knowledge Graph has topped the Spider 2.0 text to SQL accuracy leaderboard, and other tools have scored highly on the BERT SQL benchmark. In the case of homegrown agentic tools, his team uses LangChain as a core framework, fine-tunes models with standard retrieval-augmented generation (RAG) and other in-house algorithms, and partners closely with Microsoft, using the tech giant's search functionality for their vector store. Ultimately, though, it's important not to just fuse agentic AI or other advanced tools into everything for the sake of it, Markus advised. "Sometimes I've seen a solution over engineered." Instead, builders should ask themselves whether a given tool actually needs to be agentic.
The takeaway is brutally simple: scale without discipline is just noise. AT&T didn’t slash costs by throwing more AI at the problem. They did it by knowing when *not* to use AI.
When to swap a component. When a plain old lookup beats a RAG pipeline. The 90% saving isn’t a magic trick; it’s the reward for refusing to overengineer.
Ask yourself: does this tool need to be agentic? If the answer is no, you just saved your budget. If yes, then build ruthlessly, test brutally, and stay ready to plug something else in.
That’s the real orchestration.
Related Reading
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google tests visual 'magazine-style' UI for Gemini 3 Pro users
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
From Archives Peace Corps recruits volunteers to sell AI education tools to developing nations
From Archives MIT's PhysiOpt Merges GenAI with Shape Optimization for Custom Accessories
Common Questions Answered
How did AT&T reduce its AI orchestration costs by 90%?
AT&T transitioned from a monolithic model pipeline to a LangChain-based multi-agent stack that allows for modular component management. The new approach enables engineers to pilot, plug in, and remove different AI components easily, dramatically reducing computational overhead and increasing flexibility in their AI infrastructure.
What volume of tokens was AT&T processing daily before their AI orchestration overhaul?
AT&T was handling approximately eight billion tokens each day, which exposed significant inefficiencies in their original AI orchestration layer. This massive token volume drove the company to seek a more scalable and cost-effective framework for managing their AI computational resources.
What key strategy did AT&T's chief data officer Andy Markus implement to improve AI orchestration?
Andy Markus implemented a LangChain-based multi-agent architecture where large "super agents" can delegate work to smaller components. This approach provides unprecedented flexibility, allowing the team to rapidly test, swap, and retire different AI services without disrupting the entire system.
