Editorial illustration for Together AI Releases Together Compiler to Speed Up Agent Inference 2.6x
Together AI Compiler Speeds Up Agent Inference 2.6x
Together AI has started running a new optimization system called Together Compiler, or TKCC, across several of its serverless inference endpoints. The pitch is simple: a model engine tuned for one traffic pattern goes stale the moment that pattern shifts, so the tuning has to keep running instead of stopping after a single pass. Together tested the system on Kimi K3, GLM 5.3, MiniMax M3.1 Flash, DeepSeek v4.1 Flash and Llama 1B, spreading the work across NVIDIA B200, B300 and GB300 chips, H100s, AMD's MI355X, and AWS Trainium2, using no more than two nodes per model.
The results varied by hardware and model. Together reports throughput gains of up to 2.6x over the best published InferenceX configurations, with each optimization run finishing in under 24 hours. The standout case hit 1,111 P90 tokens per second per user on MiniMax M3.1 Flash running on B200 at batch size 1, a 1.8x jump from the 633 TPS baseline.
Not every setup moved that much. GLM 5.3 on the MI355X only edged from 463 to 480 TPS, a roughly 4% gain.
Together AI has introduced Together Compiler (TKCC), an agent-driven system that keeps re-optimizing inference as production traffic changes. Together Compiler already runs on several of Together’s serverless endpoints. The core claim for developers: an engine tuned once goes stale, so tuning should never stop.
Why this matters
The headline number, 2.6x faster inference, is almost beside the point next to the 252x throughput claim for agent experimentation. If that holds up outside Together AI's own benchmarks, it changes the economics of running continuous optimization loops rather than tuning an engine once and walking away. For founders burning GPU budget on B200s, H100s or MI355X clusters, a compiler that keeps re-ranking tensor, expert, data and context parallelism as traffic shifts is a real cost lever, not a demo trick.
Researchers should note the methodology detail: two-tier testing covering 1,354 candidates versus 168 for a single slow tier is a meaningful sample-size jump, and testing across Kimi K3, GLM 5.3, MiniMax M3.1 Flash, DeepSeek v4.1 Flash and even a tiny Llama 1B suggests they tried to stress different model shapes. Still, every figure here comes from Together AI itself, no third-party reproduction yet. Worth watching whether independent teams replicate the 2.6x and 252x numbers on their own traffic patterns before treating "tuning never stops" as settled practice rather than a vendor's pitch.
Related Reading
Open Source D&B rebuilds 642 million‑business database after AI agents hit limits
Open Source Box CEO Aaron Levie Says AI Will Drive "Biggest Cybersecurity Upgrade
Open Source Study: 6.7% of Deepfake Requests on Hugging Face Targeted Children
From Archives Google AI Research debuts Agile Deliberation for gray-area image classifiers
From Archives Ricursive AI releases Ripple-65B preview scoring 80.5 on Terminal-Bench 2.1
Common Questions Answered
What is Together Compiler (TKCC) and how does it improve inference speed?
Together Compiler is an agent-driven optimization system developed by Together AI that continuously re-optimizes inference as production traffic patterns change. Rather than tuning a model engine once and allowing it to become stale, TKCC keeps the tuning process running, delivering a claimed 2.6x faster inference speed compared to traditional single-pass optimization approaches.
Which AI models and hardware chips did Together AI test the Together Compiler on?
Together AI tested Together Compiler across multiple models including Kimi K3, GLM 5.3, MiniMax M3.1 Flash, DeepSeek v4.1 Flash, and Llama 1B. The system was evaluated on various NVIDIA hardware including B200, B300, GB300 chips, and H100s, demonstrating broad compatibility across different model and infrastructure combinations.
What is the significance of the 252x throughput improvement claim for agent experimentation?
The 252x throughput improvement for agent experimentation suggests that continuous optimization loops become economically viable for developers, fundamentally changing how inference optimization is approached. This dramatic improvement means that the cost-benefit analysis for running continuous re-optimization on GPU clusters shifts dramatically, making it practical for founders to maintain dynamic tuning rather than static configurations.
How does Together Compiler adapt to changing production traffic patterns?
Together Compiler continuously re-ranks tensor, expert, data, and context parallelism configurations as traffic shifts in production environments. This agent-driven approach ensures that the optimization engine remains tuned to current traffic conditions rather than becoming stale with a single initial optimization pass, allowing the system to maintain peak performance as usage patterns evolve.
Where is Together Compiler currently available for developers to use?
Together Compiler (TKCC) is already running on several of Together AI's serverless inference endpoints, making it accessible to developers through Together's platform. This means developers can immediately benefit from the continuous optimization capabilities without needing to implement the system themselves.
Further Reading
- Together AI Releases Together Compiler: Agents Make Inference Up to 2.6x Faster - MarkTechPost
- Together AI Launches Together Compiler for Continual Optimization of Open Source AI Models - Business Wire
- Together Compiler: Continual Compilation for AI Inference - Together AI
- Polymorphic Inference: Self-Tuning Inference for Open Models - Together AI
