Editorial illustration for Local LLM Performance Varies Widely on Calendar and Email Tasks
Local LLMs Struggle With Calendar and Email Tasks
Local LLM Performance Varies Widely on Calendar and Email Tasks
A month ago, a 30B parameter model choked on the job of running a personal AI assistant, scoring 22.8 out of 100 against Claude's 89.4 and stuffing malformed tool-call syntax into a quarter of its answers. That test ran on a single RTX 3090, capped at a 16K context window, replaying real tasks pulled from an actual agent called Jarvis rather than synthetic benchmarks. The failure wasn't subtle.
It wasn't a matter of the local model being a little worse at phrasing or a bit slower on inference. Something in the pipeline was broken.
So the hardware changed. Three RTX 3090s instead of one, a 256K context window instead of 16K, and a jump to a 122B model. The same 27 tasks got replayed against the same frozen Claude baseline, scored the same way.
What came back looked nothing like the first run, and the cost math looked even less like it. That comparison, run twice on two different rigs with two different model sizes, is the actual subject here: not whether local LLMs can theoretically match a hosted model, but what happens when you actually swap the brain of a working assistant and measure what breaks.
On a single RTX 3090 capped at a 16K context, a 30B model scored 22.8/100 to Claude’s 89.4, and leaked malformed tool-call syntax into a quarter of its answers — broken, not merely worse. On three 3090s with a 256K context, a 122B model scored 80.0, eliminated the malformed calls entirely (0 of 27), and cost $0.000969 per task against Claude’s $0.763 — roughly 787× cheaper.
Why this matters
For anyone building or buying local AI agents, this test is a reminder that headline benchmark scores can hide as much as they reveal. Bump the context window from 16K to 256K and swap a 30B model for a 122B one, and the score jumps from 22.8 to 80.0, with malformed tool calls dropping from a quarter of responses to zero. That's a hardware and architecture story, not just a "bigger model is better" one.
But the more useful finding is the bucketing bug: 16 of 28 tasks touched calendar, email, notes, or messaging, where a safety wrapper mocking Outlook COM calls flattened scores to arbitrary 15s and 45s regardless of actual model quality. If you're evaluating local LLMs against a cloud baseline like Claude, the eval harness itself can manufacture a gap that has nothing to do with reasoning ability. At $0.000969 per task, the economics of running a 122B model on three 3090s look compelling, but only if your benchmark is actually measuring the model and not an artifact of how your test environment mocks real-world tools.
Related Reading
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google tests visual 'magazine-style' UI for Gemini 3 Pro users
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
From Archives Saber CEO Denies Using ChatGPT to Replace Writers on 'Rideshare Stimulator
From Archives Gemini app hits one billion users as Google expands AI across services
Common Questions Answered
Why did the 30B parameter local model perform so poorly on the Jarvis personal AI assistant tasks?
The 30B model was severely constrained by running on a single RTX 3090 with only a 16K context window, which limited its ability to handle the complexity of real calendar and email tasks. Beyond the low score of 22.8 out of 100, the model also produced malformed tool-call syntax in approximately 25% of its answers, indicating fundamental failures rather than minor performance degradation compared to Claude's 89.4 score.
How much did scaling to a 122B model and increasing context window improve performance on these tasks?
Upgrading to a 122B model running on three RTX 3090s with a 256K context window dramatically improved the score from 22.8 to 80.0, while completely eliminating malformed tool calls (reducing them from 25% of responses to zero). This demonstrates that hardware configuration and context window size are critical factors in local LLM performance, not just model parameter count.
What is the cost difference between running the improved local LLM setup versus using Claude for these tasks?
The 122B local model costs approximately $0.000969 per task compared to Claude's $0.763 per task, making the local solution roughly 787 times cheaper. This significant cost advantage comes while achieving substantially better performance on the real-world Jarvis agent tasks, making local LLMs economically compelling for personal AI assistant applications.
Why do headline benchmark scores potentially mislead when evaluating local LLMs for AI agents?
Headline benchmarks often fail to capture how dramatically performance can change based on hardware constraints and architectural decisions rather than just model size. The test results show that context window and GPU configuration can swing scores from 22.8 to 80.0 and eliminate critical failure modes like malformed tool calls, revealing that real-world performance depends on the complete system setup rather than model parameters alone.
Further Reading
- Local LLM Benchmark: Administrative Tasks - AAI Labs
- I Tested 13 Local LLMs on Tool Calling | 2026 Eval Results - JD Hodges
- Local AI Email and Calendar Triage 2026 - PromptQuorum
- Building an email to calendar LLM - Brandon's Digital Garden
- How To Create A Private Ai Assistant That Manages Your Calendar Emails And Grocery Lists Using Only Local LLM - Alibaba.com
