Brumby-14B-Base Qwen3 variant uses Power Retention, avoids full training cost
Why does a 14‑billion‑parameter model matter when the budget barely stretches to a single training run? While the Qwen‑3 architecture has been praised for raw capability, the Brumby‑14B‑Base variant takes a different tack. Instead of pouring resources into a full‑scale pre‑training cycle, its developers applied a method called Power Retention, essentially re‑using the weight matrix of an earlier generation and fine‑tuning it for the new task set.
The result is a model that reaches comparable performance without the expense of starting from zero. Buckman, one of the team’s leads, points out that this approach could reshape how quickly new models get into the hands of researchers. He notes that building on existing weights “is a critical accelerant for the adoption of a new model.” In other words, the cost savings aren’t just a nice‑to‑have—they’re the reason Brumby could even exist at the price tag it carries.
As Buckman puts it,
“Brumby could not be trained from scratch for that price.”
"Brumby could not be trained from scratch for that price." Still, Buckman emphasized the significance of that result: "The reason this is important is that the ability to build on the weights of the previous generation of model architectures is a critical accelerant for the adoption of a new modeling paradigm." He argues this demonstrates how attention-free systems can catch up to transformer performance "for orders-of-magnitude less" investment. In the loss curves released by Manifest AI, Brumby's training loss quickly converges to that of the Qwen3 baseline within 3,000 training steps, even as the architecture diverges significantly from its transformer origins.
Can a single technique cut costs? Brumby-14B-Base, a Qwen3 variant, applies Power Retention to sidestep the expense of full‑scale training. The model reuses weights from its predecessor, a strategy Buckman calls a critical accelerant for new model adoption.
Yet the article notes that Brumby couldn't be trained from scratch for that price, underscoring the financial barrier that Power Retention seeks to lower. By building on existing architecture, the approach leans on the attention mechanism that has defined every major LLM since 2017. However, the broader impact remains uncertain; it's unclear whether other developers will replicate the method or if performance trade‑offs will emerge.
The technique offers a pragmatic shortcut, but without independent benchmarks the claim of cost efficiency lacks external validation. In practice, the result is a model that appears to achieve comparable capability while avoiding the full training bill. Whether this will reshape development pipelines or stay a niche solution is still an open question.
Further Reading
- Brumby-14B-Base: The Strongest Attention-Free Base Model - Manifest AI
- Qwen3: A New Generation of Large Language Models - arXiv
- Ultimate Guide - The Best Qwen3 Models in 2025 - SiliconFlow
Common Questions Answered
What is the Power Retention method used in Brumby-14B-Base?
Power Retention is a technique that re‑uses the weight matrix from an earlier generation model and fine‑tunes it for a new task set. By avoiding a full‑scale pre‑training run, it dramatically reduces computational cost while preserving performance.
Why does Buckman consider Power Retention a critical accelerant for new model adoption?
Buckman argues that building on the weights of a previous architecture speeds up development and lowers financial barriers, enabling faster adoption of novel modeling paradigms. This accelerant effect is especially important for attention‑free systems trying to match transformer performance.
How does Brumby-14B-Base compare to training a Qwen‑3 model from scratch in terms of cost?
The article states that Brumby‑14B‑Base could not be trained from scratch for the same price, highlighting the prohibitive expense of full‑scale training. Power Retention allows the model to achieve comparable results at a fraction of the cost.
What does the article suggest about the performance of attention‑free systems versus transformers?
It suggests that attention‑free systems, when using techniques like Power Retention, can catch up to transformer performance while requiring orders‑of‑magnitude less investment. This challenges the notion that transformers are the only viable high‑performance architecture.