Editorial illustration for Stanford Open-Sources Memorilla, a Latent Memory Module for Frozen LLMs
Stanford Releases Memorilla: Memory Module for Frozen LLMs
Stanford's SNAP lab, led by Jure Leskovec, just put a memory system for large language models out in the open under Apache 2.0. It's called Memorilla, and the pitch is blunt: feed it an entire document collection, and it hands a frozen LLM exactly 16 memory tokens to work with, no matter how big that collection gets. The numbers back up the claim. On FactKG, a frozen Qwen3-8B hit 84.33% accuracy using Memorilla's 16 tokens, beating the 80.43% score it got with the full document dump stuffed into context, and doing it with 108 times fewer tokens.
The module itself is a 194-million-parameter trainable layer that sits between a frozen Qwen3-Embedding-4B encoder and a frozen Qwen3-8B decoder. Neither of those large models gets touched during training. Co-authors from Toronto, Penn and MIT worked alongside the Stanford team on the project, which ran its training recipes on single H100 80GB GPUs under Linux and Python 3.11. Code, checkpoints and training recipes are all public now, which opens the door to scrutiny of how a fixed-size memory budget holds up against retrieval methods and full-context prompting across different benchmarks.
On FactKG, a frozen Qwen3-8B scored 84.33% with Memorilla, against 80.43% with every document in context. It did that with 108x fewer document tokens. Code, training recipes and checkpoints are public under Apache 2.0.
Why this matters
The numbers here are a mixed bag, and that's the useful part. A 108x reduction in document tokens with a four-point accuracy gain on FactKG is a real win for anyone running long-context RAG pipelines at scale, where token costs compound fast. But Memorilla losing to LLMLingua-2 on PubMedQA by ten points, on abstracts of just 324 tokens, tells us compression only pays off when there's actually something to compress.
And MemoRAG beating it on PersonalizationV4 and NarrativeQA while burning 690x and 50x more tokens is a reminder that "fewer tokens" and "better answer" are two different axes, not one. For founders building on frozen LLMs, Memorilla's Apache 2.0 release is worth testing against your own corpus size before assuming it generalizes. For researchers, the FactKG result is the headline, but the PubMedQA gap is the one to interrogate.
The real question going forward is where the breakeven point sits between document length and compression benefit, and Stanford's open checkpoints finally give people a way to test that themselves instead of taking the benchmark table on faith.
Related Reading
Research & Benchmarks Microsoft AI Chief: "If It Isn't Safe We Shouldn't Build It
Research & Benchmarks Google's FACTS benchmark shows 70% factuality ceiling across four tests
Research & Benchmarks Nvidia's DLSS 4.5 beta adds 6x Multi Frame Generation for RTX 50 GPUs
From Archives OpenAI board member warns AI takeover of society could be months away
From Archives Former OpenAI researcher maps six Chinese AI models to Nvidia Blackwell
Common Questions Answered
How does Memorilla reduce document tokens while maintaining accuracy on frozen LLMs?
Memorilla is a latent memory module that compresses entire document collections into just 16 memory tokens, regardless of the collection size. On the FactKG benchmark, a frozen Qwen3-8B model achieved 84.33% accuracy using Memorilla's 16 tokens compared to 80.43% with the full document context, while using 108x fewer document tokens overall.
What is the practical advantage of Memorilla for long-context RAG pipelines?
Memorilla offers significant cost savings for large-scale retrieval-augmented generation systems by dramatically reducing token consumption. The 108x reduction in document tokens translates to substantial savings in token costs, which is particularly valuable when running long-context RAG pipelines where token expenses compound rapidly.
Under what license is Memorilla's code and training resources available?
Memorilla has been open-sourced by Stanford's SNAP lab under the Apache 2.0 license. The release includes not only the code but also training recipes and model checkpoints, making it freely available for public use and modification.
When does Memorilla's compression approach provide the most benefit?
Memorilla's compression strategy is most effective when there is substantial document content to compress, such as in large document collections. However, the article notes that compression benefits diminish with smaller document sets, as demonstrated by its performance on PubMedQA abstracts of only 324 tokens, where it underperformed compared to alternative methods like LLMLingua-2.
Who developed Memorilla and which institution led this research?
Memorilla was developed by Stanford's SNAP lab, led by researcher Jure Leskovec. The lab open-sourced the memory system to make it accessible to the broader research and development community.
Further Reading
- Memorilla: Latent Semantic Memory for LLMs - arXiv
- Stanford Research Open-Sources Memorilla, a Latent Memory Module for Frozen LLMs - MarkTechPost
- Sixteen tokens for a whole library: what Memorilla's compression actually buys - Traictory
- Stanford’s Memorilla squeezes entire conversation histories into 16 memory tokens - AI Frontier Post
- Memorilla: learned memory tokens for frozen LLMs - AI News
