Editorial illustration for MEMO trains a memory model on new knowledge with two roles, no LLM changes
MEMO trains a memory model on new knowledge with two...
Large language models have an embarrassing secret: once trained, they cannot learn new facts without expensive retraining or risky fine-tuning. MEMO shatters that limitation with a simple but radical architectural split. Two models, two roles, one pipeline.
A small, dedicated MEMORY model absorbs fresh knowledge from raw documents, facts, cross-document relationships, everything. The EXECUTIVE model? Frozen.
Untouched. It never sees the source material. Instead, it asks targeted sub-questions, reasons over the MEMORY model’s answers, and delivers the final response.
The result works with any LLM, including closed APIs. No weights, no logits, no fine-tuning. Just black-box access and a five-step training pipeline that extracts, consolidates, verifies, surfaces entities, and synthesizes across documents.
MEMO proves that memorization and reasoning can be cleanly decoupled, and that new knowledge need not force the core model to change a single parameter.
Complex user queries are decomposed across three sequential stages. No documents are retrieved — all answers come from internalized parametric knowledge.
The implications are clear. MEMO doesn’t ask the LLM to change. It doesn’t need to.
Instead, it builds a separate, specialized brain, lean, trained, and precise, that absorbs new knowledge without disturbing the executive’s architecture. The executive stays frozen, safe, and universal. The memory model does the heavy lifting, learning facts and weaving cross-document threads until the knowledge lives in its parameters.
At inference, the source documents vanish. Only the learned patterns remain. This is not a patch.
It is a structural shift. By decoupling memory from reasoning, MEMO sidesteps the brutal trade-offs of fine-tuning. No catastrophic forgetting.
No weight leaks. No need to touch a single parameter in your production LLM. The executive model, whether open-weight or locked behind an API, simply asks better questions.
The memory model answers from what it has internalized. That’s it. The five-step pipeline is the engine.
Fact extraction, consolidation, verification, entity surfacing, and cross-document synthesis. Each step tightens the signal. The last step, cross-document synthesis, is where the real magic happens.
It forces the memory model to connect dots across sources, to build relationships, to think in networks rather than isolated facts. The result is a model that doesn’t just recall. It understands context.
And it works with a 14-billion-parameter model as the memory bank. No weights exposed. No logits required.
Just black-box API calls. That is the quiet revolution here: you can inject new knowledge into a system without ever touching the system itself. The executive stays pristine.
The memory model does the learning. The two roles never blur. This is how we scale knowledge without breaking the model.
This is how we keep the reasoning engine frozen while the world keeps changing. MEMO proves that the future of LLM adaptation isn’t bigger models or more fine-tuning. It’s smarter architecture.
Two roles. One job. No compromises.
Related Reading
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google tests visual 'magazine-style' UI for Gemini 3 Pro users
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
From Archives GEM framework casts LLM data curation as hyperspherical variational problem
From Archives Experienced users supervise Claude only when it deviates, not step‑by‑step
Common Questions Answered
How does MEMO allow language models to learn new knowledge without retraining or fine-tuning?
MEMO uses an architectural split with two separate models: a dedicated MEMORY model that absorbs fresh knowledge from raw documents and cross-document relationships, while the EXECUTIVE model remains frozen and untouched. This approach eliminates the need for expensive retraining or risky fine-tuning of the main language model, as the memory model handles all new knowledge acquisition independently.
What is the role of the EXECUTIVE model in the MEMO pipeline?
The EXECUTIVE model in MEMO remains frozen and never sees the source material directly. Instead, it asks targeted sub-questions to the MEMORY model, which provides the learned knowledge without requiring any changes to the executive's original architecture or parameters.
How does the MEMORY model in MEMO process and store new information?
The MEMORY model is a small, specialized, and dedicated component that absorbs new knowledge by learning facts and weaving cross-document threads until the knowledge lives in its parameters. It trains on raw documents and cross-document relationships, becoming lean, trained, and precise in its knowledge representation.
What happens to source documents at inference time in the MEMO system?
At inference time in MEMO, the source documents vanish completely, and only the learned patterns that the memory model has absorbed remain. This means the system operates efficiently without needing to access or reference the original training documents during deployment.
Why is keeping the EXECUTIVE model frozen important in the MEMO architecture?
Keeping the EXECUTIVE model frozen ensures it remains safe and universal, preserving its original capabilities and preventing any disruption to its established architecture. By delegating all new knowledge learning to the separate MEMORY model, MEMO protects the integrity and reliability of the core language model while enabling continuous knowledge updates.
Further Reading
- MeMo: Memory as a Model — arXiv
- MeMo: Memory as a Model — OpenReview
- Memorization vs. Reasoning: Updating LLMs with New Knowledge — ACL Anthology
- MeMo: A New Way to Keep Language Models Updated Without Changing the Core Model — MachineBrief
- MemoRAG – Enhance RAG with memory-based knowledge — Hacker News
