Editorial illustration for Princeton Researcher's RLT Fixes 96 Blocks per Token with Unbounded Depth
Princeton's RLT Fixes LLM Context Recomputation Issue
Princeton Researcher's RLT Fixes 96 Blocks per Token with Unbounded Depth
Every decoder-only language model in wide use today has the same blind spot at inference time. Compute the final layer for token 5, and none of that work carries over when the model starts token 6. Each new position has to rebuild context from scratch, leaning entirely on attention over cached keys and values rather than on anything resembling a running internal state. Yifan Zhang, a researcher at Princeton, has written a technical report proposing an architecture that closes this gap: the Recurrent Looped Transformer, or RLT.
The design pairs a causal encoder with a recurrent decoder, and the decoder is where the departure happens. Instead of resetting between tokens, RLT carries forward a complete state made up of the decoder's last hidden output and its sliding-window attention cache at every layer, unbroken across the prompt and the generated response. Zhang's report lays out the architecture, the execution schedule, and a reinforcement-learning replay contract for it. It's a specification, not a benchmarked system, and it says so directly: no measured results on efficiency, reasoning quality, or scaling are included yet.
The decoder’s final hidden state and its layerwise sliding-window attention (SWA) cache are carried into the next token, across both prompt and response, with no reset at the boundary.
Why this matters
RLT is a proposal on paper, not a benchmark result, and that gap matters. Carrying decoder state across token boundaries could give transformers something like persistent memory instead of the stateless-position trick attention has relied on. But Zhang's own Appendix B flags the hard part: the state-to-state Jacobian has cross terms running through the decoder KV cache, so you can't just detach one tensor and call it truncated BPTT.
Every detached tensor has to be named, or gradients leak through the cache in ways that are easy to miss and hard to debug. For anyone building multi-turn serving on top of this, that means exact prefix snapshots, not approximations, which is a real engineering tax before you get to training stability or throughput numbers. We'd treat this as a serious architectural idea worth tracking, not something to prototype into production yet.
The next thing to watch is whether Zhang or anyone else publishes actual training curves and memory-cost measurements against standard SWA baselines. Until then, 96 blocks per token is a design claim, not a demonstrated result.
Related Reading
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google tests visual 'magazine-style' UI for Gemini 3 Pro users
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
From Archives GPT-6 Astra Rated Stronger Economic Performer by Andon Labs
From Archives OpenAI says GPT-6 Astra requires leaner prompts, fewer guardrails
Common Questions Answered
What is the main limitation of decoder-only language models that Yifan Zhang's RLT architecture addresses?
Current decoder-only language models have a blind spot where compute work for one token does not carry over to the next token, forcing each new position to rebuild context from scratch using only attention over cached keys and values. Zhang's Recurrent Looped Transformer (RLT) closes this gap by carrying the decoder's final hidden state and layerwise sliding-window attention cache across every token boundary without resetting.
How does RLT's approach to maintaining state differ from traditional transformer attention mechanisms?
Traditional transformers rely on a stateless-position trick using attention mechanisms, whereas RLT proposes giving transformers something like persistent memory by carrying decoder state across token boundaries. This means the final hidden state and sliding-window attention cache persist from one token to the next, both during prompt and response generation, creating a more continuous computational flow.
What technical challenge does Yifan Zhang identify in implementing RLT's state-to-state transitions?
Zhang notes in Appendix B that the state-to-state Jacobian has cross terms running through the decoder KV cache, which prevents simple tensor detachment for truncated backpropagation through time (BPTT). Every detached tensor must be explicitly named and tracked to properly compute gradients, making the implementation more complex than standard transformer training.
What does RLT fix regarding the number of blocks per token?
According to the headline, RLT fixes 96 blocks per token with unbounded depth, suggesting it resolves computational inefficiencies or redundancies that occur in standard decoder-only models when processing sequential tokens. This improvement relates to how the model maintains and utilizes state information across the entire depth of the architecture.
Why is RLT currently considered a proposal rather than a proven advancement?
RLT is a technical proposal on paper rather than a benchmark result, meaning it has not yet been empirically validated through testing and performance measurements. While carrying decoder state across token boundaries is theoretically promising for giving transformers persistent memory capabilities, the practical implementation challenges and actual performance gains remain to be demonstrated.
Further Reading
- Recurrent Looped Transformer - Project page
- Recurrent Looped Transformer: Infinite Depth Claim (2026) - ExplainX AI
- KV Cache for LLM Inference: Memory, Reuse, and Tuning - QubitTool
- GPT Decoder-Only - LLMBase - LLMBase
- FOCUS: DLLMs Know How to Tame Their Compute Bound - arXiv
