Visualizing transformer paths: Qwen, Llama, Gemma models show 1.60-2.24x order effect alignment.

Editorial illustration for Order effect aligns Qwen, Llama, Gemma transformer paths 1.60–2.24x

Order effect aligns Qwen, Llama, Gemma transformer paths...

By AI Daily Post Edited by Brian Petersen, Editor-in-Chief

October 21, 2026 • 3 min read

A team testing three open-weight model families, Qwen, Llama and Gemma, found that scrambling the order of a transformer's layers doesn't just degrade performance randomly. It changes behavior in a specific, measurable way, with effects ranging from 1.60x to 2.24x depending on the model. That's the kind of finding that doesn't fit neatly into the usual toolkit for studying how large language models work.

Most mechanistic interpretability research looks for function in parameters, attention heads or individual neurons. That approach can tell you which weight matters for which output, but it says nothing about how a single forward pass actually unfolds step by step through the network. The layers fire in sequence, each one handing its output to the next, and that sequence itself has rarely been treated as something worth measuring on its own terms.

This work takes a different angle. Instead of asking what a layer does, it asks what happens when you track the full trajectory of layer states as a single ordered process, then test whether that order is doing real computational work rather than just reflecting arbitrary architecture choices.

Across representative Qwen, Llama and Gemma checkpoints, realized order produced paths 1.60--2.24 times as aligned with their endpoint chord as endpoint-preserving shuffles. The order effect transferred to a lexical task, but was stronger in three random initializations of the same Qwen architecture, identifying an architecture-compatible scaffold rather than a training-specific law.

High-dimensional layer-state trajectories reveal the internal dynamics of transformer inference - ArXiv Machine Learning

Why this matters

This result gives researchers a concrete way to test whether a transformer's layer order is doing real work, rather than assuming it from architecture diagrams alone. The 1.60–2.24x alignment figure, consistent across Qwen, Llama and Gemma, suggests the ordered path itself carries information that random shuffles destroy. That's worth attention for anyone building interpretability tools or debugging model behavior, since it points to trajectory shape as a measurable signal, not just a metaphor.

The more interesting wrinkle is the finding on three random initializations of the same Qwen architecture: the order effect showed up stronger there than in the cross-architecture comparison. That's odd, and it deserves scrutiny before anyone builds conclusions on top of it. Does initialization seed matter more than architecture family for this kind of structural signal?

If so, claims about "what Llama does differently from Qwen" need a harder look, because some of that difference might just be noise from training runs, not design. We'd want this replicated on more checkpoints and more tasks before treating it as settled.