Researchers Extract ViT Hidden Layers via Black-Box Queries

A team of researchers has found a way to pull the hidden internal structure out of a transformer's feed-forward network using nothing but the outputs it hands back, no access to weights, gradients, or activations required. The target is the FFN branch itself, the two-layer block with GELU or SiLU activations that sits inside every transformer layer and does a lot of the model's actual computational work. Rather than guessing at weights directly, the attack goes after curvature: feeding chosen inputs and measuring how the output bends in response, then using that second-order signal to reconstruct the rank-one directions baked into the network's hidden layer.

The method treats the problem as a decomposition puzzle, working out when those directions can be pinned down uniquely and how to do it with as few queries as possible. Tested against CIFAR-10 vision transformers trained independently of each other, the approach recovers those hidden directions with strong consistency across separate models, repeated attempts, and every transformer block in the stack. What follows is a look at just how few queries it takes, and how well the recovered structure holds up once someone tries to rebuild a working substitute from it.