Editorial illustration for MLP Networks Fit High-Frequency Functions One Oscillation at a Time
MLP Networks Fit High-Frequency Functions One...
Neural networks are supposed to learn the easy stuff first. The broad strokes, the low frequencies. That’s the established story of spectral bias.
It’s wrong, or at least incomplete. The real question isn’t what they learn, but how they learn the hard parts, the jagged high-frequency stuff. The answer is oddly physical.
An MLP network doesn’t smoothly absorb a whole function. It attacks from the edges. It fits high-frequency functions one oscillation at a time, starting at the boundary and working its way in.
This boundary-driven process forces a new perspective. The behavior of your target function right at the edge of its domain now matters, independently of its overall frequency content. That’s a novel and useful decoupling.
It means you can’t just think about Fourier components. You have to think about where the function starts and stops. As the network trains, you can watch its method.
It iteratively assembles a set of custom, step-like basis functions. This isn’t the smooth, innate oscillation of a SIREN network. It’s a different, more deliberate construction project.
And it proves something critical: even a relatively wide network, with 100 hidden units, isn’t just lazily applying a static kernel. It’s firmly in the feature-learning regime, building its tools from scratch.
Our argument is that MLP neural networks fit high-frequency functions from the boundaries in, learning to represent one oscillation at a time. We showed that the behavior of the target function near the boundary can have a significant effect on the training process, independent of the frequency content of the target, which we believe to be a novel insight. Furthermore, we showed that the MLP networks we studied iteratively built step-like basis functions, which contrasted dramatically with the oscillatory behavior of the basis built by a SIREN network. The basis function perspective is interesting, as it shows that even the training of relatively wide networks (\(H=100\)) is in the "feature learning" regime.
The old narrative of spectral bias is a blurry top-down view. This is the granular, bottom-up reality. Training is a sequential conquest.
The boundary sets the beat. The network follows, assembling its bespoke staircase of steps to climb the function from the edges inward. The contrast with SIREN is stark and telling.
Architecture doesn’t just change what you learn. It dictates the fundamental choreography of learning itself. If we understand this mechanism, we can start to design for it.
We could build networks that purposefully exploit this boundary-first fitting, or cleverly avoid its constraints. The next move is obvious. Stop just watching the network converge.
Start watching it build.
Related Reading
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google tests visual 'magazine-style' UI for Gemini 3 Pro users
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
From Archives SafeGene Introduces Reusable Safety-Adapter for Cross-Task Model Families
From Archives FAIR-Calib Introduces Two-Stage PTQ Framework for Diffusion LLM Quantization
Common Questions Answered
How do MLP networks fit high-frequency functions differently than previously understood?
MLP networks don't smoothly learn entire functions at once, but instead attack high-frequency functions one oscillation at a time, starting from the boundary and working inward. This sequential conquest approach contradicts the traditional narrative of spectral bias, which suggested networks learn low frequencies before tackling harder, jagged high-frequency components.
What is spectral bias and why is the article's explanation considered incomplete?
Spectral bias is the established theory that neural networks learn easy, low-frequency information first before learning complex high-frequency details. The article argues this narrative is incomplete because it doesn't explain the actual mechanism of how networks learn the hard parts—the real question isn't what they learn, but how they learn high-frequency functions.
How does the boundary condition influence MLP network training?
According to the article, the boundary sets the beat for how MLP networks learn, acting as the starting point from which networks assemble their learning process. The network follows this boundary condition, building a sequential structure to climb the function from the edges inward rather than learning uniformly across the entire function space.
What is the key difference between MLP networks and SIREN architectures in terms of learning?
The article emphasizes that architecture fundamentally dictates the choreography of learning itself, with SIREN and MLP networks showing stark contrasts in how they approach function fitting. This suggests that architectural choices don't just change what networks learn, but rather how they learn at a fundamental level.
