Editorial illustration for MLP Networks Fit High-Frequency Functions One Oscillation at a Time
MLP Networks Fit High-Frequency Functions One...
MLP Networks Fit High-Frequency Functions One Oscillation at a Time
Why does a neural net sometimes crawl when asked to capture a sharp spike? The answer lies in a phenomenon first highlighted in 2019: the spectral bias. As Conor Rowan and Finn Murphy‑Blanchard explain, networks tend to fit a regression target’s low‑frequency components before tackling the higher frequencies that give a function its fine detail.
Figure 1 in their paper illustrates this stepwise progression—first a smooth curve, then a gradual sharpening as training continues. Because the “frequency content” is defined by the target’s Fourier transform, learning a high‑frequency function can demand many epochs. Subsequent studies have confirmed the difficulty, pointing to the rapid decay of the Fourier spectra of common activations such as ReLU, tanh, and sigmoid.
An influential line of work uses the Neural Tangent Kernel to model an infinitely wide network as a linear dynamical system; there, convergence speed is shown to be inversely tied to a mode’s frequency. The result is a clear, theory‑backed picture of why low‑frequency patterns dominate early learning, while high‑frequency details lag behind.
Our argument is that MLP neural networks fit high-frequency functions from the boundaries in, learning to represent one oscillation at a time. We showed that the behavior of the target function near the boundary can have a significant effect on the training process, independent of the frequency content of the target, which we believe to be a novel insight. Furthermore, we showed that the MLP networks we studied iteratively built step-like basis functions, which contrasted dramatically with the oscillatory behavior of the basis built by a SIREN network. The basis function perspective is interesting, as it shows that even the training of relatively wide networks (\(H=100\)) is in the "feature learning" regime.
Why this matters
We’ve seen the spectral bias narrative dominate discussions of how multilayer perceptrons learn. This work nudges us toward a more nuanced view: MLPs may be stitching high‑frequency detail into a model starting at the edges, adding one oscillation after another. If the function’s boundary behavior can steer training independently of its overall frequency makeup, then data preprocessing or domain‑specific padding could become a lever for faster convergence.
For developers, the implication is modest—adjusting boundary conditions might shave a few epochs off a training run, but the paper stops short of quantifying gains across architectures. Researchers gain a concrete hypothesis to test: does “boundary‑inward” fitting hold for deeper networks or convolutional layers?
Founders should stay cautious. The authors demonstrate an effect, yet they do not claim it solves the broader challenges of high‑frequency learning or guarantee robustness in noisy settings. Until replication studies confirm the phenomenon at scale, we remain uncertain whether this insight will translate into practical tooling or remain a theoretical curiosity.
Further Reading
Related Reading
LLMs & Generative AI LWiAI Podcast #228: OpenAI unveils GPT-5.2, Runway rolls out first world model
LLMs & Generative AI OpenAI's Codex powers Lovable AI, letting millions create apps from text
LLMs & Generative AI Google releases FunctionGemma, a tiny model for natural-language mobile control
From Archives SafeGene Introduces Reusable Safety-Adapter for Cross-Task Model Families
From Archives