Editorial illustration for Combining hidden neurons still yields a line, highlighting activation's role
Combining hidden neurons still yields a line,...
Adding neurons should, in theory, grant a network more power. It doesn't. A 1970 textbook by Minsky and Papert holds the stubborn math: combine any number of purely linear neurons, and your final output is just addition and multiplication.
You cannot coax a curve from that. This dead end is why a simple spreadsheet formula can't learn a face. Real-world data—stock prices, weather patterns—is messy, full of loops and cliffs.
To escape, networks employ a brutal, simple trick. They break the line.
Even though we combined the outputs of multiple hidden neurons, the final result is still a line. This brings us to the most important concept in deep learning and neural networks: Activation Functions. Activation Functions Here, our data follows a non-linear pattern, but a straight line can only model linear relationships.
No matter how many linear neurons we combine, the output is still a linear function. Then how do these neural networks learn complex patterns such as curves, shapes, images, and text? We passed the outputs of the two hidden neurons directly to the output layer.
But instead of passing them directly to the output layer, the hidden neurons' outputs are first transformed using a special function called an activation function. It introduces non-linearity into the network, allowing it to learn complex patterns.
That’s the entire magic. The activation function is a gatekeeper. It takes a neuron's linear output and crushes it.
The common ReLU function, for instance, ruthlessly turns all negative values to zero. This single, non-linear operation shatters the straight line. Suddenly, chains of these bent outputs can approximate any curve—a principle formalized by the 1989 Universal Approximation Theorem.
Without this, you have fancy multiplication. With it, you have the foundation for Siri's speech recognition and DeepMind's protein folding predictions. The neuron calculates.
The activation makes it matter.
Related Reading
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google tests visual 'magazine-style' UI for Gemini 3 Pro users
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
From Archives Three NLTK tricks, including MWETokenizer, preserve domain terms in NLP
From Archives Sakana AI Fugu Ultra aims to match models; base Fugu low‑latency coding and chat
Common Questions Answered
Why can't purely linear neurons create curves in neural networks?
According to Minsky and Papert's 1970 textbook, combining any number of purely linear neurons produces only addition and multiplication operations, which cannot generate curves. This mathematical limitation means that linear neurons alone cannot learn complex, non-linear patterns found in real-world data like stock prices or weather patterns.
What role does the activation function play in breaking linearity?
The activation function acts as a gatekeeper that takes a neuron's linear output and applies a non-linear transformation to it. For example, the ReLU function ruthlessly turns all negative values to zero, which shatters the straight line and allows chains of bent outputs to approximate any curve.
How does the Universal Approximation Theorem relate to activation functions?
The 1989 Universal Approximation Theorem formalizes the principle that chains of non-linear activated neurons can approximate any curve. This theorem demonstrates that with activation functions, neural networks have the mathematical foundation to learn complex patterns, enabling applications like Siri's speech recognition.
What is the difference between a spreadsheet formula and a neural network in learning faces?
A simple spreadsheet formula cannot learn faces because it relies on purely linear operations, which cannot coax curves from data. Neural networks overcome this limitation through activation functions that introduce non-linearity, allowing them to learn the complex, curved patterns necessary to recognize facial features.
Further Reading
- A primer on neural networks — PMC / NIH
- Why & how two or more hidden layers w/ nonlinear activation ... — Neural Networks from Scratch
- A Unique Perspective on What Hidden Layers and Activation ... — Daily Dose of Data Science
- Inside a Neural Network: How Hidden Layers, Weights, and Biases Work — DEV Community
