Neural Networks
Neural Networks
Neural networks build predictions from simple neurons stacked in layers. This lesson covers a single neuron, forward propagation and how backpropagation trains one.
advanced · 18 min read
Lesson content
Logistic regression predicts a category by weighing features and passing the result through a curve.
A neural network takes that exact idea and repeats it: many such units side by side, then the whole arrangement stacked in layers, each layer reading what the layer below it reported.
That repetition is the whole difference. One logistic regression can only draw one boundary. A stack of them can draw a boundary of almost any shape, which is what lets a network learn patterns a single curve could never capture.
Definition
Neural network
A model built from many small decision-making units, called neurons, arranged in layers, where each layer's output feeds into the next.
Formal definition
A machine learning model composed of layers of interconnected units (neurons), each computing a weighted sum of its inputs plus a bias and passing the result through a non-linear activation function, trained end-to-end by adjusting every weight to reduce a loss function.
Deciding whether to go for a run based on the weather
Suppose you decide whether to go for a run by weighing three clues: the temperature, the chance of rain, and how windy it is.
You do not weigh them equally. Rain might matter to you far more than wind, so you mentally multiply each clue by how much you personally care about it and add up the results.
Then you factor in your baseline mood for the day, which shifts the whole calculation up or down regardless of the weather. That is the bias term.
If the total clears some threshold, you go. That weighing, summing, shifting and threshold-checking is exactly what one neuron does, with numbers instead of weather.
Inside a single neuron
Receive inputs
Take in a set of numbers, the features, like temperature, rain chance and wind speed.
Multiply each input by a weight
Every input has its own weight, a number that says how much that particular input matters to this neuron's decision.
Add them up, plus a bias
Sum all the weighted inputs together, then add one more number, the bias, which shifts the neuron's baseline tendency up or down, independent of any input.
Pass the total through an activation function
Feed that sum into an activation function, which reshapes it, commonly squashing it into a range like 0 to 1, or zeroing out negative values entirely.
Output the result
The activation function's output is the neuron's single output number, ready to be used directly or fed into the next layer of neurons.
Why bother with an activation function instead of just using the weighted sum directly?
A weighted sum, on its own, can only ever draw a straight-line boundary between categories.
Stacking more of them changes nothing. Without an activation in between, fifty layers add up to one giant straight line, because a linear function of a linear function is still linear. The depth is real and the expressive power is not.
An activation function introduces a bend. Stacking many bent layers lets a network build genuinely curved decision boundaries, and with enough of them it can approximate essentially any shape.
This is important enough that a later lesson is devoted entirely to which bend to use.
A single neuron behaves a lot like logistic regression: weigh the inputs, add a bias, squash the result.
A network's power comes from arranging many of them into layers. An input layer receives the raw features. One or more hidden layers sit in between. An output layer produces the final prediction.
Each neuron in a layer receives the outputs of every neuron in the layer before it. So the second layer is not looking at your data; it is looking at what the first layer made of your data, which is what lets later layers work with increasingly abstract things.
They are called hidden layers only because their outputs are never observed directly. Nothing about them is mysterious, and you can print them.
Definition
Forward propagation
Feeding an example's features through every layer of the network, one layer at a time, until the output layer produces a prediction.
Formal definition
The process of computing a network's output by applying each layer's weights, biases and activation function in sequence, from the input layer to the output layer.
Why does a network need hidden layers at all, instead of just one layer?
Because some real patterns cannot be separated by any single boundary, however it is bent.
Picture data shaped like two nested rings, one class inside the other. No straight cut separates them, and neither does any single curve of the kind one layer can draw.
A hidden layer lets the network learn a few simple partial patterns first, then combine those in the next layer into something more complex. Two half-boundaries that individually separate nothing can be combined into one that works.
It is the same reason a few simple shapes can be assembled into almost any complicated one. Depth buys composition, and composition is where the power is.
Definition
Loss
A single number that measures how wrong the network's predictions are on a batch of examples, bigger loss means worse predictions.
Formal definition
A scalar function of the network's predictions and the true labels, chosen so that its minimum corresponds to the network making its most accurate possible predictions.
How does the network know which weights to change and by how much?
It uses backpropagation.
Forward propagation produces a prediction and the loss measures how wrong it was. Backpropagation then works from the output layer back to the input, applying the chain rule at each step to calculate how much each individual weight contributed to that error.
Every weight then gets nudged a small amount in whichever direction would have reduced the error. That is the same downhill-nudging idea as linear regression's gradient descent, carried out layer by layer through the whole network.
The reason it is worth a name of its own is efficiency. Computing each weight's effect separately would be hopeless at this scale; backpropagation gets all of them in one backward sweep that costs about as much as the forward pass did.
That forward-then-backward cycle — predict, measure the loss, work backward, nudge every weight a little — repeats over and over across the training data.
Three settings control how the repetition happens, and each is worth knowing on its own, because getting any of them badly wrong will stop the network learning no matter how good the architecture is.
Definition
Epoch
One complete pass through the entire training dataset.
Formal definition
A single full iteration in which every training example has been used once for a forward pass, loss calculation and backward weight update.
Definition
Batch size
How many training examples the network looks at before it updates its weights once.
Formal definition
The number of training examples processed together in one forward-and-backward pass before the weights are adjusted. An epoch is made up of many batches unless the batch size equals the full dataset.
Definition
Learning rate
How big a step the network takes when nudging a weight during each update.
Formal definition
A hyperparameter that scales the size of each weight adjustment during training. Too high causes unstable, overshooting updates and too low makes training impractically slow.
Example
A network that a single neuron cannot replace
Consider a tiny dataset with two binary switches, where the correct output is "yes" only when exactly one of them is on. Both off means no. Both on also means no.
Plot those four cases and the two "yes" points sit on opposite corners of a square, with the two "no" points on the other two corners. No straight line separates them, and no amount of moving one line around will help.
So one neuron always gets some of these four examples wrong, however long you train it. This is not a training failure; the model simply cannot express the answer.
A network with one hidden layer of two neurons solves it. Each hidden neuron learns one straight boundary, neither of which separates the classes alone, and the output neuron combines them into the bent shape the pattern actually needs.
This problem is called XOR, and it stalled the field for over a decade until people worked out how to train the hidden layer.
from sklearn.neural_network import MLPClassifier# The XOR-style pattern: output is 1 only when exactly one input is 1X = [[0, 0], [0, 1], [1, 0], [1, 1]]y = [0, 1, 1, 0]model = MLPClassifier(hidden_layer_sizes=(4,), activation="relu", learning_rate_init=0.05, max_iter=5000, random_state=0)model.fit(X, y)print("predictions:", model.predict(X))from sklearn.neural_network import MLPClassifierMLPClassifier (multi-layer perceptron) is scikit-learn's implementation of the layered, neuron-based network described above.
model = MLPClassifier(hidden_layer_sizes=(4,), activation="relu", learning_rate_init=0.05, max_iter=5000, random_state=0)hidden_layer_sizes=(4,) adds one hidden layer of 4 neurons between the input and output, exactly the kind of hidden layer that lets the network bend its boundary. learning_rate_init sets the step size from the definition above and max_iter caps how many training epochs it is allowed to run.
model.fit(X, y).fit(X, y) runs forward propagation, computes the loss and runs backpropagation repeatedly until the loss stops improving or max_iter is reached.
print("predictions:", model.predict(X))After enough training, .predict(X) should recover the correct "exactly one switch on" pattern, something a single neuron, without a hidden layer, could not learn.
When to use
- The relationship between inputs and output is too complex or curved for simpler models like logistic regression or a decision tree to capture well
- You have a large amount of training data, neural networks generally need more data than simpler models to train reliably
- The problem involves rich, high-dimensional inputs like images, audio, or text, where deep learning architectures excel
When to avoid
- You have a small dataset, a neural network can easily overfit when there is not much data to learn from
- You need a model whose reasoning is easy to explain to a person, a network's many weights do not translate into simple, readable rules
- A simpler model already performs well, added complexity is only worth it when it buys real improvement
Assuming you need to understand the underlying calculus to use neural networks effectively.
Frameworks and libraries compute the backward pass automatically, what matters day to day is the intuition (predict, measure error, nudge weights backward) and tuning settings like learning rate and epochs.
Setting the learning rate far too high, expecting training to go faster.
A learning rate that is too high causes the loss to bounce around or blow up instead of steadily decreasing, if training looks unstable, try a smaller learning rate first.
Assuming a bigger network with more layers is always better.
A larger network needs more data and more training time and without enough data it tends to overfit rather than perform better, match network size to the amount of data available.
Key takeaway
- A single neuron computes a weighted sum of its inputs plus a bias, then passes the result through an activation function.
- Stacking neurons into layers, with an activation function at each one, lets a network learn curved patterns a single neuron never could.
- Epoch, batch size and learning rate each control a different part of how training repeats and how big each weight update is.
- Backpropagation works backward from the loss to figure out how much each weight contributed to the error, then nudges every weight to reduce it.
Finished this lesson?
Guest progress is saved on this device.