ML Fundamentals
Overfitting and Underfitting
Overfitting memorizes noise. Underfitting misses the pattern. This lesson explains the bias and variance trade-off, and the practical signs and fixes for each failure.
beginner · 12 min read
Lesson content
The central struggle in machine learning is finding a model that is just complex enough.
Too simple and it cannot capture the real pattern. Too complex and it starts memorizing the noise in the training data instead of the signal. Neither one works on new data, which is the only thing that counts.
Telling the two apart from a pair of numbers is a skill you will use on every project you ever run.
Definition
Overfitting and underfitting
Overfitting is when a model learns the training data too well, noise included, and then fails on new data. Underfitting is when a model is too simple to capture the pattern, so it fails on both.
Formal definition
Overfitting: low training error with high generalization error, driven by high variance. Underfitting: high error on both training and unseen data, driven by high bias.
Look at the lower half of that figure for a moment, because it carries the whole idea.
The gray line is training error. It falls as the model gets more complex, and it never stops falling. A model with enough capacity can drive training error to zero on any dataset, which is why training error alone tells you nothing.
The accent line is test error. It falls at first, because early complexity is buying real pattern. Then it turns, because later complexity is buying noise. The gap between the two lines at the right-hand edge is overfitting, made visible.
Studying for an exam
Underfitting is skimming the material so lightly that you learn nothing and fail every question.
Overfitting is memorizing last year's paper word for word. You would score full marks on that exact paper and fall apart on this year's different questions.
What you want sits between them: learning the concepts well enough that they transfer to questions nobody showed you. A student who can only answer questions they have already seen has not learned the subject, however impressive their practice scores look.
Definition
The bias-variance trade-off
Bias is error from a model being too simple to fit the pattern. Variance is error from a model so flexible that it changes wildly depending on which training data it happened to get. Lowering one tends to raise the other.
Formal definition
Expected generalization error decomposes into bias (systematic error from the model's assumptions), variance (sensitivity to the particular training sample) and irreducible noise. Model complexity trades bias against variance.
The word variance is worth pausing on, because it does not mean what it sounds like.
It is not about the spread of your data. It is about how much the fitted model would change if you handed it a different random sample from the same source.
A straight line fitted to two different samples comes out almost the same both times: low variance. A wiggly curve through every point comes out completely different: high variance. That instability is exactly why it fails on data it has not seen.
| Underfitting | Good fit | Overfitting | |
|---|---|---|---|
| Training score | Poor | Good | Excellent, sometimes perfect |
| Validation score | Poor | Good | Clearly worse than training |
| Gap between them | Small | Small | Large |
| Cause | Too simple, high bias | Complexity about right | Too flexible, high variance |
| First thing to try | Better features | Nothing, ship it | More data |
Training score
- Underfitting
- Poor
- Good fit
- Good
- Overfitting
- Excellent, sometimes perfect
Validation score
- Underfitting
- Poor
- Good fit
- Good
- Overfitting
- Clearly worse than training
Gap between them
- Underfitting
- Small
- Good fit
- Small
- Overfitting
- Large
Cause
- Underfitting
- Too simple, high bias
- Good fit
- Complexity about right
- Overfitting
- Too flexible, high variance
First thing to try
- Underfitting
- Better features
- Good fit
- Nothing, ship it
- Overfitting
- More data
How do I diagnose which one I have?
Compare two numbers: the training score and the validation score.
If both are poor, the model cannot even fit data it has already seen. That is underfitting, and you need better features or a more capable model.
If the training score is excellent and the validation score is much worse, the model has memorized the training set. That is overfitting, and the size of the gap is your measure of how badly.
If both are good and close together, you are done. Stop tuning.
Watching that gap as you change the model is how practitioners steer between the two, and it is the single most useful habit in this lesson.
| K | Training accuracy | Validation accuracy | Reading |
|---|---|---|---|
| 1 | 100% | 78% | Overfitting badly |
| 5 | 91% | 86% | Still a visible gap |
| 15 | 88% | 88% | About right |
| 50 | 81% | 80% | Starting to underfit |
| 200 | 68% | 68% | Underfitting badly |
K = 1 scores exactly 100% on training data, every time, on any dataset. Each point's nearest neighbor is itself, so it always predicts its own label. A perfect training score that means nothing at all is the clearest example of why the second column exists.
Example
Reading that table the way a practitioner would
Start at the top. K = 1 gives a 22-point gap, which is a flashing red light.
As K rises, the model is forced to consult more neighbors, so it can no longer carve out a tiny region around every single training point. The gap closes.
At K = 15 the two columns agree, and the validation score is at its highest. That is the answer.
Past that, both numbers fall together. The gap stays small, which tells you this is no longer overfitting; the model has simply become too blunt to see the pattern. Two failures that look nothing alike in the numbers, even though both produce a bad model.
Fixing overfitting with a simpler model is the second-best answer
Every list of fixes for overfitting puts simplification near the top: fewer features, a shallower tree, stronger regularization. Those work, and they cost you something. A simpler model has more bias, so you are trading one error for another.
More data does not trade anything away. It lets you keep the capacity and still generalize, because there is less room for the model to find coincidences.
So the order is: get more data if you can, augment what you have if you cannot, and only then start cutting capacity. Beginners reach for the last option first because it is the only free one.
Declaring victory on a high training score.
A training score is what the model can recite, not what it has learned. It is only interesting next to a validation score.
Assuming any gap between training and validation means overfitting.
A small gap is normal and healthy. Worry about the size of the gap and whether validation is still improving, not about the gap existing.
Tuning until the test score looks good.
The moment you choose something because of a test score, the test set has become a validation set and your final number is optimistic. Tune on validation, touch test once.
Treating underfitting as the safe failure.
An underfit model is confidently wrong on everything and gives you no signal about what to fix. An overfit model at least proves the pattern is learnable.
Believing complexity means number of parameters.
A heavily regularized network with millions of parameters can behave far more simply than a small unregularized tree. Effective complexity is what the trade-off is about.
Key takeaway
- Underfitting is too simple to learn the pattern. Overfitting memorizes noise instead of signal.
- Bias comes from being too simple, variance from being too flexible, and you trade one against the other.
- Diagnose from two numbers: the training score, the validation score and the gap between them.
- Training error always falls with complexity. Test error falls, turns and climbs, and the turn is what you are looking for.
- Fix underfitting with better features or more capacity. Fix overfitting with more data first, then less capacity.
- A perfect training score is a warning sign rather than an achievement.
Finished this lesson?
Guest progress is saved on this device.