ML Fundamentals
Training, Validation and Test Sets
Splitting data into training, validation and test sets is how you measure a model honestly. This lesson explains each set's job and why the test set may be touched only once.
beginner · 12 min read
Lesson content
A model that has memorised its training data will ace a test drawn from that same data. It will also tell you nothing whatsoever about how it handles the real world.
To measure a model honestly you have to judge it on examples it has never seen. That one requirement is the reason data gets cut into separate pieces.
The cut is not arbitrary. Each piece has a job, and the jobs differ mainly in how often you are allowed to look.
- Training set
The largest slice. The examples the model actually fits its parameters to.
This is where learning happens. A score here tells you almost nothing on its own, because the model was built to do well on it.
- Validation set
A held-out slice used to compare models and tune hyperparameters while you are still developing.
It stands in for unseen data during the part of the project where you keep changing your mind, which is what keeps the test set clean.
- Test set
A slice locked away and opened exactly once, at the very end, to estimate real-world performance.
It is your only honest number. Everything else you measured was measured on data that influenced your choices.
The sizes matter less than people expect. What matters is how often each slice gets consulted.
The training set is read on every single pass of training, thousands of times. The validation set is read once per candidate model, so perhaps a few hundred times across a project. The test set is read once, ever.
That descending frequency is also a descending scale of contamination. The more often a slice influences your decisions, the less it can tell you about data you have genuinely never seen.
| Rows available | Train | Validation | Test | Rows in each held-out slice |
|---|---|---|---|---|
| 1,000 | 60% | 20% | 20% | 200 each |
| 100,000 | 80% | 10% | 10% | 10,000 each |
| 10,000,000 | 98% | 1% | 1% | 100,000 each |
The percentages shrink as the dataset grows, and the absolute sizes grow anyway. That is the right way round. A held-out slice only needs to be large enough to measure a score reliably; once it is, every extra row is better spent on training.
Why can I not just reuse the test set to check progress as I go?
Because every time you look at the test set and then change something in response, a little of that set leaks into your decisions.
Do it once and the damage is negligible. Do it forty times across a project, picking whichever variant scored best, and you have effectively fitted your choices to that specific test set. Its score becomes optimistic and stops predicting anything.
This is the same mechanism as overfitting, one level up. The model overfits the training set; you overfit the test set.
That is what the validation set is for. Consult it as often as you like. The test set stays sealed until you are finished choosing.
Definition
Data leakage
When information from outside the training set sneaks into training, making the model look better than it truly is.
Formal definition
The use of information during training that would not be available at prediction time, including any statistic computed over the full dataset before splitting, leading to over-optimistic performance estimates.
Split before you do anything else at all
This is the rule beginners break most often, and it is invisible when you break it.
Compute a mean to fill missing values, fit a scaler, pick features by correlation, or even just look at the overall distribution, and if you did it on the whole dataset then test information is already inside your model.
Split first. Then fit every preprocessing step on the training split alone, and apply the fitted step to validation and test without refitting.
A leaked pipeline does not crash or warn you. It quietly gives you a number that is too good, and you only discover the truth in production.
Example
How leakage actually shows up
A team building a hospital readmission model scales every numeric column before splitting, using the mean and standard deviation of the full dataset.
That is enough. The scaler has now seen the test patients, so each training row is expressed in units that encode something about the test set. Their validation score comes out around two points higher than it should.
Two points sounds survivable. The problem is that they then use that inflated number to choose between two models, and pick the wrong one.
Leakage rarely destroys a project outright. It quietly corrupts the comparisons you make, which is worse, because you never find out which decision it spoiled.
from sklearn.model_selection import train_test_split# Step one, before anything else: carve off the test set.X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=42, stratify=y)# Split what's left again to get a validation set.X_train, X_val, y_train, y_val = train_test_split( X_train, y_train, test_size=0.25, random_state=42, stratify=y_train)# 0.8 * 0.25 = 0.2, so the final split is 60 / 20 / 20.print(len(X_train), len(X_val), len(X_test))# Step one, before anything else: carve off the test set.
X_train, X_test, y_train, y_test = train_test_split(The very first step. Everything that follows, scaling, feature selection, model choice, happens without these rows.
X, y, test_size=0.2, random_state=42, stratify=yrandom_state fixes the shuffle so the split is reproducible. stratify=y keeps the class balance the same in every slice, which matters enormously when one class is rare.
X_train, y_train, test_size=0.25, random_state=42, stratify=y_trainNote this splits X_train, not X. Splitting X again here is a common slip that puts test rows back into validation.
# 0.8 * 0.25 = 0.2, so the final split is 60 / 20 / 20.Taking 25% of the remaining 80% gives 20% of the original, which is why this second test_size is 0.25 rather than 0.2.
| Question it answers | How often you may look | What a good score proves | |
|---|---|---|---|
| Training set | Can the model fit at all? | Constantly | Very little on its own |
| Validation set | Which model and settings are best? | Every time you tune | This variant beat the others |
| Test set | How will this do in the real world? | Once | A number you can put in a report |
Training set
- Question it answers
- Can the model fit at all?
- How often you may look
- Constantly
- What a good score proves
- Very little on its own
Validation set
- Question it answers
- Which model and settings are best?
- How often you may look
- Every time you tune
- What a good score proves
- This variant beat the others
Test set
- Question it answers
- How will this do in the real world?
- How often you may look
- Once
- What a good score proves
- A number you can put in a report
Homework, mock exam, real exam
The training set is your homework. You work through it repeatedly and you can check the answers as often as you like.
The validation set is the mock exam. You sit it, see where you went wrong, revise and sit another one. That is entirely legitimate and it is how you improve.
The test set is the real exam. You sit it once. If somebody let you see the paper beforehand, your grade stops meaning anything, no matter how well you did.
The temptation, always, is to peek at the real paper because it is right there in the folder. Do not.
Scaling or imputing before splitting.
Fit every transformation on the training split and apply it to the others. A scikit-learn Pipeline makes this the default rather than something you must remember.
Splitting time-series data randomly.
A random split lets the model train on Thursday to predict Wednesday. For anything with a time order, cut by date: train on the past, test on the future.
Splitting rows when the rows are not independent.
Ten photographs of the same patient must all land in the same slice. Split by patient, not by photograph, or the model recognises the patient rather than the disease.
Reporting the best of several test scores.
The best of five looks is not an honest estimate of anything. If you have already looked more than once, say so, or gather fresh data for a clean final number.
Skipping the validation set and tuning on test.
This is the most common shortcut and it silently invalidates your headline number. If data is too scarce for three slices, use cross-validation on the training portion rather than borrowing the test set.
Key takeaway
- A score on training data is misleading, because the model was built to do well on exactly those rows.
- Training fits the model, validation guides your choices and test gives one final honest estimate.
- The three sets differ mainly in how often you are allowed to consult them.
- Use the test set exactly once. Reusing it means you overfit it yourself.
- Split before any preprocessing, or statistics from the held-out rows leak into training.
- As datasets grow the held-out percentages shrink while the held-out row counts still rise.
Finished this lesson?
Guest progress is saved on this device.