ML Fundamentals
Evaluating Model Performance
The right metric depends on what a mistake costs. This lesson covers accuracy, precision, recall, F1 and the confusion matrix for classification, plus MAE, RMSE and R² for regression.
beginner · 14 min read
Lesson content
"How good is the model?" has no single answer. It depends entirely on what a mistake costs.
A cancer screen that misses real cases fails in a completely different way from a spam filter that quarantines a real email. Both might report 95%, and only one of them is in trouble.
Picking the metric is a modeling decision every bit as consequential as picking the algorithm. Choose the wrong one and it will hide precisely the failure you most needed to see.
Why is accuracy not enough?
Accuracy is the fraction of predictions that are correct. That is a fine summary when the classes are roughly balanced and a terrible one when they are not.
Suppose 1% of card transactions are fraudulent. A model that answers "not fraud" to absolutely everything, that contains no logic at all, is 99% accurate.
It catches nothing. It would be equally accurate if you replaced it with a piece of cardboard. The headline number is high because the common class dominates it, and the only class anybody cares about has vanished into the rounding.
Whenever one class is rare, or its errors are expensive, you need metrics that look at the classes separately.
| Metric | Value | What it is actually telling you |
|---|---|---|
| Accuracy | 99.0% | 99,000 legitimate transactions were correctly left alone |
| Recall | 0% | Not one of the 1,000 frauds was caught |
| Precision | undefined | It never flagged anything, so there is nothing to be right about |
| F1 | 0 | Correctly reports that this model is worthless |
Every one of these numbers describes the same model. Accuracy is the only one that makes it look acceptable, which is exactly why it is the default and exactly why you should not trust it alone.
- Precision
Of the cases the model flagged as positive, what fraction really were positive?
High precision means few false alarms. This is what you want when acting on a positive is expensive or annoying.
- Recall
Of the cases that really were positive, what fraction did the model catch?
High recall means few misses. This is what you want when missing a positive is dangerous.
- F1 score
One number balancing precision and recall, computed as their harmonic mean.
The harmonic mean, rather than the ordinary average, so a model cannot score well by being excellent at one and hopeless at the other.
The math
All three metrics are ratios pulled out of the same four counts. Precision divides by what the model claimed. Recall divides by what was actually there. That one difference in the denominator is the whole distinction.
precision = TP / (TP + FP)
recall = TP / (TP + FN)
accuracy = (TP + TN) / (TP + TN + FP + FN)
F1 = 2 · (precision · recall) / (precision + recall)
- TP
- true positives: really spam, correctly flagged
- FP
- false positives: real mail, wrongly flagged
- FN
- false negatives: really spam, wrongly let through
- TN
- true negatives: real mail, correctly let through
Example
The figure's numbers, worked through
The filter above caught 170 spam, missed 30, wrongly binned 40 real emails and correctly passed 760.
Precision = 170 / (170 + 40) = 170 / 210 = 81%. Of everything it called spam, four in five really was.
Recall = 170 / (170 + 30) = 170 / 200 = 85%. Of all the spam that arrived, it caught roughly six in seven.
F1 = 2 × (0.81 × 0.85) / (0.81 + 0.85) = 83%.
Accuracy = (170 + 760) / 1,000 = 93%.
Now read that last number next to the others. 93% sounds excellent. It is also true that 30 spam landed in someone's inbox and 40 real emails were binned, and the accuracy figure mentions neither.
Precision and recall pull against each other, and the thing that moves them is the threshold.
A classifier does not really output "spam" or "not spam". It outputs a score, and something compares that score to a cut-off. Lower the cut-off and the filter flags more things: recall rises, precision falls. Raise it and the opposite happens.
So you are not usually choosing between models. You are choosing where on one model's trade-off curve you want to sit, and that choice belongs to whoever bears the cost of the errors.
Example
Same two metrics, opposite priorities
For a cancer screening program, recall matters most. Missing a real case can be fatal, while a false alarm costs a follow-up scan and a frightening fortnight. You push the threshold down and accept the false alarms.
For a spam filter, precision matters more. Wrongly quarantining a real, important email is worse than letting the odd spam through, because nobody checks the spam folder. You push the threshold up.
Neither choice is a property of the algorithm. Both come from asking which mistake you would rather explain to the person affected by it.
Never compare two models on two different metrics
It sounds obvious written down, and it happens constantly.
Model A gets 94% accuracy. Model B gets an F1 of 0.71. Which is better? The question has no answer, and quoting them side by side in a report invites everyone reading it to guess.
Decide the metric in stage one of the project, before you have any results to be tempted by. Write it down. Then every model gets scored the same way, on the same held-out data, and comparisons mean something.
Choosing the metric after seeing the scores is a way of choosing the winner, not of measuring one.
Regression needs a different toolkit entirely, because a numeric prediction is not right or wrong. It is a distance from the truth.
That changes the question from "how often" to "how far", and the interesting decision becomes how harshly to treat a single large miss.
| What it measures | Units | Character | |
|---|---|---|---|
| MAE | Mean absolute error | Same as the target | Treats every unit of error equally. Robust to outliers |
| RMSE | Root mean squared error | Same as the target | Squares before averaging, so one big miss dominates |
| R² | Fraction of variance explained | None, it is a ratio | 1 is perfect, 0 is no better than always guessing the mean |
MAE
- What it measures
- Mean absolute error
- Units
- Same as the target
- Character
- Treats every unit of error equally. Robust to outliers
RMSE
- What it measures
- Root mean squared error
- Units
- Same as the target
- Character
- Squares before averaging, so one big miss dominates
R²
- What it measures
- Fraction of variance explained
- Units
- None, it is a ratio
- Character
- 1 is perfect, 0 is no better than always guessing the mean
| Model | Errors on three predictions | MAE | RMSE |
|---|---|---|---|
| A, consistently a bit off | 20, 20, 20 | 20.0 | 20.0 |
| B, perfect twice then badly wrong | 0, 0, 60 | 20.0 | 34.6 |
MAE cannot tell these apart. RMSE says B is much worse, because squaring turns one 60 into far more than three 20s. If a single large error is genuinely worse for you than several small ones, that is the metric you want. If it is not, MAE is the honest choice.
from sklearn.metrics import ( confusion_matrix, classification_report, mean_absolute_error, root_mean_squared_error, r2_score,)# Classification: start from the matrix, not from a single number.print(confusion_matrix(y_true, y_pred))print(classification_report(y_true, y_pred, digits=3))# Regression: how far off, and how much of the variation is explained.print(mean_absolute_error(y_true, y_pred))print(root_mean_squared_error(y_true, y_pred))print(r2_score(y_true, y_pred))print(confusion_matrix(y_true, y_pred))The confusion matrix is the starting point. Every classification metric in this lesson is a ratio of its four cells, so printing it first means you can always work out where a surprising score came from.
print(classification_report(y_true, y_pred, digits=3))classification_report prints precision, recall and F1 for every class at once, which is the fastest way to spot a class the model is quietly ignoring.
print(root_mean_squared_error(y_true, y_pred))Compare this with MAE on the line above. When RMSE is much the larger of the two, a few big misses are doing the damage rather than general imprecision.
Reporting accuracy on imbalanced data.
Work out what the majority-class baseline scores first. If your model cannot clearly beat a model that always guesses the common class, accuracy is hiding that from you.
Chasing a higher F1 without asking which error costs more.
F1 assumes false positives and false negatives are equally bad. When they are not, and they usually are not, optimize the one that matters and report both.
Reading R² as a percentage of predictions that were correct.
It is the fraction of the target's variance the model explains. It can even go negative, which means you would have done better predicting the mean every time.
Comparing RMSE across different datasets.
RMSE carries the units of the target, so a house-price RMSE of 12 and a temperature RMSE of 12 have nothing to do with each other. Use R² when you need a comparison that travels.
Evaluating on the training set because the numbers are nicer.
Every metric here is only meaningful on data the model has never seen. A perfect confusion matrix on training rows is a description of memorization.
Key takeaway
- The right metric follows from what a mistake costs, so choose it before you have results.
- Accuracy hides failure on rare classes, and rare classes are usually the point.
- Precision counts false alarms, recall counts misses, and F1 balances the two.
- Precision and recall are traded against each other by moving the decision threshold.
- Regression asks how far rather than how often: MAE treats errors evenly, RMSE punishes big misses, R² measures variance explained.
- Never compare two models on two different metrics, and never score any of them on training data.
Finished this lesson?
Guest progress is saved on this device.