ML Projects
Choosing the Right Algorithm for Your Problem
With dozens of algorithms available, how do you choose? This lesson gives a practical decision process based on your task, data size and need for interpretability.
intermediate · 13 min read
Lesson content
After learning a dozen algorithms, the natural question is: which one do I actually use?
There is no single answer, and anyone who gives you one is selling something. There is, however, a reliable process, and it starts from a fact worth sitting with: on most problems the choice of algorithm is not what decides the outcome.
So rather than reaching for the most sophisticated model, you narrow the field by three things in order. What the task is, which rules out most of the list immediately. What your data looks like, which rules out most of the rest. What you need from the result, which usually decides between the two or three survivors.
Then you start simple anyway, because a baseline tells you what the sophisticated number even means.
A practical decision process
Identify the task
Predicting a number is regression. Predicting a category is classification. Finding groups with no labels is clustering. Predicting the next thing in a sequence is its own case again. This one question rules out most of the list before you have thought about anything else.
Establish a simple baseline
Fit something simple and fast first: ridge or logistic regression, or one shallow tree. It takes ten minutes, it sets the bar every fancier model has to clear, and often enough it is already good enough to ship.
Weigh your constraints
Do you have to explain an individual prediction to a customer or a regulator? How many rows do you actually have? How many milliseconds does a prediction get? Does the model have to retrain nightly on one machine? Each answer removes candidates, and they remove more of them than accuracy does.
Add complexity only if it pays
Move to a random forest or gradient boosting when the baseline falls short of what the problem needs, and only when the measured gain is worth the opacity you are buying. Write the baseline score and the new score next to each other before deciding; the gap is usually smaller than it felt.
Why start with a simple model instead of the best one?
Because a simple baseline is fast to build, easy to debug, and it tells you what "good" even means for your problem.
Often it is good enough and you are finished. When it is not, it becomes the yardstick: a complex model that barely beats it is not worth its extra cost and opacity, and now you can see that rather than guess it.
Starting simple also surfaces data problems that a flexible model would hide. Leakage, a broken feature, mislabeled rows: a linear model shows you these as a score that is impossibly good or inexplicably bad, while gradient boosting absorbs them and produces something plausible.
Complexity added deliberately, to close a gap you measured, beats complexity you started with by default.
| Situation | Sensible default |
|---|---|
| Tabular data, want a strong result | Gradient boosting (XGBoost/LightGBM) |
| Need to explain every prediction | Linear/logistic regression or a single tree |
| Small dataset, quick baseline | Logistic regression or KNN |
| Images | A CNN, usually via transfer learning |
| Text | Naive Bayes baseline, then transformers |
| No labels, want groups | K-means or DBSCAN |
Tabular data, want a strong result
- Sensible default
- Gradient boosting (XGBoost/LightGBM)
Need to explain every prediction
- Sensible default
- Linear/logistic regression or a single tree
Small dataset, quick baseline
- Sensible default
- Logistic regression or KNN
Images
- Sensible default
- A CNN, usually via transfer learning
Text
- Sensible default
- Naive Bayes baseline, then transformers
No labels, want groups
- Sensible default
- K-means or DBSCAN
The best algorithm cannot rescue bad data
Before agonizing over model choice, check that the data is clean, the features are sensible and nothing is leaking.
A great model on flawed data loses to a plain model on good data, every time and by a wide margin. The reason is simple: the algorithm can only work with the information in front of it, and no amount of capacity invents a signal that is not there.
The corollary is that algorithm choice is rarely where the biggest gains hide. It is just the most visible decision, and the one with the most blog posts written about it.
| Data | Target | Start with | Then try |
|---|---|---|---|
| Tabular, under 1,000 rows | Number | Ridge regression | Random forest |
| Tabular, thousands of rows | Category | Logistic regression as a baseline | Gradient boosting |
| Tabular, must be explainable | Either | One shallow decision tree | A linear model with few features |
| Images | Category | A pretrained CNN with a new head | Fine-tune more layers |
| Text | Category | TF-IDF with logistic regression | A pretrained transformer |
| No labels at all | None | K-means, and look at the result | DBSCAN, PCA |
Every row starts with something simple and fast. That is deliberate: the baseline tells you whether there is signal at all, and it takes ten minutes. Without it a mediocre score from a complicated model is uninterpretable.
Example
The ten minutes that reframes a project
Before choosing anything clever, fit the dumbest possible model and write the number down.
For classification, predict the majority class. For regression, predict the mean.
Suppose the baseline scores 0.82 and your gradient boosting model scores 0.84. That is a very different situation from the one you thought you were in, and it means either the features carry little signal or the problem is nearly solved already.
Now suppose the baseline is 0.51 and your model is 0.84. That is real, and it tells you the work is paying off.
The same 0.84 means opposite things depending on a number that took ten minutes to get. Get it first.
The algorithm is rarely the thing holding you back
It is the most visible decision, which is why it attracts the most attention, and it is usually not the binding constraint.
On most tabular problems the gap between a well-tuned random forest and a well-tuned gradient booster is a point or two. The gap between good features and poor features is often ten.
So when a model underperforms, work down this list before changing the algorithm: is there a data leak inflating or deflating the score, is the metric right for the problem, are there features the model cannot infer but you could compute, is there simply not enough data?
Swapping algorithms is the easiest thing to try and it is rarely the thing that was wrong.
Starting with the most powerful model available.
You have nothing to compare it against. Start with a baseline so the sophisticated number means something.
Choosing on benchmark results rather than your data.
A benchmark leaderboard reflects one dataset. Your row count, feature types and noise level decide what works for you.
Ignoring the constraints that are not accuracy.
Prediction latency, model size, explainability and retraining cost rule out models that would have scored best.
Tuning hyperparameters before checking the features.
Tuning buys a point or two. A missing feature can be worth ten.
Using deep learning on a small table.
Gradient boosting beats neural networks on most tabular problems, and it trains in seconds rather than hours.
Key takeaway
- First fix the task type, regression, classification or clustering, which rules out most algorithms.
- Start with a simple baseline. It is often enough, and it always sets the bar.
- Let interpretability, data size and speed prune the remaining choices.
- Add complexity only to close a measured gap and remember data quality beats algorithm choice.
Finished this lesson?
Guest progress is saved on this device.