ML Fundamentals
How Machines Learn From Data: Features, Labels and Models
Every supervised model is built from three pieces: features (the inputs), labels (the answers) and a model that maps one to the other. This lesson defines the vocabulary you will use everywhere else.
beginner · 11 min read
Lesson content
Before any algorithm comes a shared vocabulary that every machine learning discussion assumes you already have. Almost all of it reduces to one picture: a table.
Rows are the things you are learning about. Columns are what you know and what you want to predict. Get comfortable with four words now and every later lesson on this site reads more easily.
- Feature
One measurable input. A column in your table, such as a house's floor area or a patient's age.
Features are everything the model gets to look at. Choosing good ones often matters more than choosing a clever algorithm.
- Label
The answer you want to predict. The target column, such as the house's price or whether an email is spam.
Having labels is exactly what makes a problem supervised. No labels means a different family of methods entirely.
- Example
One complete row: a set of feature values, plus its label if you have one.
Models learn from many examples. More good examples usually beats a better algorithm on fewer.
- Model
The mapping from features to a prediction that the algorithm builds out of the examples.
This is the thing you keep. Training is the process; the model is the product.
The table below has three features and one label. Each row is one house somebody actually sold, so each row is one example.
The model never sees the price column when it makes a prediction. It sees the three features, produces a number, and training is the business of making that number land close to the price that was really paid.
| Size (m²) | Bedrooms | Age (yrs) | Price ($1000s), label |
|---|---|---|---|
| 70 | 2 | 10 | 200 |
| 96 | 3 | 4 | 294 |
| 52 | 1 | 20 | 120 |
The first three columns are features the model reads. The last column is the label it learns to predict. A fourth house (120 m², 4 bedrooms, 2 years old) is deliberately kept out of this table, because a model that has never seen it is the only fair way to check the model works.
What does training a model actually change?
A model holds internal numbers called parameters. For a straight-line model those are the slope and the intercept. For the house model below they are the three numbers multiplying size, bedrooms and age.
Training adjusts those parameters so the model's predictions on the training examples land as close as possible to the real labels. Nothing mystical happens. The algorithm measures how wrong it currently is, nudges the parameters to be a little less wrong, and repeats.
When training stops you keep whatever values the parameters settled on. That fitted state is the model.
The math
A model is a formula with blanks in it. Training fills in the blanks. Once they are filled you can hand the formula any row of features and read off a prediction.
price = w₁ · size + w₂ · bedrooms + w₃ · age
fitted on the three rows above:
w₁ = 2.5 per m²
w₂ = 20 per bedroom
w₃ = −1.5 per year of age
- w₁, w₂, w₃
- the parameters, one weight per feature, found by training
- size, bedrooms, age
- the features, read from a row of the table
- price
- the label, which is what the formula predicts
Example
Running it by hand
Check it against the first row of the table: 2.5 × 70 + 20 × 2 − 1.5 × 10 = 175 + 40 − 15 = 200. That is exactly the price recorded for that house.
Now try the house that was deliberately held out, 120 m², 4 bedrooms, 2 years old:
2.5 × 120 + 20 × 4 − 1.5 × 2 = 300 + 80 − 3 = 377.
The model has never seen that house. It still produces a number, because the three parameters carry everything it learned from the rows it did see. That is the entire point of keeping the model rather than the data.
Notice what the signs are telling you. Floor area and bedrooms push the price up, age pushes it down.
Nobody wrote those signs into the program. They fell out of the arithmetic of fitting three rows. That is the whole trick of supervised learning in one line.
Definition
Parameters and hyperparameters
Parameters are the numbers the model learns from the data during training. Hyperparameters are the settings you choose before training starts, such as how complex the model is allowed to be.
Formal definition
Parameters are values fit by the learning algorithm to minimize error on the training data. Hyperparameters are configuration values, fixed before training, that govern the model's capacity or the learning process itself.
| Who sets it | When | Example | |
|---|---|---|---|
| Parameter | The learning algorithm | During training | The 2.5 multiplying floor area |
| Parameter | The learning algorithm | During training | A weight inside a neural network |
| Hyperparameter | You | Before training | K, the number of neighbors in KNN |
| Hyperparameter | You | Before training | How deep a decision tree may grow |
Parameter
- Who sets it
- The learning algorithm
- When
- During training
- Example
- The 2.5 multiplying floor area
Parameter
- Who sets it
- The learning algorithm
- When
- During training
- Example
- A weight inside a neural network
Hyperparameter
- Who sets it
- You
- When
- Before training
- Example
- K, the number of neighbors in KNN
Hyperparameter
- Who sets it
- You
- When
- Before training
- Example
- How deep a decision tree may grow
Example
A test that always works
If the data determines it, it is a parameter. If you determine it before seeing any results, it is a hyperparameter.
In a K-Nearest-Neighbors model you pick K yourself, so K is a hyperparameter. In linear regression the algorithm finds the slope and intercept, so those are parameters.
The test holds even in deep learning, where a network has millions of parameters and perhaps a dozen hyperparameters. The millions are learned. The dozen are your decisions.
A recipe and the cook
Think of a hyperparameter as the oven temperature you set before you start, and a parameter as how long the cake actually needs at that temperature.
You choose the temperature from experience. The right baking time is discovered by watching the cake.
Change the temperature and every baking time you learned before becomes wrong. That is exactly why changing a hyperparameter means training all over again.
The model can only see the columns you give it
The house model above knows nothing about the neighborhood, the school catchment, whether the roof leaks or how the market moved last year. None of those are columns in the table, so none of them exist as far as the model is concerned.
When a model underperforms, the instinct is to reach for a fancier algorithm. Far more often the missing piece is a feature nobody collected.
This is why the Data and Preprocessing path on this site is longer than the one on algorithms.
Treating the label as just another column and leaving it in the features.
Feeding the label back in as an input gives you a model that scores perfectly and predicts nothing. Split features from label before you train, every time.
Calling every number inside a model a parameter.
Parameters are learned from the data. Anything you chose yourself is a hyperparameter, even if it lives in the same object.
Judging the model by how well it reproduces its own training rows.
Of course it fits them. It was built to. The held-out house is the only honest check, which is what the next lessons are about.
Assuming more features always helps.
Irrelevant columns give the model more ways to find coincidences in your training data. Good features beat many features.
Key takeaway
- Data is a table: rows are examples, columns are features and the target column is the label.
- A model is the mapping from features to a prediction that training builds.
- Training adjusts the model's parameters until its predictions match the labels as closely as it can.
- Parameters are learned from data. Hyperparameters are set by you before training.
- A fitted model is just its parameters, which is why you can save one to a file and load it somewhere else.
- The model sees only the columns you gave it, so a missing feature is invisible rather than merely unhelpful.
Finished this lesson?
Guest progress is saved on this device.