Data & Preprocessing
Understanding Your Dataset: Rows, Columns and Data Types
Before cleaning or modeling anything, you have to know what you are holding. This lesson covers the first look: shape, types, ranges, gaps and the questions each one answers.
beginner · 12 min read
Lesson content
Every project starts the same way, and almost nobody gives it enough time: find out what you are actually holding.
Not what the column names suggest. Not what the person who sent you the file said. What is in it.
The routine below takes about two minutes and routinely saves a week, because most of what goes wrong later is already visible in it.
Definition
Dataset
A table of examples: each row is one thing you are learning about, each column is something you know about it.
Formal definition
A collection of observations over a common set of variables, typically represented as a two-dimensional structure with n rows and p columns, one of which may be designated the target.
- Numeric (continuous)
A measurement where the arithmetic means something: price, area, temperature.
These can be scaled and averaged. The gap between 10 and 20 genuinely equals the gap between 20 and 30.
- Categorical (nominal)
A label with no order: city, color, payment method.
Encoding these as 1, 2, 3 tells the model London is less than Paris, which is nonsense it will happily learn.
- Ordinal
A label that does have an order, but no meaningful spacing: small, medium, large.
The order is real and the arithmetic is not, so these sit awkwardly between the other two and need a deliberate choice.
- Datetime
A moment in time, which almost never helps as a raw value.
Its usefulness comes from what you derive: hour, weekday, month, time since some event.
import pandas as pddf = pd.read_csv("housing.csv")df.shape # (1460, 12) how much is there?df.head() # what does one row look like?df.info() # types, and how much is missingdf.describe() # ranges, and anything impossibledf.isna().sum() # gaps per columndf["neighborhood"].value_counts() # categories, and rare onesdf.duplicated().sum() # rows that appear twicedf.info() # types, and how much is missinginfo() earns its place by showing dtype and non-null count together. A numeric column typed as object means something non-numeric is hiding in it.
df.describe() # ranges, and anything impossibleRead describe() for impossibilities rather than for averages. A minimum age of −3 or a maximum price of 0 tells you more than any mean.
df["neighborhood"].value_counts() # categories, and rare onesvalue_counts() shows the rare categories. A category with four rows in it will not survive a train/test split intact, which matters later.
| Command | The question | What a bad answer looks like |
|---|---|---|
df.shape | How much data is there? | Fewer rows than you have columns |
df.head() | What is one row? | Headers in the first row, or shifted columns |
df.info() | What type is each column? | A number column typed object |
df.describe() | What are the ranges? | A negative age, a zero price, a max of 999999 |
df.isna().sum() | What is missing? | A column that is 80% empty |
df.duplicated().sum() | Are rows repeated? | Anything above zero, until you know why |
None of these fixes anything. They tell you what needs fixing, which is a different job and has to come first.
Why spend time on this instead of getting to the model?
Because every hour here saves several later, and the savings are invisible so people skip it.
A column typed as text instead of a number will not error. It will quietly make every comparison a string comparison, and your accuracy will be mediocre for reasons no amount of model tuning fixes.
Duplicated rows will not error either. They will put the same example in both your training and test sets, and hand you a score that is too good.
The pattern is consistent: data problems rarely announce themselves. They show up as a model that underperforms, and by then you are debugging the wrong thing.
Example
What describe() catches in ten seconds
A housing dataset comes back with min of 0 for year_built and a max of 9999 for lot_size.
Neither is real. The zero is a missing value that somebody filled in with zero before you got the file, and the 9999 is a sentinel meaning "not recorded".
If you had not looked, both would have gone straight into the model as genuine numbers: a house built in year zero, and a lot the size of a county. The model would have taken them entirely seriously.
Sentinel values disguised as real numbers are one of the most common things describe() catches, and one of the most damaging things it catches.
The target column deserves its own look
Everything above applies to your features. Run it on the target as well, separately, because the target decides what kind of problem you have.
Is it a number or a category? If it is a category, how balanced are the classes? A 99-to-1 split changes which metric means anything, which is a whole lesson of its own.
If it is a number, is it skewed? A long right tail on price is extremely common, and it changes both which model fits and whether you should predict the log of it instead.
And check for leakage while you are there: a column that could only be known after the thing you are predicting has already happened will give you a wonderful score and a useless model.
| Looks like | Treat it as | Watch for | |
|---|---|---|---|
price | 120000, 245000 | Numeric | Skew, and zeros meaning missing |
neighborhood | "Bryant", "Elm" | Categorical | Rare categories, spelling variants |
condition | "poor", "fair", "good" | Ordinal | The order is real, the spacing is not |
sold_at | 2026-03-14 | Datetime | Useless raw, valuable once split up |
id | 1, 2, 3 | Neither | Drop it. It is numeric and means nothing |
price
- Looks like
- 120000, 245000
- Treat it as
- Numeric
- Watch for
- Skew, and zeros meaning missing
neighborhood
- Looks like
- "Bryant", "Elm"
- Treat it as
- Categorical
- Watch for
- Rare categories, spelling variants
condition
- Looks like
- "poor", "fair", "good"
- Treat it as
- Ordinal
- Watch for
- The order is real, the spacing is not
sold_at
- Looks like
- 2026-03-14
- Treat it as
- Datetime
- Watch for
- Useless raw, valuable once split up
id
- Looks like
- 1, 2, 3
- Treat it as
- Neither
- Watch for
- Drop it. It is numeric and means nothing
Reading the label before you cook
A cook checks what is in the box before deciding on the dish. Not because it is enjoyable, but because finding out halfway through is worse.
Is anything out of date? Is there less here than the label says? Is this the ingredient it claims to be?
The first look is that. Two minutes of reading the label, so that the next four hours are spent on the actual problem.
Trusting the column names.
A column called age may hold birth years, and a column called income may hold thousands or units depending on who exported it. Look at the values.
Reading describe() for the averages.
The mean is rarely the interesting part. Read the min and max, because that is where the impossible values live.
Treating an ID column as a feature.
It is numeric, so nothing will stop you. A model can find spurious patterns in row order, and it will.
Skipping duplicated().
Duplicates put the same example on both sides of a split and inflate your score. Check before you split, not after.
Looking at features and not at the target.
The target's type, balance and skew determine your metric and your model. It is the most important column in the file.
Key takeaway
- A dataset is rows of examples and columns of things you know, one of which is usually the target.
- Columns come in kinds, and numeric, categorical, ordinal and datetime each need different handling.
- Run shape, head, info, describe, isna and value_counts before anything else.
- Read
describe()for impossible values rather than for averages. - Data problems do not error. They show up later as a model that quietly underperforms.
- Give the target column its own look, including a check for anything that leaks the answer.
Finished this lesson?
Guest progress is saved on this device.