Intuition
Feel the idea before the formula.
The interactive machine-learning reference
Every algorithm. The same six questions.
Most explainers show you one beautiful animation and then stop. Here, every algorithm answers the same six questions, and you answer them back by dragging, splitting and breaking things yourself. Each chapter runs two tracks: Explanation for the idea, the maths and the interactives, and Code for the from-scratch implementation, walked line by line.
regulars · many visits, modest spend
big spenders · rare visits, high spend
occasional · low on both
iteration 0 · J = …
The same six questions, every time
Learning ten algorithms is hard. Learning one shape ten times over is not. Every chapter on Lysen answers these six in this order, so by the third chapter you already know where to look.
The algorithm
Every step of the loop, with numbers.
Evaluation
How you know it worked.
Failure modes
Break it, then fix it.
Prompting AI
Make an assistant build it well.
Beyond AI
What no prompt can decide.
Not sure where to start?
Answer a couple of questions and the site will point at the right chapter. No sign-up, nothing stored.
The map
Every algorithm sits in two boxes: how it learns, and what job you point it at. A filled chip is what its chapter teaches; an outlined chip is what else it can do. Where several algorithms are scored the same way, that scoring gets its own shared page rather than being repeated in each chapter.
Supervised learning
learns from examples where somebody already wrote down the answerRegression · predict a number
Classification · predict a label
Live
Distance based · no training
K-nearest neighbours
You are whatever your closest neighbours are. No training step at all.
ClassificationRegression
Live
Linear · gradient descent
Logistic regression
A line that reports a probability instead of a value, despite the name.
Classification
Live
Tree based
Decision trees
Ask the one question that separates the groups best. Then ask again.
ClassificationRegression
Live
Shared within the group
Evaluation & comparison
Accuracy and why it lies, the confusion matrix, precision and recall, and the three classifiers side by side.
Classification
Shared across all of supervised learning
Live
Bias and variance
Every chapter ends with a dial: k, penalty strength, tree depth. They are the same dial, and this is what it does. Too simple against too eager, how to read the gap between training and held-out error, and how to tell whether more data would help before you go and collect it.
Unsupervised learning
no answers are given, so the structure has to be found in the data itselfClustering · find the groups
← All algorithms
Unsupervised · clustering
K-means
The problem it solves: you have data, and nothing is labelled. Nobody has told you which rows belong together, or even how many kinds of row exist. Clustering is the family that hunts for that hidden structure on its own, and K-means is its workhorse, simple enough to run by hand and fast enough for millions of rows.
The idea in one sentence: guess where the group centers are, then let the data correct your guess. Every point joins whichever center is nearest, then every center slides to the middle of the points that joined it, and you repeat until nothing moves. All the depth is in what surrounds it: how you start it, how you check it, and knowing when it is fooling you.
Used for
customer segmentationimage colour compressiongrouping documentsflagging unusual recordsexploring data before modelling
Learning typeUnsupervised
Primary taskClustering
MechanismDistance based
Running example for the whole chapter: the customers of a café. Each point is one customer, where x = visits per month and y = average spend in euros. The owner suspects there are different types of customer but has never labelled anybody. Can the data tell us?
01
Intuition: what it does, and how
What it does. Imagine you own a café and you have a dot on a chart for every customer. Nobody has told you what kind of customer each one is. K-means takes that cloud of dots and splits it into k groups, where you pick the number k yourself. Each group gets one point that represents it, called a center. You can think of it as the imaginary average customer of that group.
How it does it. The whole method is two moves, repeated. Assign: every dot joins whichever center is nearest right now. Update: every center slides to the average position of the dots that just joined it. Each move makes the other slightly out of date, so you simply do them again, a bit like a shaken box of marbles settling down.
1a The problem, before any algorithm
This is exactly what the café owner has. Dots, no names, no colours. Look at it for a moment and your eye has probably already found the groups by itself. Clustering is the attempt to give a computer that same instinct, using nothing but the distances between points.
One honest warning before we start. Your eye is also the reason clustering is risky, because people see groups in completely random data too. Module 3 is about telling the difference.
1b Be the algorithm: run the two moves by hand
cluster 1
cluster 2
cluster 3
unassigned
Click the plot to drop 3 centers (0/3 placed), or press Place 3 at random. Bad placements are welcome here, because watching them recover is the point.
Two experiments. First, drop all three centers in one corner and press the buttons. Watch the loop untangle itself; that self-correction is the whole intuition. Second, drop two centers inside the same visible group and see whether they ever escape. Sometimes they do not. Remember that, because it comes back as failure mode 4a.
02
The algorithm: every step of the loop
What you just did by hand is written out below as a recipe. Programmers call it pseudocode: the steps in order, in words, with nothing in it that belongs to any one language.
1initialize k centers μ₁ … μₖ // place the starting centers
2repeat:
3 assign: cᵢ ← argminⱼ ‖xᵢ − μⱼ‖ // each point joins its nearest center
4 update: μⱼ ← mean of points with cᵢ = j // each center moves to its average
5until assignments stop changing // nothing moved, so stop
2repeat:
3 assign: cᵢ ← argminⱼ ‖xᵢ − μⱼ‖ // each point joins its nearest center
4 update: μⱼ ← mean of points with cᵢ = j // each center moves to its average
5until assignments stop changing // nothing moved, so stop
Those symbols look worse than they are. μ (say "mew") is the name for a center, μⱼ is center number j, xᵢ is data point number i, the double bars ‖ ‖ mean "the distance between", and argmin means "whichever one gives the smallest answer". So line 3 reads: each point looks at the distance to every center and joins the closest one.
2a The score it is quietly shrinking
J = Σⱼ Σx∈Cⱼ ‖x − μⱼ‖² // the score the algorithm shrinks
…J at this step
…points that changed group
Read J out loud as: for every group, take each point, measure how far it is from its own center, square it, and add everything up. One number for the whole clustering, and smaller is better; people call it WCSS. Assigning sends every point to its nearest center, so it cannot make J worse. Moving each center to the average is the best possible spot, so that cannot either. A number that never rises and cannot fall forever has to stop falling, which is why the loop always halts. Notice the word that is missing from that promise: best.
CuriousWhy the mean, and why is that the same question as the squaring?
Line 4 says each center moves to the mean of its points. That looks like an obvious choice and it is not one: it is forced by the squares in J, and the two travel together.
Hold the assignments still and ask where one center should sit. You want the μ that makes Σ‖x − μ‖² as small as possible, so differentiate that sum with respect to μ and set it to zero:
d/dμ Σ (x − μ)² = −2 Σ (x − μ) = 0
// so Σx − nμ = 0, so μ = (Σx) / n
// so Σx − nμ = 0, so μ = (Σx) / n
The mean is not a sensible summary somebody chose. It is the exact solution, and it is the answer only because the score squares. Swap the squares for plain distances and the same argument lands on the median instead, which is an algorithm with its own name, k-medians.
That is worth holding onto, because it explains the failure you can drag around in before you get there. Squaring makes a far-away customer shout, the mean has to listen, and the center goes to meet them. A median would barely move. The outlier problem is not a bug in the update step; it is the price of the score, and the same trade shows up in linear regression for exactly the same reason.
CuriousIf J always falls, how can the silhouette have a peak?
Jumping ahead to module 3 for a moment, because the two scores there behave so differently that it looks like one of them must be broken. J falls every time you raise k. The silhouette does not. Both are measured on the same clustering.
J only ever asks one question: how far is each point from its own center? Give the algorithm another center and every point can only end up nearer to one, or stay where it is. So J falls, always, all the way down to zero when there are as many centers as customers. It is not measuring whether the groups are any good. It is measuring how tightly they are packed, and more groups are always tighter.
The silhouette asks a second question alongside the first: how far is each point from the group next door? That is what changes everything. Split a real group in half and you have made both halves tighter, which J likes, but you have also put every point in one half right next to a different group, which the silhouette hates. The two effects pull against each other, so the score has somewhere to peak.
The general lesson is worth more than the two formulas. A score with one term usually has a trivial optimum you can reach by pushing that term to its limit, and a score that actually chooses between models almost always has two terms in tension. You will meet that shape again as the bias and variance trade, where it is the whole page.
03
Evaluation: scoring a model when nobody knows the answer
Most models can be marked like a test, because someone wrote down the correct answers. Clustering has no answer key, so instead we score the shape of the result, using numbers worked out purely from the geometry. Together they answer the question the algorithm never asks you: what should k be?
Every point here is a complete run of K-means, best of eight attempts. J always falls as k grows. With as many centers as customers it would hit zero and tell you nothing. So you do not look for the lowest point, you look for the bend. Click any k to inspect it.
3chosen k
…J (WCSS)
…silhouette
s(x) = ( b(x) − a(x) ) / max(a(x), b(x))
// a = mean distance to my own group
// b = mean distance to the nearest other group
// a = mean distance to my own group
// b = mean distance to the nearest other group
The silhouette asks each point: am I closer to my own group, or to the group next door? It lands between +1 and −1, and unlike J it does not always improve as k grows, so it has a genuine peak. Here both methods point at k = 3. When they disagree on real data, that is useful information: it usually means your groups are not neat round blobs.
04
Failure modes: break it, then fix it
Two things go wrong often enough to have names. Each one is a lever you can pull on the interactive below, and each has a defence you can write into a prompt in module 5.
4a One extreme customer moves the whole center
Drag the ringed customer anywhere on the plot. Its center follows, because a mean has no defence against an extreme value.
0.0center moved by
0points that changed group
Why it happens. The update step takes an ordinary average, and an average is dragged by whatever is furthest out. One customer who spent €400 can bend a whole segment around themselves. What to do: inspect extreme values before clustering rather than after, and if they are real and you cannot drop them, use K-medoids, whose centers must be actual customers and therefore cannot drift into empty space.
4b An unlucky start it never escapes
…
Same data, same k, same code. The only difference is where the centers happened to start. K-means is guaranteed to stop, but not guaranteed to stop anywhere good, so a single run is never evidence. What to do: start with k-means++, which spreads the first centers out on purpose, and run it ten times keeping the lowest J. Both are one argument each in every library, and both are missing from the lazy prompt in module 5.
05
Prompting AI: making an assistant build it properly
An assistant will happily write K-means from the laziest possible request. The difference between a weak prompt and a strong one is not the quality of the code, it is whether the traps from modules 3 and 4 get handled or quietly skipped.
"Cluster my customer data with k-means in Python."
What you typically get back
Working code that picks k = 3 for no stated reason, uses the raw columns without rescaling, starts once, and finishes with a confident summary. It runs. It also decides how many customer groups your business has on your behalf, ignores that euros and visit counts live on different scales, and gives you no way to check whether the groups are real.
k chosen for youno feature scalingno elbow or silhouetteone start, no repeatsno outlier checkno interpretation
"I have café customer data: visits_per_month (0–30) and avg_spend_eur (0–100), about 2,000 rows, unlabelled. Write Python that:
1. checks for outliers and reports them before doing anything else,
2. rescales both columns to a common range and explains why that matters here,
3. runs K-means for k = 1…10 using k-means++ with 10 restarts each,
4. plots the elbow curve and silhouette scores, then recommends a k and says why,
5. re-runs the chosen k with different seeds and reports how stable the groups are,
6. describes each group in plain language (mean per column, and size),
7. lists which K-means assumptions my data might break, and what to try instead."
What changes
Same library, same algorithm, but now every failure mode from module 4 has a line defending against it. Outliers get inspected (4a), unlucky starts get repeats (4b), the choice of k has evidence behind it (module 3), and the model is asked to admit when it is the wrong tool. You have written the chapter into the prompt, and that is the actual skill.
outliers inspected firstscaling handledk justified with evidencerepeats and stabilitygroups describedassumptions surfaced
06
Beyond AI: choosing the tool, owning the call
Two things survive even a perfect prompt. The first is the choice of algorithm. An assistant reaches for K-means because everybody does, but whether that is right depends on features of your data you have to notice yourself. The second is the set of judgment calls, the questions with no score attached at all.
| If your problem looks like this | Reach for | Because | The price you pay |
|---|---|---|---|
| Compact, roundish groups of similar size, numeric columns, a large dataset, and speed matters | K-meansthis chapter | Fast and simple, handles millions of rows, and centers describe as an average member. | You must pick k. Sensitive to outliers, starting positions and scale. Assumes round groups. |
| Outliers you cannot remove, or you want each center to be a real example | K-medoidsalso called PAM | Centers are actual data points, so extreme values cannot drag them, and any distance measure works. | Much slower; painful beyond roughly 10,000–50,000 rows. You still pick k. |
| You do not know how many groups exist, shapes are irregular, and some points belong to nothing | DBSCANnot covered here | A group is a crowded region, so it finds any shape, labels leftovers as noise, and never asks for k. | Two fiddly settings, and it struggles when some groups are much denser than others. |
| Groups that overlap, or you want "70% group A, 30% group B" instead of one hard label | Gaussian mixtureGMM | Percentage membership rather than a single label, and stretched oval groups at any angle. | More settings to fit, still sensitive to starting positions, still assumes blob shapes. |
6b The judgment calls no prompt makes for you
Do the groups mean anything?
K-means will split pure noise into k groups, draw a convincing chart and post a decent silhouette. Whether "group 2" is a real type of customer worth a marketing budget is a business question. No score answers it.
Which columns, and on what scale?
Cluster on visits and spend and you get loyalty groups. Add age and postcode and you get different people entirely. Choosing what "similar" means is the biggest decision, and it happens before any code runs.
Outliers: mistakes or discoveries?
Module 4a treated an outlier as a threat to the average. But the customer spending €400 might be the most valuable thing in your data. Typing error or real signal is a question about where the data came from.
Acting on the groups is on you
Charging "group 3" more, or leaving them out of an offer, can quietly repeat a bias hidden in the columns you chose. The algorithm optimised a number. The consequences were never in the formula.
Your K_means class, one idea per cell, with every snippet explained on its own instead of buried in one block. Each cell points back to the Explanation panel where you can drag the same step.
1
Set up the class and its knobs
hyperparametersimport numpy as np class K_means: def __init__(self, k=3, tolerance=1e-4, epochs=300, seed=0): self.k = k self.tolerance = tolerance self.epochs = epochs self.rng = np.random.default_rng(seed)
Three knobs decide everything. k is how many groups to look for, and you choose it, because the algorithm cannot work it out. That is what module 3 is about. tolerance and epochs decide when to stop: quit once the centers barely move, or bail after epochs rounds regardless.
2
Start: spread the first centers out
module 4b · initializedef _init_centers(self, X): # k-means++: first center random, then prefer points far from # the centers already chosen. This is failure mode 4b's defence. centers = [X[self.rng.integers(len(X))]] while len(centers) < self.k: d2 = np.min(((X[:, None] - np.array(centers))**2).sum(-1), axis=1) centers.append(X[self.rng.choice(len(X), p=d2 / d2.sum())]) return np.array(centers)
Random starts are cheap but two centers can land in the same group and waste one. Sampling proportional to squared distance spreads them out instead. Drag the segmented control in 4b to see what an unlucky start costs.
3
Assign: every point joins its nearest center
module 2 · line 3def _assign(self, X, centers): # squared distances, points × centers. No square root needed, # because argmin of d² is the same as argmin of d. d2 = ((X[:, None, :] - centers[None, :, :])**2).sum(-1) return d2.argmin(axis=1), d2.min(axis=1).sum()
The second return value is J, the score from module 2, and you get it free, because you already computed every distance. Returning it here is what makes the loss curve in that panel possible.
4
Update, and know when to stop
module 2 · lines 4–5def fit(self, X): X = np.asarray(X, dtype=float) centers = self._init_centers(X) self.history = [] for _ in range(self.epochs): labels, J = self._assign(X, centers) self.history.append(J) new = np.array([X[labels == j].mean(0) if (labels == j).any() else centers[j] for j in range(self.k)]) if np.abs(new - centers).max() < self.tolerance: centers = new; break centers = new self.centers_, self.labels_ = centers, labels return self
One subtlety worth the line it costs: if (labels == j).any() guards against an empty cluster. If no point picks a center, its mean is undefined and NumPy hands you nan, which silently poisons every later round. Keeping the old position is the cheapest sane fix.
← All algorithms
Supervised · regression
Linear regression
The problem it solves: a linear regression model is a straight line which tries to fit all of the data points as tight as possible. You have a number you want to predict, for example a price, a demand or a temperature, and one or more numbers you already know. Any method that predicts a number rather than a category belongs to a family called regression, and this is where that family starts.
The idea in one sentence: draw a line, see how close it is to all of the data points, then repeat, trying to make the distance to the data points lower. That is the entire algorithm. Everything else in this chapter is about what "how close" means, which direction to move next, and the four ways a straight line will quietly lie to you.
Used for
house and rent pricingdemand forecastingtrend linesrisk scoringthe baseline every model must beat
Learning typeSupervised
Primary taskRegression
MechanismLinear · gradient descent
Running example for the whole chapter: houses for sale. Every point is one house, where x = size in m² and y = the asking price in euros. One column keeps every idea visible on a flat chart, and this is the simple version, so no ridge, no lasso. Those are in module 4, once you have seen the problem they solve.
01
Intuition: draw the perfect line
What it does. Suppose we have data about the prices of houses, so we have the amount of square meters against the price of the house. Every house is one dot on a chart. The model is the line you draw through those dots, and once you have that line you can read a price off it for any size you like, including sizes nobody in your data has. That is both the useful part and, later, the dangerous part.
How it does it. Draw a line, see how close it is to all of the data points, then repeat, trying to make the distance to the data points lower. A line only has two numbers in it: the slope w, which is how many euros one extra square meter adds, and the initial value β, which is where the line starts. Fitting means finding the pair of numbers that leaves the smallest total distance to the dots.
Why the line is worth having. Two reasons, and they are different. The first is prediction: a new house comes on the market at 96 m² and you want a number. The second is explanation: the slope is a sentence you can say out loud in a meeting, something like "in this city, a square meter is worth about €2,670", and most models cannot give you that sentence at all.
1a The problem, before any algorithm
This is what you actually get: 52 houses, a size and a price for each, and nothing else. Your eye has already drawn the line, roughly, before you finished reading this sentence. That is exactly the point. The relationship is obvious to you, and the whole chapter is about writing down what your eye just did, so a computer can do the same thing on twelve columns you cannot picture.
Bigger houses cost more. Nobody needed maths for that. What you cannot do by eye is say how much more, per square meter, or how wrong that answer is.
1b Be the algorithm: fit the line by hand
your line
the miss on each house
best possible line
price = 240,000 + 1,500 × size
…mean absolute error
…RMSE
…mean squared error (€²)
…worse than best by
Tune the two sliders to fit the line as close to the data points as possible, and watch the three errors underneath. They all say the same thing in different units, and they all get smaller as the line gets better. The button does it in one shot with the maths from §2c, but try to beat it by hand first, it is harder than it looks, and that difficulty is the reason the rest of the chapter exists.
02
The algorithm: every step of the loop
What you just did with two sliders is written out below as a recipe. Programmers call this kind of numbered recipe pseudocode, which just means code written for humans instead of for a computer.
1initialize the slope w and the initial value β // any two numbers will do (§2a)
2repeat:
3 see how well the line fits // one number for the whole line (§2b)
4 update the initial value and the weights // step downhill (§2c)
5until a certain difference threshold is met // it stopped improving (§2d)
2repeat:
3 see how well the line fits // one number for the whole line (§2b)
4 update the initial value and the weights // step downhill (§2c)
5until a certain difference threshold is met // it stopped improving (§2d)
Three of those five lines need real numbers before a computer can run them: how you start, what "how well it fits" means, and how you update. Each gets its own panel below, and each panel repeats the line it is about so you never have to scroll back up here.
2a Line 1: how to initialize
1 initialize the slope w and the initial value β ← the line this panel is about
Initialization is quite simple: you can start with any initial weight values and any initial value β. As you saw in module 1, the loop gets closer to the perfect line every time it runs. Of course, a starting line that is further away takes longer to get to the perfect line than a closer one, so if you can estimate something sensible, start there.
In practice that means: zeros are a perfectly good start, a rough guess is a slightly better one, and the only genuinely bad start is one so extreme that the first steps overshoot. Pick a starting line below and watch it walk. Same data, same learning rate, same target, only the first line differs.
…
…step
…RMSE now
…steps to settle
Notice what does not change: the line it ends on. All three land in the same place, because for a straight line there is only one valley to fall into, and that stops being true the moment a model gets more interesting than this one. What changes is only how long you wait.
2b Line 3: seeing how well the line fits
3 see how well the line fits // turn a whole line into one number
For one house the error is easy. Take the real price, subtract the price the line predicts, and you have how far off you were. Do that for all 52 houses and you have 52 numbers. What you want is one number, so that you can hold two lines next to each other and say which one is better.
The most common one is the root mean squared error (RMSE). Square every error, take the mean of those squares, then take the square root of the result. The squaring makes everything positive and punishes a big miss much harder than a small one. The square root at the end matters more than it looks: without it you are holding a number measured in euros squared, and nobody has any feeling for what a euro squared is. With it, you are back in euros.
RMSE = √( (1/n) Σ (yᵢ − ŷᵢ)² ) // ŷ = β + w·x, the price the line predicts
MAE = (1/n) Σ |yᵢ − ŷᵢ| // the same misses, without the squaring
MAE = (1/n) Σ |yᵢ − ŷᵢ| // the same misses, without the squaring
The other one is the mean absolute error (MAE). Take the size of every error, ignore the sign, and average them. It answers a question you can say out loud: on an average house, how many euros is this line wrong by? Both are worked out over all houses, and for both, smaller is better.
They sound like the same thing said twice, and on this data they are not. Below is what each one does with a miss, and which houses end up owning the score.
…× worse to MAE
…× worse to squared error
Both curves are drawn against the same misses, and scaled so the biggest one on the chart counts as 100. MAE is a straight line: twice the miss, twice the punishment. Squared error is a parabola: twice the miss, four times the punishment. Drag the slider and compare the two multipliers, which are measured against a €20k miss.
share of the MAE
share of the squared error
…
…MAE of the best line
…RMSE of the same line
…RMSE ÷ MAE
Same line, same 52 houses, two different numbers, and RMSE is always the bigger of the two. The gap between them is a measurement in its own right: it is small when your misses are all about the same size, and it grows the moment a few houses are much further out than the rest. If RMSE is far above MAE, go and look at your extremes before you do anything else.
CuriousWhy not just use the differences?
A question which might have come up in your mind is: why not just use the differences, or the average of the differences? That is because sometimes the line is above the data points, which makes the difference negative, and when the point is above the line it is positive. So they cancel each other out.
Which means a line that is badly wrong, but wrong evenly above and below, scores exactly as well as the perfect line. Try it: both lines below have an average difference of about zero, and only one of them is any good.
…mean of the raw differences
…mean absolute error
line is under the house (+)
line is over the house (−)
The first number cannot tell the two lines apart. The second one can, and it is not close. That is why, if you just want the difference, you take the absolute error, or the mean absolute error (MAE).
CuriousWhy then squared?
So the absolute error already fixes the cancelling problem, and it is easier to explain. Why does everybody square instead? Because of outliers.
An outlier is a data point that sits far away from the pattern the rest of the points make. In this data it would be a house that is cheap for its size because it is falling apart, or one that costs double because it comes with half a canal. Squaring means a miss of €200,000 counts one hundred times as much as a miss of €20,000, not ten times as much. So the squared error is desperate to reduce that one house, and it will tilt the whole line to do it. The absolute error mostly shrugs at it.
Drag the ringed house around and watch the two lines come apart. This panel uses half the market, 26 houses, because the fewer rows you have the louder one strange row shouts, so small data and outliers are a dangerous pair. Neither line is cheating; they are answering different questions.
fitted on squared error
fitted on absolute error
Drag the ringed house. It starts where it belongs, so both lines start close together.
…gap between the two lines at 190 m²
…squared-error line moved by
So which do you use? If the strange house is a typing error, you want the line that ignores it, and you should also go and fix the data. If it is real and you will be asked to price houses like it, a line that pretends it does not exist is worse than useless. That decision is not in the formula, and it comes back in module 6.
2c Line 4: update, or how to get closer to the perfect line
4 update the initial value and the weights // the only line that actually changes anything
You now have one number saying how wrong the line is. The last piece is what to do about it, and here is the picture that makes it obvious.
Think of it as a landscape. Every pair of numbers (w, β) is a spot on a map, and the error of that pair is the height of the ground there. Bad lines are high up on a hill. Good lines are down in a valley. The perfect line is the lowest point in the valley. Fitting is walking downhill in that landscape, except you are blindfolded, so all you can do is feel which way the ground slopes under your feet and take a step that way. Then feel again, and step again.
That is gradient descent. The slope under your feet is the gradient, and how big a step you take is the learning rate η. Two things go wrong, and both are visible below. Steps that are too small mean you are still walking at sunset. Steps that are too big and you jump clean over the valley floor and land higher up the other side, and the next jump is worse, and the line flies off the chart.
The stronger the colour, the lower the ground, so the teal streak is the valley floor, and the cross marks its lowest point. Notice the valley is a long diagonal ditch rather than a round bowl: slope and initial value trade off against each other, so many combinations are nearly as good as each other, and the walk down the middle is slow.
0steps taken
…RMSE here
Press Walk downhill. Each dot is one pass over all 52 houses.
Push η past about 1.0 and run it again. The path stops walking and starts bouncing between the walls of the ditch, and the error goes up instead of down. That single slider is the most common reason a hand-written model "does not work", and it is why every library ships with a cautious default.
CuriousWhere do the update rules actually come from?
If you have seen derivatives, this is one line of work. The error J is a function of w and β, so the partial derivative of J with respect to w is the slope of the ground in the w direction, and the same for β. Differentiate the squared error and you get:
J(w, β) = (1/n) Σ (yᵢ − (β + w·xᵢ))²
∂J/∂w = −(2/n) Σ xᵢ (yᵢ − ŷᵢ) // slope of the ground, w direction
∂J/∂β = −(2/n) Σ (yᵢ − ŷᵢ) // slope of the ground, β direction
w ← w − η · ∂J/∂w // step downhill, not up: hence the minus
β ← β − η · ∂J/∂β
∂J/∂w = −(2/n) Σ xᵢ (yᵢ − ŷᵢ) // slope of the ground, w direction
∂J/∂β = −(2/n) Σ (yᵢ − ŷᵢ) // slope of the ground, β direction
w ← w − η · ∂J/∂w // step downhill, not up: hence the minus
β ← β − η · ∂J/∂β
Read the two derivatives out loud and they stop being symbols. Both are built from the same misses (yᵢ − ŷᵢ). The one for β is just their average, because moving the initial value shifts every prediction by the same amount. The one for w weights each miss by that house's size, because tilting the line moves a 200 m² house four times as far as a 50 m² house. And the minus sign in the update is the whole trick: the derivative points uphill, and you want to go down.
The squared error is chosen partly for the reason in the previous box, and partly because this is what its derivative looks like: clean, one line, no special cases. The absolute error has a corner at zero where it has no derivative at all, which is why fitting it needs a different and slower method.
2d The full process, start to finish
2 repeat (fit, update) … 5 until a certain difference threshold is met
price = … + … × size
…RMSE now
…improvement this step
Drag the slider and watch three things move together: the line on the left, the two numbers in the formula, and the error curve. The improvement per step gets smaller and smaller, and once it drops under the threshold the loop stops. That is line 5 of the pseudocode, and it is the whole reason the algorithm ever finishes.
Press "New random start" a few times. The starting line is different every time, the path is different every time, and the numbers it lands on are the same to within a rounding error. That is worth trusting on this model, and worth being suspicious of on any bigger one.
03
Evaluation: how you know it worked
You have a line and you have an error in euros. That still does not answer the question somebody will actually ask you, which is: is that good? An RMSE of €48,000 sounds terrible for a garage and excellent for a castle. A score on its own means nothing; it only means something next to another score.
3a Better than what? The laziest possible model
So compare it to the laziest model there is: ignore the size completely and guess the average price for every single house. That model is free, it is always available, and if your line cannot beat it then your line has learned nothing. R² is exactly that comparison, written as a fraction: how much of the error the average-guesser makes has your line removed?
R² = 1 − ( Σ(yᵢ − ŷᵢ)² / Σ(yᵢ − ȳ)² ) // your errors ÷ the average-guesser's errors
…MAE
…RMSE
…R²
…
R² of 0 means "no better than guessing the average". 1 means every point sits exactly on the line. Negative means you have managed to do worse than the average, which happens more often than people admit, usually on the rows the model has never seen.
The second chart is the one professionals look at first. It plots each miss against the house's size, so a good fit looks like a shapeless cloud around zero. Any pattern in it, whether a curve, a fan or a drift, is the model telling you which assumption it has broken. Both patterns you can produce there are module 4.
3b The score that counts is on houses it has never seen
train · the line is fitted on these
test · held back
…RMSE on train
…RMSE on test
…difference
Marking your own homework is the oldest mistake in machine learning. So you hide a slice of the data, usually 20 to 30 percent, fit on the rest, and score on the slice the model has never seen. Shuffle the split a few times and watch the test score wobble by a few thousand euros: with 52 houses, which ones happen to land in the test set matters, and that wobble is itself information about how much you should trust any single number.
For a straight line the two scores stay close, because a line has almost no freedom to memorise anything. Remember how calm this panel is, because in §4b you will make that gap explode on purpose.
04
Failure modes: how a straight line lies, and how to counter each
A line will always give you an answer. It has no way of telling you that the answer is nonsense, and it never refuses. These are the four ways it goes wrong, in the order you are likely to meet them, each with the thing you do about it.
4a The data is not linear, so make a new column
Here is a different question about the same market: how does the price depend on the distance to the city centre? The first kilometre out of the centre costs a fortune, and the difference between 9 and 10 km away is almost nothing. That is a curve, and no straight line can be a curve.
The fix is not a different algorithm. It is a different column. This is called feature creation: you build a new column out of the ones you already have, distance squared in this case, and hand it to the same linear model. The model is still linear in its weights, which is all it ever needed to be; it just now has a bent column to be linear about.
…RMSE
…R²
…
Look at the residual chart, not the score. With one column the misses make a clean smile: over-predicting in the middle, under-predicting at both ends. That shape is the model saying "there is structure here I cannot reach". With the squared column added it collapses into a shapeless cloud, which is what "nothing left to explain" looks like.
The catch: you had to know to add that column. Nothing in the algorithm suggests it, and this is one of the places where knowing the subject beats knowing the maths.
4b Too much freedom, so hold it back with λ
If a squared column helps, why not a cubed one, and a ninth power while we are at it? Because a flexible enough curve stops following the pattern and starts following the noise: it bends to pass through individual houses that only sit where they sit by chance. That is variance: the model changes wildly when the data changes slightly, and it looks brilliant on the houses you fitted and embarrassing on the ones you did not.
The counter is regularization: add a penalty for large weights to the thing you are minimising, controlled by a number called λ (lambda). Now the model has to buy every bend with error it saves elsewhere. λ at zero is no penalty at all. λ too high and you have flattened the model into a horizontal line that ignores the data. Somewhere in between is the point where the test score is best, and that point is found by trying, not by thinking.
minimise Σ (yᵢ − ŷᵢ)² + λ · Σ wⱼ² // ridge: the penalty is the squared weights
…RMSE on train
…RMSE on test
…
the 12 houses it was fitted on
held-out houses
Start at λ = 0 and compare the two scores. The train error is beautiful and the test error is five times worse, and on the chart the filled houses sit almost on the curve while the hollow ones do not. Slide λ up and watch the trade: the train error gets steadily worse, and once the penalty starts to bite the test error falls fast, bottoming out around the middle of the slider with the two scores close together. They never cross: a model is always at least as good on the houses it was fitted to. Push it to the end and both rise together.
This penalty is ridge. Lasso is the same idea with |w| instead of w², which has the useful side effect of pushing useless weights to exactly zero, so it also picks your columns for you. Both are one argument in every library, and both are what this chapter left out of the simple version.
4c One strange house pulls the whole line
You already broke this one yourself in §2b. A single extreme point drags a squared-error line towards it, because squaring makes that one house shout louder than twenty ordinary ones. Everything about the fit is contaminated: the slope you quote in the meeting, the price you predict for every other house, and the R² you are proud of.
The counter, in order. First, look at the point and find out what it is: a data-entry mistake, a different kind of property, or a real house at a real price. Second, if it is a mistake, fix or drop it and say in writing that you did. Third, if it is real but rare, fit with the absolute error or a Huber loss, which behaves like squared error near the middle and like absolute error out at the extremes. What you must not do is delete it because it improves your score; that is not modelling, that is decorating.
4d Data that looks linear, and is not
This is the quiet one, because nothing on your screen looks wrong. Inside the range you measured, the line fits beautifully and every score agrees. The lie only starts when you ask about a size nobody in your data has. The model has never seen a 350 m² house; it does not know that price per square meter flattens out at the top of the market, because nothing in the data it was given says so. It just keeps the slope going forever, confidently.
…the line says
…reality
…off by
…
Inside the shaded band, where the houses are, the line is excellent. Drag past it and the error grows without anything on the chart warning you. The counter is a habit, not a formula: write down the range each column was fitted on, refuse to predict outside it, and be suspicious of any model whose training data does not contain examples of the thing you are asking about.
05
Prompting AI: making an assistant build it properly
An assistant will write you a working linear regression from one lazy sentence. The code will run, and it will skip every single thing in modules 3 and 4. Compare the vague and the strong version, then build your own and watch which traps each line closes.
"Fit a linear regression on my house price data in Python."
What you typically get back
Six lines that fit on everything, print an R² of 0.88, and stop. No split, so that R² is marking its own homework. No residual plot, so a curve in the data would go unnoticed. No look at the extremes, so one wrecked house can be steering the slope. And a coefficient reported to four decimal places, which reads as precision and is nothing of the kind.
no train/test splitno residual plotno outlier checkno range recordedR² quoted aloneno baseline to beat
"I have 52 houses with size_m2 (48–190) and price_eur, and I want to predict price from size. Write Python that:
1. plots the data first and reports the min and max of every column,
2. flags points more than 3 standard deviations from the fitted line, shows them to me, and does not drop anything on its own,
3. splits into train and test, fits on train only, and reports MAE, RMSE and R² for both,
4. compares against the baseline of always predicting the mean, and says whether the model beats it,
5. plots the residuals against size and tells me in words whether it sees a pattern,
6. states the fitted line as a sentence with the units, rounded honestly,
7. lists the range each column was fitted on, and warns that predictions outside it are extrapolation."
What changes
Same library, same one-line model. But now every failure mode from module 4 has something watching for it: the residual plot catches 4a, the flagged points catch 4c, the recorded range catches 4d, and the honest split makes 4b impossible to hide. You wrote the chapter into the prompt, and that is the actual skill.
split before fittingresiduals inspectedoutliers surfaced, not droppedbaseline comparedrange recordedline stated in units
06
Beyond AI: choosing the tool, owning the call
Two things survive even a perfect prompt. The first is which model to reach for, which depends on things about your data that you have to notice. The second is the set of judgment calls, which have no score attached at all, and for regression the first of those is the biggest one in the whole field.
| If your problem looks like this | Reach for | Because | The price you pay |
|---|---|---|---|
| A roughly straight relationship, few columns, and somebody will ask you to explain the model | Linear regressionthis chapter | Fast, needs little data, and the slope is a sentence a human can act on. | Straight only. Sensitive to outliers and to extrapolation, and it says nothing about cause. |
| A clearly curved but smooth relationship | Polynomial featuresor splines | Same model, bent columns. Keeps the interpretation roughly intact. | Easy to overfit, and the tails go wild fast. Needs the λ from §4b. |
| Many columns, some of them useless, and you want the model to choose | Ridge / Lassochapter planned | The penalty shrinks weak weights, and lasso pushes them to exactly zero, which is column selection for free. | One more setting to tune, and coefficients are no longer unbiased estimates. |
| Steps and thresholds, or columns that only matter in combination | Decision treeor gradient boosting | Finds interactions and jumps by itself, and needs no scaling at all. | No smooth trend, no simple sentence to quote, and it will not extrapolate at all. |
| You are predicting a label, not a number | Logistic regressionnext chapter | Same machinery, different last step: it reports a probability instead of a value. | Different scores entirely: accuracy lies, so precision and recall move in. |
6b The judgment calls no prompt makes for you
Correlation might exist, causation maybe not
Your model says a square meter is worth €2,670. It does not say that adding a square meter to your house creates €2,670. Big houses also sit in different streets, with different gardens and different neighbours, and the line has quietly bundled all of that into one number. The moment somebody uses your slope to decide what to do rather than what to expect, you have to say this out loud.
Which columns you feed it
Add postcode and the model gets better and starts pricing neighbourhoods, which in most cities means it starts pricing who lives there. A column can carry history that nobody typed into it, and the algorithm cannot tell the difference between a fair predictor and a proxy.
What an error costs, in real life
RMSE treats €40,000 too high and €40,000 too low as the same mistake. For a seller they are not the same at all. If being wrong in one direction is worse than the other, the score you optimised is not the score you care about, and no library will notice.
What it is used for
The same line that estimates a price can be used to reject a mortgage or to set someone's rent. The maths does not change, but what you owe the person on the other end of the prediction does, and "the model said so" has never been an answer.
Linear regression from scratch, one idea per cell, in the same shape as the pseudocode in module 2. Every cell points back at the panel where you can drag the same step.
1
The class and its three knobs
hyperparametersimport numpy as np class LinearRegression: def __init__(self, lr=0.05, epochs=2000, tolerance=1e-7): self.lr = lr # η is the step size from §2c self.epochs = epochs # hard stop, so a bad η cannot run forever self.tolerance = tolerance # the "difference threshold" from line 5
Three knobs, and only one of them is interesting. lr is the learning rate you pushed too far in §2c. The other two both answer the same question from opposite ends, which is when to stop: tolerance stops when the loop is no longer improving, epochs stops when it has run out of patience.
2
Fit: the five lines of pseudocode, for real
module 2 · lines 1–5def fit(self, X, y): X = np.asarray(X, dtype=float).reshape(len(y), -1) y = np.asarray(y, dtype=float) # see cell 4: without this, one column dominates every step self.mu_, self.sd_ = X.mean(0), X.std(0) + 1e-12 Z = (X - self.mu_) / self.sd_ w = np.zeros(Z.shape[1]) # line 1: start anywhere (§2a) b, n, self.history_ = 0.0, len(y), [] for _ in range(self.epochs): # line 2: repeat err = y - (b + Z @ w) # line 3: how well it fits (§2b) self.history_.append(np.sqrt((err**2).mean())) gw = -2 / n * (Z.T @ err) # line 4: which way is downhill (§2c) gb = -2 / n * err.sum() w, b = w - self.lr * gw, b - self.lr * gb if len(self.history_) > 1 and \ abs(self.history_[-2] - self.history_[-1]) < self.tolerance: break # line 5: stopped improving # undo the scaling so the coefficients are in real units again self.coef_ = w / self.sd_ self.intercept_ = b - self.coef_ @ self.mu_ return self
The five commented lines are the five lines of the pseudocode, in order. Everything else is bookkeeping. Note that history_ is kept: it is the error curve in §2d, and it costs one number per pass. If a fit ever misbehaves, that list is the first thing to print. A curve that rises tells you η is too big, and a curve still falling steeply at the last epoch tells you it simply ran out of rounds.
3
Predict, and score honestly
module 3def predict(self, X): X = np.asarray(X, dtype=float).reshape(-1, len(self.coef_)) return self.intercept_ + X @ self.coef_ def score(self, X, y): y = np.asarray(y, dtype=float) err = y - self.predict(X) # R² compares against the laziest model there is: the mean (§3a) return { "mae": np.abs(err).mean(), "rmse": np.sqrt((err**2).mean()), "r2": 1 - (err**2).sum() / ((y - y.mean())**2).sum(), }
Returning all three together is deliberate. Quoting R² alone is the single most common way a regression result gets oversold, because it hides the units: an R² of 0.88 sounds excellent right up until you notice the RMSE is €39,000 on houses that average €400,000.
4
The one line that stops it working
the scaling trap# Drop the standardisation from fit() and try two columns: # size_m2 ranges over 48 … 190 # bedrooms ranges over 1 … 5 # # The gradient for a column is scaled by that column's size, so the # step that is right for bedrooms is ~40× too small for size, and the # step that is right for size makes bedrooms explode. One learning # rate has to serve both, and it cannot. # # In the landscape picture from §2c: unscaled columns turn the round # bowl into a knife-thin ravine, and descent bounces between the walls # instead of walking down it. model = LinearRegression(lr=0.05).fit(X_train, y_train) print(f"price = {model.intercept_:,.0f} + {model.coef_[0]:,.0f} × size") print(model.score(X_test, y_test)) # the only score worth quoting
This is the from-scratch bug people spend an afternoon on. The fix is three lines and it lives in fit, before the loop: subtract the mean, divide by the standard deviation, and undo it at the end so the coefficients come back in euros per square meter. Libraries do it for you and never mention it, which is exactly why writing it once by hand is worth the afternoon.
← All algorithms
Supervised · classification
Logistic regression
The problem it solves: linear regression predicts a number. Logistic regression predicts which of two groups something belongs to. It does this by taking the exact same straight line you already built, and squashing its output into a probability between 0 and 1. Same line, same loop, same way of learning. The only new part is the squash.
What to read first: this chapter assumes you have been through linear regression. Everything about fitting a line, walking downhill and choosing a learning rate is reused here rather than repeated, and where that happens there is a link back to the exact section.
Used for
pass or failspam or not spamwill this customer churncredit decisionsmedical triage
Learning typeSupervised
Primary taskClassification
Also doesNothing else
MechanismLinear · gradient descent
Running example for the whole chapter: hours studied against whether a student passed the exam. Every point is one student, where x = hours studied and y = 1 for a pass, 0 for a fail. There is nothing in between, which is already different from house prices.
01
Intuition: keep the line, fix its output
1a Why the old line breaks
We have students. For each one we know how many hours they studied, and whether they passed. Passing is 1, failing is 0. There is no in between, which is already different from house prices.
Try fitting a straight line to it like we did before.
passed (1)
failed (0)
your line
impossible answers
…predicts at 18 hours
…predicts at 1 hour
…students given an impossible answer
Move the sliders, or press the button to fit the best line by RMSE, exactly as in the previous chapter.
A prediction of 1.4 means nothing. There is no outcome more passing than passing. And a negative probability is not a thing at all. So the straight line is producing answers that cannot exist.
We do not want to throw the line away. We want to keep it and fix its output.
1b The squash
This is the sigmoid. It takes any number, no matter how large or how negative, and squeezes it into the range between 0 and 1.
p = 1 / (1 + e−z) // any number in, a probability out
Feed it a big positive number and you get something close to 1. Feed it a big negative number and you get something close to 0. Feed it exactly 0 and you get exactly 0.5, right in the middle.
p = 0.500
Push the slider all the way to either end and watch the readout. At z = 8 the answer is 0.9997, at z = −8 it is 0.0003. Notice what it never does: it never quite reaches 1, and never quite reaches 0. The model is never fully certain. Hold onto that, it comes back in module 2.
1c And z is just the old line
Here is the part almost nothing explains properly, so we are going to be blunt about it.
z = w·x + b // the linear regression line, completely unchanged
p = 1 / (1 + e−z) // and then the squash
p = 1 / (1 + e−z) // and then the squash
That is it. That is the same line from linear regression, unchanged. The sigmoid does not replace the line. It sits after the line and cleans up its output.
Left: z = w·x + b, the linear regression line
Right: p, the same line after the squash
Both graphs are driven by the same two numbers. Drag either slider and watch them move together. Hover a student, or use the follow slider, to trace one person across both.
passed
failed
z = 0 on the left, p = 0.5 on the right
Those two marks always sit at the same x, no matter how you drag. That x is the decision boundary. The place where the old line crosses zero is the place where the model becomes undecided.
Play with the two sliders and you can feel what each one does now:
- w controls how sharp the flip is. Large w makes the curve almost a cliff, so the model goes from confident fail to confident pass over a very short stretch of hours. Small w makes it a gentle slope, so the model stays unsure across a wide range.
- b slides the whole curve left and right. It moves where the flip happens without changing how sharp it is.
CuriousWhat does w being negative mean?
Everything above assumed more hours means more likely to pass. If w is negative the curve flips, and more of the thing means less likely. That is not an error, it is the model telling you the feature works in the opposite direction.
Set w below zero in the interactive above and watch the curve mirror itself. On this data it is obviously wrong, because studying does help. On a feature like "number of nights out before the exam" a negative weight would be exactly right, and reading that sign off the model is one of the most useful things logistic regression gives you.
02
The algorithm: every step of the loop
1initialize w and b // §2a
2repeat:
3 turn each student into z, then squash z into a probability p
4 measure how wrong those probabilities were // §2b
5 update w and b to be less wrong // §2c
6until the improvement gets small enough
2repeat:
3 turn each student into z, then squash z into a probability p
4 measure how wrong those probabilities were // §2b
5 update w and b to be less wrong // §2c
6until the improvement gets small enough
If that looks familiar it is because it is the same loop as linear regression, with one extra line in the middle. That is the whole design.
2a Initialization
Same as before. Any starting values work, closer guesses just get there faster.
Already covered: Linear regression §2a, where you can watch three different starting lines walk to the same answer. Go there →The one difference worth a sentence: starting both w and b at zero is genuinely fine here, and is what most libraries do. That gives every student a probability of exactly 0.5, which is the model honestly saying it has no idea yet.
2b Measuring how wrong we are
We cannot use RMSE anymore, and the reason is interesting enough that it gets its own tangent below. What we use instead is called log loss.
The idea is to score the model on how surprised it should have been.
- The student passed, and the model said 0.9. Barely surprised. Tiny loss.
- The student passed, and the model said 0.5. Fairly surprised. Medium loss.
- The student passed, and the model said 0.02. The model was confident and it was wrong. Enormous loss.
So the rule is: take the probability the model gave to the outcome that actually happened, and punish it for being low. Written out, for a single student:
if the student passed: loss = −log(p)
if the student failed: loss = −log(1 − p)
if the student failed: loss = −log(1 − p)
Then average that over every student, exactly like we averaged the squared errors before.
loss = 0.105
…times worse than saying 0.9
Drag towards 0.01 with passed selected and watch the number explode. The curve is not symmetric around 0.5: being a bit wrong is cheap, being very confidently wrong is astronomically expensive. That asymmetry is the entire design of log loss, and it is what makes the model care about its worst answers rather than its average one.
This is why the sigmoid never touching 0 or 1 matters. If the model were allowed to say 1.0 and turned out to be wrong, the loss would be infinite. The squash protects it from ever making a claim it cannot survive being wrong about.
CuriousWhy not just use RMSE like before?
You could technically square the difference between p and the true label. It would even sort of work. But remember the mountain from the linear regression chapter, where we walked downhill to find the lowest point.
With RMSE on top of a sigmoid, the mountain stops being a nice bowl. It grows flat plateaus at the edges, and it is exactly the confidently wrong models that sit on those plateaus. On a flat plateau you cannot tell which way is down, so the model barely learns from its worst mistakes. That is the opposite of what you want.
Log loss is shaped so that the more wrong you are, the steeper the ground under you. The worse the mistake, the harder it pushes.
Squared error on top of the sigmoid
…distance travelled
…log loss of the model it reached
Log loss
…distance travelled
…log loss of the model it reached
Both marbles start at the same spot, on a model that is confidently wrong. Same data, same learning rate, same number of steps. The only difference is the shape of the ground under them.
2c Updating w and b
Identical to linear regression. Walk downhill, take steps sized by the learning rate, stop when steps stop helping.
Already covered: Linear regression §2c, the error landscape and the mountain you walk down. Go there →But there is one thing worth showing rather than hiding, because it is a genuinely satisfying payoff. In linear regression, the update was driven by (prediction − actual) times the feature. In logistic regression, after all the calculus on the sigmoid and the log loss, the update is driven by (probability − actual) times the feature.
// linear regression
∂J/∂w = (1/n) Σ (ŷᵢ − yᵢ) · xᵢ
// logistic regression
∂J/∂w = (1/n) Σ (pᵢ − yᵢ) · xᵢ
∂J/∂w = (1/n) Σ (ŷᵢ − yᵢ) · xᵢ
// logistic regression
∂J/∂w = (1/n) Σ (pᵢ − yᵢ) · xᵢ
It is the same formula. Not similar, the same. The mess of the sigmoid and the mess of the logarithm cancel each other out perfectly and leave you with the update rule you already know.
CuriousWhere does that cancellation come from?
For readers who know derivatives. The sigmoid has an unusually tidy derivative, and the derivative of the log loss contains its exact reciprocal. They meet in the chain rule and cancel:
∂p/∂z = p(1 − p) // the sigmoid's derivative
∂L/∂p = −y/p + (1−y)/(1−p) = (p − y) / [ p(1 − p) ] // the log loss's
∂L/∂z = ∂L/∂p · ∂p/∂z = (p − y) / [ p(1 − p) ] · p(1 − p) = p − y
∂L/∂w = (p − y) · x // and z = wx + b finishes it
∂L/∂p = −y/p + (1−y)/(1−p) = (p − y) / [ p(1 − p) ] // the log loss's
∂L/∂z = ∂L/∂p · ∂p/∂z = (p − y) / [ p(1 − p) ] · p(1 − p) = p − y
∂L/∂w = (p − y) · x // and z = wx + b finishes it
That is not a coincidence and it is not luck. Log loss is the loss you get by asking which w and b make the data you actually observed most likely, and the sigmoid is the function that turns a line into those probabilities. They are two halves of the same construction, so their messes are built to cancel.
2d The full process
z = … · x + …
…log loss
…accuracy at 0.5
Press New random start a few times. It lands in the same place regardless of where it starts, and the curve gets sharper as it grows more confident. That sharpening is the model deciding it is allowed to commit: early on it hedges near 0.5 everywhere, and by the end it is willing to say 0.95 for a student who studied fifteen hours.
2e Two features, and where the separating line comes from
Everything so far used one feature, so the boundary was a single point on the x axis. Real problems have more, so let us add hours slept.
z = w₁ · hours studied + w₂ · hours slept + b // same equation, one more term
passed
failed
the line where z = 0, so p = 0.5
…log loss
…accuracy
Notice the shading has a soft edge, not a hard one. The model does not flip from fail to pass at the line, it fades through it. The line you see is exactly the place where the model says fifty percent, and nothing more special than that.
This is the picture most people are shown first, a line separating two clouds of points. It is correct, but it hides where the line comes from. The line is not drawn between the groups directly. It is the set of places where w·x + b happens to equal zero, which is the same as where the sigmoid happens to equal 0.5.
03
Evaluation: the cut is yours to choose
The model does not output pass or fail. It outputs a probability. Turning that probability into a decision means picking a cut, and that cut is yours to choose, not the model's.
0.5 is the obvious cut, and it is only the obvious one because the sigmoid is symmetric around it. It is not automatically the right one.
predicted correctly
predicted wrongly
your threshold
…wrongly told they will pass
…wrongly told they will fail
…accuracy
Drag the threshold up and down. You cannot get both counters to zero, because the two groups overlap.
Every step in one direction buys you fewer of one mistake and more of the other. That trade is not a bug in the model, it is the shape of the data: some students who studied a lot failed anyway, and no threshold can unpick that.
Which of those two mistakes is worse depends entirely on what the prediction is for, and no formula can tell you that. If it is used to offer extra tutoring, wrongly flagging a passing student costs an hour of someone's time. If it were used to decide who gets to sit the exam at all, the same mistake costs a person their exam. Same number, completely different weight.
The rest of the scoring lives on the shared evaluation and comparison page: the confusion matrix, accuracy, precision, recall and F1, used by KNN and decision trees too. This chapter only covers the threshold, because that is the part specific to models that output probabilities. Go there →04
Failure modes, and how to counter each
4a The boundary is not a straight line
If one group sits in a ring around the other, no straight line will ever separate them. Same fix as linear regression: build new features out of the ones you have, and a line in the new space becomes a curve in the old one.
…accuracy
…log loss
…
The algorithm did not change. Only what it was fed. The model is still drawing one straight boundary, it is just drawing it in a space where distance from the centre is one of the axes, and a straight line there is a circle back here.
4b One group is much rarer than the other
If 95 percent of students pass, a model that predicts pass for absolutely everyone is right 95 percent of the time and is completely useless. This is the single most common way a classifier fools the person who built it.
…accuracy
…failing students it catches
…accuracy of "everyone passes"
…
Slide it right and watch accuracy climb while the model gets less and less useful. At 99 percent the model catches almost nobody who is actually going to fail, and still scores 99. The counter: never quote accuracy on unbalanced data. Look at how many of the rare group you actually catch, weight the rare class more heavily when fitting, or resample so the classes are closer to even.
4c The groups separate perfectly
This one is counterintuitive. If a line splits the two groups cleanly with a gap between them, the model can always shrink its loss a bit further by making the curve steeper, and then steeper again, forever. Nothing stops it, and w runs away towards infinity.
The fix is regularization, the same lambda idea from linear regression, which adds a penalty for large weights and gives the runaway something to push back against.
Already covered: Linear regression §4b, where you can slide lambda and watch a model stop chasing its own tail. Go there →4d Outliers, and one place this model is steadier
Worth a short comparison. In linear regression, one wild point could drag the whole line towards it, because squared error grows without limit. Here a single misplaced student can only ever contribute so much, because the sigmoid caps how wrong any one prediction can look. Logistic regression is genuinely steadier than linear regression on this one.
4e Confident far outside the data
Every student we have studied between 0 and 20 hours. Ask about someone who studied 60 and the model will happily say 99.99 percent. It has never seen anything like that and it will not tell you so.
4f The judgment point
A probability is not a verdict
When the model says 30 percent, it is not saying this student will fail. It is saying that among students who look like this one, roughly three in ten passed. That statement is about a group. The student in front of you is one person, and they are not obliged to match the average of a group they were sorted into by two numbers.
The failure is not statistical, it is what happens next. The moment a probability gets used to decide who receives help and who is written off, the model has stopped describing the world and started shaping it. Students predicted to fail get less attention, then fail more, and the model looks more correct every year.
05
What if there are more than two groups
Everything so far has been pass or fail. But plenty of questions have three or more answers: grade A, B or C, or which of four sports someone plays. Logistic regression only knows how to draw one boundary between two groups, so we build the multi-class version out of several two-group models.
One versus rest. Train one model per class, and each one answers a deliberately simple question: is this an A, or is it anything else? With three classes you get three models, each producing a probability, and you take whichever is highest. Three classes means three models.
One versus one. Train one model for every possible pair instead. A against B, A against C, B against C. Each model votes for one class, and the most voted class wins. Three classes means three models here too, but with five classes it is ten, and it grows fast.
grade A
grade B
grade C
…
…models trained
…accuracy
Watch the boundaries appear one at a time. One versus rest draws one line per class, one versus one draws one line per pair, and the two schemes can disagree in the awkward middle region where the clouds meet, because neither has a rule for a point that every model wants to claim.
One versus rest is the usual default, because it stays cheap as classes pile up. One versus one is worth the cost when the classes are hard to tell apart in pairs, since each model only ever has to worry about two of them at a time.
CuriousThere is also a direct version
You do not strictly have to stitch together two-group models. There is a version called softmax regression which generalises the sigmoid to produce probabilities for all classes at once, adding up to 1, in a single model rather than several. It is fitted the same way, with the same loop, and it is what neural networks use at their final layer. When that chapter exists, this is the paragraph it starts from.
06
Prompting AI: making an assistant build it properly
An assistant will write you a working classifier from one lazy sentence. It will report an accuracy, it will sound pleased with itself, and it will quietly make three decisions that were yours to make.
"Predict whether students pass using logistic regression in Python."
What you typically get back
Five lines that call .predict(), print an accuracy of 0.85, and stop. .predict() already threw the probabilities away and applied a cut of 0.5 without mentioning it. Nothing says how many students failed in the data, so you cannot tell whether 0.85 is good or whether "everyone passes" would have scored the same. And nothing looks at which of the two mistakes it is making.
labels, not probabilitiesthreshold hidden at 0.5class balance unknownaccuracy quoted aloneno baselineno split
"I have 40 students with hours_studied (0 to 20) and passed (1 or 0), and I want to predict passing. Write Python that:
1. reports how many passed and how many failed before anything else, and warns me if the classes are unbalanced,
2. splits into train and test, and fits on train only,
3. returns probabilities with predict_proba, not labels,
4. states which threshold it used and why, and shows how the two kinds of mistake trade off as that threshold moves,
5. compares against the baseline of always predicting the majority class,
6. reports the confusion matrix rather than accuracy alone,
7. prints the fitted w and b, and says in words which direction the feature pushes the answer,
8. records the range of hours it was fitted on and warns that predictions outside it are extrapolation."
What changes
Same library, same one-line model. But now the threshold is a decision you can see and argue with, the balance is on the table before any score is quoted, and the weights come back as a sentence rather than an array. Points 1, 3 and 4 are the three that are specific to this chapter, and they are exactly the three the lazy version skips.
probabilities returnedthreshold stated and justifiedbalance checked firstconfusion matrixbaseline comparedweights read out loud
07
Beyond AI: choosing the tool, owning the call
It is tempting to file logistic regression as the simple thing you learn before the real models. That is the wrong framing. It is fast, it is auditable, and you can read straight off the weights which feature pushed the answer which way, which is exactly what almost nothing else on this site can offer. Often it is the right answer rather than a stepping stone, and the burden of proof sits with whatever you want to replace it with.
| If your problem looks like this | Reach for | Because | The price you pay |
|---|---|---|---|
| Two groups, a roughly straight boundary, and somebody will ask you why the model said that | Logistic regressionthis chapter | Fast, needs little data, gives calibrated probabilities, and every weight is a sentence you can defend. | One straight boundary only. Needs the features to already be informative. |
| The boundary is clearly curved, but you still want probabilities you can read | Logistic regressionwith built features | Squares, products and distances turn a straight boundary into a curved one, as in §4a. | You have to know which features to build. Nothing suggests them for you. |
| Thresholds and interactions everywhere, and no clean line anywhere | Decision treeor gradient boosting | Finds the splits and the interactions on its own, and needs no scaling at all. | Harder to audit, needs more data, and probabilities come out poorly calibrated. |
| The classes are separated by which examples they sit near, not by a line | K-nearest neighboursthe next chapter | No training step at all, and it copes with strange boundary shapes. | Slow at prediction time, and it gives you no explanation whatsoever. |
| You are predicting a number, not a group | Linear regressionprevious chapter | Same machinery, one step shorter: no squash, and the error is measured in the units you care about. | Different scores entirely, and one wild point can drag the whole line. |
7b The judgment calls no prompt makes for you
Where you put the cut
Module 3 showed the threshold trading one mistake for the other. Which of the two you would rather make is a statement about consequences, not about data, and it belongs to whoever has to live with them. Writing it down is part of the model.
Which features are allowed
A weight is readable, which cuts both ways. If postcode earns a large weight you can see it, and then you have to decide whether a model is allowed to use it. Being able to read the answer creates the obligation to look.
Who is missing from the data
Every student here sat the exam. The ones who dropped out earlier are not in the table at all, so the model has never seen them and cannot tell you it is missing them. No score notices an absence.
What happens to the person
The same probability can trigger an offer of help or a refusal. The maths does not change between those two uses, and the model has no idea which one it is feeding.
Logistic regression from scratch. Put it next to the linear regression listing and the diff is three lines: one squash, one loss, and nothing at all in the update.
1
The class, and the squash
module 1import numpy as np def sigmoid(z): # clip first: exp(800) is inf, and inf poisons everything downstream return 1.0 / (1.0 + np.exp(-np.clip(z, -500, 500))) class LogisticRegression: def __init__(self, lr=0.1, epochs=2000, tolerance=1e-7): self.lr, self.epochs, self.tolerance = lr, epochs, tolerance
The same three knobs as the linear version, and one new function. The clip is not decoration: a large z inside np.exp overflows to infinity, which turns the loss into nan and takes the whole fit down with it silently.
2
Fit: the same loop, one extra line
module 2def fit(self, X, y): X = np.asarray(X, dtype=float).reshape(len(y), -1) y = np.asarray(y, dtype=float) self.mu_, self.sd_ = X.mean(0), X.std(0) + 1e-12 Z = (X - self.mu_) / self.sd_ w = np.zeros(Z.shape[1]) # zeros are fine here: every p starts at 0.5 (§2a) b, n, self.history_ = 0.0, len(y), [] for _ in range(self.epochs): p = sigmoid(b + Z @ w) # line 3: squash the line into a probability self.history_.append(self.log_loss(y, p)) # line 5: (p - y) * x. Identical to linear regression, where it # was (prediction - y) * x. The sigmoid and the log cancelled. gw = (Z.T @ (p - y)) / n gb = (p - y).sum() / n w, b = w - self.lr * gw, b - self.lr * gb if len(self.history_) > 1 and \ abs(self.history_[-2] - self.history_[-1]) < self.tolerance: break self.coef_ = w / self.sd_ self.intercept_ = b - self.coef_ @ self.mu_ return self def log_loss(self, y, p, eps=1e-15): p = np.clip(p, eps, 1 - eps) # log(0) is -inf, and a real p is never 0 return -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))
Hold this next to fit in the linear regression chapter. The loop is the same, the stopping rule is the same, the rescaling is the same. One line squashes, one line scores differently, and the gradient is character for character what it was before.
3
Give back probabilities, and let the caller cut
module 3def predict_proba(self, X): X = np.asarray(X, dtype=float).reshape(-1, len(self.coef_)) return sigmoid(self.intercept_ + X @ self.coef_) def predict(self, X, threshold=0.5): # the threshold is an argument on purpose. It is a decision about # consequences, not about data, so the caller has to make it (§3). return (self.predict_proba(X) >= threshold).astype(int)
Most libraries hide the 0.5 inside predict, which is how a judgment call becomes an invisible default. Making it an argument with a stated value costs nothing and keeps the decision where it belongs.
4
The bug that is not a bug
§4c · perfect separation# Fit this on data where the two groups are cleanly separated and watch: # # epoch 100 w = 4.2 log loss = 0.061 # epoch 1000 w = 11.8 log loss = 0.007 # epoch 10000 w = 26.3 log loss = 0.0007 # # Nothing is broken. With a gap between the groups, every increase in w # makes the curve steeper and the loss slightly smaller, forever, so the # fit never converges and the weights stop meaning anything. model = LogisticRegression(lr=0.1).fit(X_train, y_train) p = model.predict_proba(X_test) # probabilities, then you choose the cut print(model.coef_, model.intercept_) # and these are readable, which is the point
The fix is the penalty from linear regression §4b: add lam * w to the gradient and the runaway has something pushing back. Every library does this by default, which is why you may never have met the problem, and also why a coefficient of 26 should always make you check whether your classes are separable rather than celebrate.
← All algorithms
Supervised · classification
K-nearest neighbours
The problem it solves: something new arrives and you have to put a label on it. KNN puts the label on by looking around. It measures how far the new thing is from every example you already have, keeps the k closest, and gives the new thing whatever label most of those k are carrying. That is the whole algorithm. There is no line to fit, no weights to learn, and no training step at all.
What to read first: nothing. This is the one chapter on the site that needs no earlier one. If you have read logistic regression you will recognise the ideas in module 3 and 4, and where that happens there is a link back rather than a repeat.
Used for
recommendationhandwriting and image matchingfraud and anomaly triagefilling in missing valuesa first honest baseline
Learning typeSupervised
Primary taskClassification
Also doesRegression
MechanismDistance based · no training
Running example for the whole chapter: a packing line that sorts fruit into crates. A belt carries apples, oranges and lemons past two sensors, so every fruit arrives as two numbers: width in centimetres and weight in grams. Seventy-four fruits have already been sorted by hand, and those are the examples the machine gets to look at. Everything that rolls past after that has to be labelled from those seventy-four alone.
01
Intuition: you have been doing this all along
1a Circle the group you can already see
Below are the seventy-four fruits the line has already sorted, plotted by width and weight. The labels are hidden, so all you have is the shape of the cloud.
Drag a loop around the fruits you think belong to one type. Do it by eye. You will be right.
Drag anywhere on the plot to draw. Let go and the loop closes itself.
…fruits inside your loop
…most common type inside
…share that type holds
Draw a loop and the fruits inside it will tell you what they actually were.
Nobody told you where the groups were. You found them by noticing that things near each other tend to be the same thing, which is the only assumption this entire algorithm makes.
Now notice what you did not do. You did not work out an equation for the boundary, and you could not write down the loop you drew as a formula if somebody asked. You looked at what was nearby and went with the local majority.
KNN never draws that loop. It does something lazier and, it turns out, more flexible: it waits until a new fruit shows up and only then asks who is standing closest.
1b Drop a fruit on the belt
Same plot, labels revealed. Click anywhere to put a new fruit on the belt, then drag it around. The lines show its distance to the k nearest sorted fruits, and those k are the only ones with a vote.
apple
orange
lemon
the fruit you dropped
Click on the plot to drop a fruit, then drag it into a crowd and watch the verdict change.
Put your fruit deep inside one crowd and every k gives the same answer. Put it on the seam between apples and oranges and the answer starts flipping as you change k, which is module 3's whole problem in one gesture.
That is the algorithm. Measure, rank, take the top k, count the labels, go with the majority. Everything left in this chapter is about the three quiet decisions hiding in that sentence: what measure means, what k should be, and whether the numbers being measured are on speaking terms with each other.
CuriousIf there is no training step, what is the model?
The data. That is not a joke or a simplification: the seventy-four sorted fruits are the model. Fitting a KNN classifier means storing them, which is why the fit is instant and why the library call that does it looks suspiciously like it forgot to do anything.
The name for this is a lazy learner. Linear and logistic regression are eager: they do the work once, boil a whole dataset down to a couple of numbers, and then throw the dataset away. KNN defers all of the work to prediction time and throws nothing away. Every chapter on this site has a moment where the cost has to be paid, and this is the only algorithm that pays it at the end rather than the beginning. Module 4e is about the bill.
02
The algorithm: every step of the loop
There is no loop over epochs here, because there is nothing to improve. The loop is over the data you already have, and it runs once per question you ask.
1store every sorted fruit, exactly as it came // this is the whole of "training"
2for each new fruit:
3 measure the distance to every stored fruit // §2a
4 sort those distances and keep the nearest k // §2b
5 count the labels among those k, and take the majority // §2c
2for each new fruit:
3 measure the distance to every stored fruit // §2a
4 sort those distances and keep the nearest k // §2b
5 count the labels among those k, and take the majority // §2c
Three lines of work, and each one hides a decision that changes the answer. We will take them in order.
2a Line 3: measure against every single one
To know which fruits are closest, there is no shortcut past measuring all of them. Nothing has been summarised, so nothing can be skipped. Seventy-four stored fruits means seventy-four distances, every time, for every new fruit forever.
Drag the pale fruit anywhere. Every line you can see is one distance the machine has to work out before it can answer.
…distances measured
…nearest of them all
…farthest of them all
The ranking that comes out
…
Only the order of that list survives into the next step. The distances themselves get thrown away the moment the sort finishes, which is a small fact with large consequences, and §2c is where they land.
2b Line 4: deciding what "near" is allowed to mean
So far "distance" has quietly meant the straight line between two dots, because that is what a plot makes you think of. It is one choice out of many, and each one draws a different shape around the new fruit.
The shape below is every point that sits at exactly the same distance from the new fruit. Under a straight line that shape is a circle. Under other measures it is not.
apple
orange
lemon
the new fruit
everything the same distance away
Both axes are in standard deviations, so the picture is square and the shapes are honest. §2c is about why that is not cheating.
d = √( Δwidth² + Δweight² )
…what the k nearest vote for
…neighbours swapped out
…
On two tidy features the four measures mostly agree, and the swap counter mostly reads zero. That is the honest result, and it is worth seeing before the table below tempts you into thinking the choice is always dramatic. It becomes dramatic when the features are counts, categories, or many.
Which measure, and when
Distance is a slot in the algorithm, not a fact about the world. Anything that turns two examples into one number and behaves sensibly can go in the slot. These are the ones you will actually meet.
| Measure | What it adds up | Reach for it when | The catch |
|---|---|---|---|
| EuclideanL2 · the straight line | The square root of the summed squared gaps. | Features are continuous, comparable and few. The default, and a fine one. | Squaring lets one badly wrong feature dominate the total, and it fades fastest as columns pile up (§4a). |
| ManhattanL1 · city blocks | The summed gaps, no squaring. | Features are separately meaningful, or there are a lot of them. Steadier than Euclidean when one feature is odd. | Depends on which way the axes point, so rotating the data changes the answer. |
| ChebyshevL∞ · the worst gap | Only the single largest gap. The rest are ignored. | Being far off on any one feature should disqualify a neighbour on its own. Tolerance checks, grids, board games. | Throws away every feature but one, so it wastes most of what you measured. |
| Minkowskithe family they all live in | The summed gaps each raised to p, then the p-th root. | You want to tune the measure rather than pick one. p=1 is Manhattan, p=2 is Euclidean, large p approaches Chebyshev. | One more knob to cross-validate, and it usually earns less than fixing your features would. |
| Cosinethe angle, not the length | The angle between two rows read as arrows from the origin. | Length is an artefact. Text, word counts, ratings, embeddings: a long document and a short one on the same subject should count as near. | Deliberately blind to magnitude, so it cannot tell a big thing from a small thing of the same shape. |
| Hamminghow many differ | The number of positions where two rows disagree. | Everything is categorical or binary. Yes and no columns, genetic sequences, error-correcting codes. | Every disagreement counts the same, so a wrong country weighs exactly as much as a wrong day of the week. |
| Jaccardoverlap of sets | What two sets share, divided by everything they cover between them. | Rows are sets of tags, ingredients, purchases or visited pages, and lengths differ. | A shared absence counts for nothing, which is right for baskets and wrong for questionnaires. |
| Mahalanobisdistance that knows the spread | The gap measured in units of how the data actually varies, with correlated features counted once. | Features are correlated and on different scales, and you would rather the measure handled that than do it by hand. | Needs enough rows to estimate the covariance, and it is fragile when you do not have them. |
| Gowermixed columns | A per-column score, numeric or categorical, averaged into one number. | The table has ages, prices, postcodes and yes-or-no columns side by side. Real tables usually do. | The averaging is a weighting decision, made for you, and rarely questioned. |
| Edit distanceand dynamic time warping | How many changes turn one sequence into the other, allowing for stretch. | Rows are not points at all: they are words, gestures, sensor traces, or two recordings of the same movement at different speeds. | Slow, and quadratic in the length of the sequences, so a full scan gets expensive fast. |
Two rules cover almost every real case. If the columns are numbers that mean the same kind of thing, start with Euclidean and only move if you have a reason. If length is noise rather than information, use cosine. Everything else on that table is a specific answer to a specific shape of data.
CuriousDoes it have to be a distance at all?
No. The algorithm only ever asks the measure to rank things, so anything that gives a consistent ordering will run. It does not even have to be a distance in the mathematical sense, which requires that nothing is nearer to something else than it is to itself, and that going via a third point is never a shortcut.
Two families get used constantly and neither is a distance:
- Similarity, the mirror image. Cosine similarity and Jaccard both answer "how alike", where bigger means closer. You sort descending instead of ascending and nothing else changes. Most libraries take the measure as an argument for exactly this reason.
- A measure the data chooses for itself. Rather than picking a formula, you can learn one, by fitting the weighting that pulls examples of the same class together and pushes the others apart. That is metric learning, and it is what an embedding model is doing when it puts two photographs of the same person near each other. Face recognition on your phone is very close to KNN run on a learned measure.
There is a real limit though. Break the triangle rule and you lose the tricks that make prediction fast (§4e), because those rely on being able to say "everything in this branch is at least so far away" without checking. A strange measure means a full scan every time, with no way out.
2c Line 5: the vote, and the ranking it counts on
The vote itself is the least interesting line in the algorithm. Count the labels among the k nearest, take whichever appears most, done.
What makes it interesting is what it is counting. The vote never sees a distance. It sees a list in order, and takes the top k. So anything that changes the order changes the answer, and the order is decided entirely by which units you happened to measure in.
Here is the problem in our own data. Width is measured in centimetres and spans about six of them. Weight is measured in grams and spans about two hundred. Ask for the straight-line distance between two fruits and the grams swamp the centimetres before you have finished typing the formula.
apple
orange
lemon
everything counted as near
…weight counts this much more than width
…accuracy on 40 unseen fruits
The five nearest, in order
…
In raw units the neighbourhood is a flat band. Anything within a few grams counts as near no matter how much wider or narrower it is, so width is not being under-weighted, it is being ignored. And width is the one thing that separates an apple from an orange here, because an orange carries a thick peel and is wider than an apple of the same weight.
The fix is one line and it is the most valuable line in this chapter: put every feature on the same footing before measuring anything. Subtract the mean of the column, divide by its spread, and now one step is one step whichever column you took it in.
for each feature: z = (value − mean of that column) ÷ spread of that column // then measure distances on z
Standardising is not a neutral act, though, and pretending otherwise is how people get surprised later. It is a claim that every feature deserves the same say. Sometimes it does not, and then you weight on purpose: the third setting above lets you decide that width is worth twice what weight is, or that weight should count for nothing. Turn one of them down to zero and watch the neighbourhood collapse into a stripe again.
Two habits keep this honest. Work out the mean and the spread on your training data only, and reuse those exact numbers on everything that arrives afterwards, or the new fruit gets judged by a ruler it helped to build. And when you report a result, convert back: a neighbour is 0.4 standard deviations away only to the algorithm. To the person reading the report it is half a centimetre and nine grams, and that is the version they can argue with.
CuriousWhat happens in a tie?
Ties are common and libraries are quiet about them. Set k to 4 in the interactive in §1b, drop a fruit on the seam between apples and oranges, and you can get two votes each. Something has to break it.
- Make k odd. The advice everyone gives. It works for exactly two classes and does nothing at all for three, where three neighbours can be an apple, an orange and a lemon.
- Let the nearest one decide. Among the tied classes, pick the one owning the single closest neighbour. Cheap and defensible.
- Add up the distances. Sum the distance of each tied class's neighbours and take the smaller total, so four votes clustered tightly beat four spread thin. This is distance weighting arriving through the back door.
- Weight every vote by nearness from the start, so a neighbour half as far counts twice. Exact ties then stop happening almost entirely, and this is usually the right answer. It has a second benefit worth more than tie-breaking: it makes a large k much safer, because distant neighbours stop being able to shout (§3).
- Shrink k by one and ask again until the tie breaks. Tidy, and it quietly turns your chosen k into a suggestion.
Worth knowing what your library does when you have not said. Scikit-learn's classifier resolves a tie by taking the class that comes first in sorted order, which for labels named "apple" and "orange" means apples win the alphabet. That rule is not wrong so much as arbitrary, and it will never mention itself in the output.
CuriousWhat if the answer is a number rather than a label?
Then you average instead of counting, and everything else stays exactly where it is. Find the k nearest, take the mean of their answers, and that is KNN regression. Use the median instead of the mean and one wild neighbour stops mattering.
It is genuinely useful for the thing regression struggles with: a relationship that is not a line and does not want to become one. Predicting the rent of a flat from its position on a map is the standard example. There is no formula from latitude and longitude to rent, but the four nearest flats know.
The catch it inherits is worth stating, because it is the one people trip over. A KNN regressor can never predict a value outside the range it has seen, since an average of stored answers is bounded by those answers. Ask linear regression about a house twice the size of anything in its data and it will confidently extrapolate, and be wrong. Ask KNN and it will hand you back the average of the biggest houses it knows, and be wrong in the opposite direction, but it will at least be wrong inside the realm of things that have actually happened.
03
Evaluation: k is the only dial, and it lies to you
Every other model on this site has a loss you can watch fall. KNN has nothing of the kind, because it never fits anything, so there is no curve to stare at and no moment where it settles. The single number you get to choose is k, and choosing it badly is the main way this algorithm goes wrong.
3a Watch the boundary breathe
The coloured background is the machine's answer for every possible fruit: drop one anywhere in the pale coral region and it goes in the apple crate. The solid dots are the seventy-four sorted fruits it learned from. The hollow rings are forty fruits it has never seen, held back to mark its homework.
The regions at k =5
Accuracy against k
…accuracy on the fruits it stored
…accuracy on the 40 it has never seen
…stored score minus unseen
…
apple
orange
lemon
held-out fruit
accuracy on the stored fruits
accuracy on the unseen forty
Three things are worth doing to that slider before you read on.
- Push it to k = 1. Accuracy on the stored fruits jumps to a perfect 100 percent, and it will do that on any dataset you ever feed it, because the nearest fruit to a stored fruit is itself, at a distance of zero. The model has memorised the answer sheet. Meanwhile accuracy on the unseen forty drops below every other small setting: every k from 3 into the forties beats it. This is the cleanest demonstration on the site of why a score on the data a model has memorised is not a score at all.
- Walk it up slowly. The islands dissolve, the boundary straightens, and the two accuracies converge. Somewhere in the low single figures the held-out score peaks. That is the trade every model makes, and here it is a single integer.
- Push it to the far end. With k at seventy-three, almost every fruit on the line is voting on every question, so the answer stops depending on the question. The whole plot turns one colour and accuracy drops to 35 percent, which is not a coincidence: it is exactly the share of apples in the test set. The model has become "always say apple" with extra steps.
So small k follows the noise and large k ignores the signal. Reading that as bias and variance is the right instinct, and it is the same dial as lambda in linear regression and depth in a decision tree, wearing different clothes.
3b How you actually pick k
Not by eye, and not from that curve. The forty held-out fruits above are drawn so you can see the shape of the problem, but the moment you choose k by looking at them you have started fitting to them, and their score stops meaning anything.
The real procedure is cross-validation: split the training fruits into five parts, and for each candidate k, fit on four parts and score on the fifth, rotating until every part has taken a turn as the scorer. Average the five, take the k with the best average, and only then let the held-out forty pass judgment, once.
Three rules of thumb that survive contact with real data:
- Odd k for two classes, to stop the most common tie. It does nothing for three classes or more, as the tie aside in §2c explains.
- Weight the votes by nearness rather than counting them equally, and large k stops being dangerous, because a neighbour on the far side of the plot can no longer shout as loudly as one next door.
- The square root of the number of stored examples is where people start. For seventy-four fruits that suggests about nine. It is a starting point for the search, not a result.
Accuracy is also the wrong score to be reading here, and this chapter has been quietly abusing it. It is fine on this data because the three crates are almost equally full. Change that and it stops being fine, which is the subject of a page shared by every classifier on the site.
The rest of the scoring lives on the shared evaluation and comparison page: the confusion matrix, precision, recall, F1 and what to do when one class is rare. Logistic regression and decision trees point at the same page rather than each explaining it again. Go there →04
Failure modes, and how to counter each
KNN fails in ways that are unusually easy to demonstrate, because there is no fitting process to hide behind. When it goes wrong, it is the data that is wrong, and you can see it.
4a Too many columns, and "near" stops meaning anything
This is the one that kills the algorithm in production, and it is deeply unintuitive: adding more information makes it worse. Not slower. Worse.
Suppose the packing line also records the belt speed, the hour of the day, the lane number, the humidity in the shed, and twenty other things it happens to have a sensor for. None of them says anything about fruit. Watch what they do anyway.
Accuracy as useless columns are added
Every stored fruit, by how far away it is
…accuracy at k = 5
…farthest fruit ÷ nearest fruit
…columns in total
…
The right-hand strip is the part to watch. With two columns the farthest fruit is around thirty times further off than the nearest, so "nearest" picks out a real, small group. Add noise columns and every fruit gets a random contribution in every one of them, the distances all drift towards the same middling value, and the crowd closes up. By twenty columns the farthest fruit is barely twice as far as the nearest. The word nearest has stopped selecting anybody in particular, and the vote is now being cast by a more or less random handful.
This is the curse of dimensionality, and it is not really about dimensions. It is about volume. Every column you add gives the space another direction to be empty in, and the same seventy-four fruits have to spread themselves across all of it. To keep the neighbourhood as densely populated as it was with two columns, you would need exponentially more fruit, and nobody has exponentially more fruit.
The counters, in the order worth trying:
- Drop columns that carry nothing. The cheapest and by far the most effective. Every column has to earn its place, and a column that does not is not neutral, it is actively harmful. Distance-based models care about this more than any other family.
- Compress rather than delete. If forty columns genuinely contain information, squeeze them into a handful of combined ones first with something like PCA, then run KNN on those.
- Weight the features instead of trusting them equally, as in §2c, so a column that barely matters barely counts.
- Switch to Manhattan or a fractional p. Squaring is what lets many small random gaps add up into a large one, so a measure that does not square degrades more slowly. It buys you a few columns, not thirty.
- Use a different family. A decision tree simply ignores a useless column, and a linear model gives it a weight near zero. Neither has to be talked out of it. If you cannot get the column count down, the honest move is to stop using KNN.
4b The feature nobody wired up
The opposite failure, and much more common than the curse in practice. Apples and oranges overlap on this plot because width and weight genuinely cannot tell them apart in the region where they meet. No k fixes that. No distance measure fixes that. The information is not in the data.
The packing line has a camera. It could measure how bumpy the peel is, and an orange is dimpled where an apple is smooth. Nobody wired that value into the table.
What the machine is given: width and weight
apple
orange
lemon
held-out fruit
put in the wrong crate
What it is not given: peel roughness
…accuracy at k = 5
…fruits in the wrong crate
…
Nothing in the left-hand picture moves when you switch. The same fruits sit in the same places and the apples and oranges still overlap exactly as much as they did. The mistakes disappear anyway, because the machine is now measuring in a direction this picture cannot show. Every failure on the left was a missing column, not a bad algorithm, and no amount of tuning would have found it.
This is the counterweight to §4a and the two have to be held together. Useless columns poison KNN, and a missing useful column caps it. The work is choosing which columns, and that work is yours: no search over k, no cross-validation and no library default has any way to tell you that the camera was pointed at the fruit the whole time.
4c One fruit in the wrong crate
Somebody put an orange in the apple pile by hand, months ago, and it has been sitting in the training data ever since. At k = 1 that single mistake gets a territory of its own, and every fruit that lands in it inherits the error.
Drag the ringed fruit into the middle of the oranges and watch it carve a hole.
…of the plot this one fruit controls
…accuracy on the unseen 40
…
The counter is k itself. Move the slider up by two and the island shuts. A single wrong label cannot outvote three right ones, so raising k is the cheapest noise filter there is, and it is the reason k = 1 is almost never the answer outside a textbook. Weighting votes by nearness helps too, though less than you would hope: the mislabelled fruit is genuinely the nearest one, so nearness weighting hands it the loudest voice of all.
Which leaves the boring fix, which is also the correct one: look at your labels. KNN is unusually good at showing you where they went wrong, because a mislabelled example produces a visible island rather than a slightly worse number. Run leave-one-out over the training set, list every fruit whose neighbours all disagree with it, and hand that list to whoever did the labelling.
4d One crate is much fuller than the others
Suppose the line handles ten thousand apples a day and two hundred lemons. Around any new fruit, the neighbourhood is mostly apples for the simple reason that the world is mostly apples, and lemons lose votes they should have won. Nothing has gone wrong mechanically. The vote is doing exactly what it was told, and the count is a popularity contest before it is a similarity judgment.
Weighting votes by nearness helps, because a lemon standing right beside the new fruit outweighs six apples further off. Beyond that the counters are the ones shared by every classifier, and they live on the evaluation page: resample so the classes are closer to even, weight the rare class up, and stop quoting accuracy, which will read 98 percent while the model never once says lemon.
4e The bill arrives at prediction time
The zero-cost training is not free, it is deferred. Every question means measuring against every stored example, so with a million rows and twenty columns, one prediction is twenty million multiplications. Linear regression answers the same question with twenty. The model that took no time to build is the one you cannot afford to use.
- Index the data with a KD-tree or a ball tree, which lets whole branches be skipped once they are provably too far away. Free accuracy, real speedup, and it quietly stops working once you have more than about twenty columns, for the reasons in §4a.
- Give up exactness. Approximate nearest neighbour libraries return almost the right neighbours in a tiny fraction of the time. Every large-scale vector search you have used today was doing this.
- Thin the stored set. Most examples sit deep inside their own crowd and never change a vote. Keep the ones near the boundaries and drop the rest.
- Remember what you are storing. The model is the training data, so shipping the model ships every row of it. For medical records or anything about identifiable people that is not an implementation detail, it is a disclosure, and it is the argument against KNN that comes from outside the maths.
4f The judgment point
"Because five things like you were"
KNN cannot give a reason. Ask logistic regression why and it will point at a weight. Ask a decision tree and it will read you the questions it asked. Ask KNN and the complete, honest answer is a list of five neighbours. There is no rule to inspect, no coefficient to argue with, and nothing to appeal against.
For fruit that is fine. For a person it is a different object entirely. A decision made this way says you were treated as your five closest matches were treated, and if those five were treated unfairly, the unfairness is copied forward exactly and called a prediction. The algorithm has no concept of the past being wrong. It only knows that it happened, nearby.
There is a strange honesty in that. Other models blur the same problem into a coefficient and make it look like reasoning. KNN hands you the five rows and lets you see whose lives the answer was actually copied from, which is uncomfortable, and better.
05
Prompting AI: making an assistant build it properly
KNN is four lines long, so an assistant will get it running instantly and correctly. That is the trap. The code will be right and the result will be wrong, because everything that decides whether KNN works happens before the algorithm is called and none of it is visible in the output.
"Classify my fruit data with KNN in Python."
What you typically get back
Four lines that import KNeighborsClassifier, fit it on everything, print an accuracy, and stop. No scaling, so whichever column happens to be measured in the largest units has silently decided every answer (§2c). A default k of 5 that was never chosen. A score taken on the same rows the model memorised, which for k = 1 would have read a flawless 100 percent. And no word about how many columns went in.
features never scaledk left at the defaultscored on stored datacolumn count uncheckedties resolved alphabeticallyno baseline
"I have 114 fruits with width_cm (5 to 10), weight_g (80 to 260) and a label of apple, orange or lemon, and I want to classify new fruit. Write Python that:
1. splits into train and test first, and touches the test set exactly once at the end,
2. standardises the features inside the pipeline, fitting the scaler on train only, and says in words why raw centimetres and grams cannot be compared,
3. chooses k by 5-fold cross-validation over a range, and prints the whole curve rather than only the winner,
4. uses distance-weighted voting and states how ties would be broken,
5. reports the confusion matrix and a majority-class baseline, not accuracy alone,
6. names how many columns are being used and warns me if that number grows,
7. prints, for three test fruits, the actual neighbours that decided them, in the original centimetres and grams."
What changes
The same four-line model, wrapped in the seven decisions that were always going to determine the result. Point 2 is the one that moves the number most on this data, and it is the one no assistant volunteers. Point 7 is the one that changes how you work: KNN is the only model on this site that can show you its evidence row by row, and asking for it turns an unexplainable classifier into the most inspectable thing you own.
scaled inside the pipelinek cross-validateddistance-weighted votestie rule statedbaseline comparedneighbours shown in real units
One extra instruction is worth more here than in any other chapter: ask it to put the scaler and the model in a single pipeline object. Scaling by hand before splitting is the most common bug in beginner KNN code, and it is invisible: the model quietly learns the mean and spread of the test set, the score comes out flattering, and nothing anywhere reports a problem.
06
Beyond AI: choosing the tool, owning the call
KNN earns its place twice. It is the honest baseline: if a neural network cannot beat "ask the five most similar rows", the network is not learning anything the data does not simply contain, and you have found that out in an afternoon. And it is the right answer outright whenever the boundary is a strange shape, the dataset is small, and new classes keep appearing, since adding a class means adding rows rather than retraining anything.
| If your problem looks like this | Reach for | Because | The price you pay |
|---|---|---|---|
| Few columns, an awkward boundary, and new categories that keep turning up | K-nearest neighboursthis chapter | No training, no assumption about the shape of the boundary, and a new class costs you one insert. | Every prediction scans everything, the model is your raw data, and it falls apart past about twenty columns. |
| Two groups, a roughly straight boundary, and somebody will ask you why | Logistic regressionprevious chapter | Instant predictions, calibrated probabilities, and a weight per feature you can read out loud. | One straight boundary only, and the features have to be informative already. |
| Many columns, mixed types, and thresholds rather than distances | Decision treeits own chapter | Ignores useless columns for free, needs no scaling at all, and reads back as a list of questions. | Unstable on small data, and a single tree overfits unless it is pruned or grown in a forest. |
| Hundreds of columns, or images, text and audio | Learned embeddingsthen KNN on top | A model compresses the raw thing into a short vector where distance means what you want it to, which is exactly what §4a says KNN needs. | You now have a second model to train, explain and maintain, and the embedding decides what "similar" means. |
| No labels at all, and you want to know what groups exist | K-meansthe clustering chapter | Same instinct about distance, opposite job: it invents the groups rather than copying labels from neighbours. | You have to choose how many groups, and it will find them whether or not they are there. |
The k in KNN and the k in K-means are not the same k, and the two algorithms are not variants of one another. K-means is unsupervised and k is how many groups to invent. KNN is supervised and k is how many opinions to ask. The only thing they share is a distance function.
6b The judgment calls no prompt makes for you
What counts as similar
Standardising says every feature deserves an equal say. Leaving the raw units in says whichever column has the biggest numbers should decide. Both are claims about the world dressed as data preparation, and one of them is always being made, including when nobody chooses.
Which columns are allowed in
KNN has no way to ignore a column. Every one you include is a direction it measures in, so including a column is voting for its relevance. That includes columns that stand in for something you would never use on purpose, like a postcode that is really an income bracket.
Whose data is stored
The model is a copy of the training rows, kept whole and shipped wherever the model goes. Nothing was summarised away, so nothing was anonymised by accident. If the rows are about people, the model is a database of people.
Who the neighbours were
A prediction here is a claim that this case should be treated as the nearest previous cases were. That is only fair if those cases were handled fairly. The algorithm cannot check, and it never asks.
K-nearest neighbours from scratch. It is the shortest listing on the site, and the only one where fit does no arithmetic. Almost every line below is about the decisions around the algorithm rather than the algorithm itself, which is a fair summary of the whole chapter.
1
Fit: remember the mean, remember the spread, keep everything
modules 1 and 2cimport numpy as np class KNN: def __init__(self, k=5, p=2, weighted=True): # p is the Minkowski exponent: 1 is Manhattan, 2 is Euclidean (§2b) self.k, self.p, self.weighted = k, p, weighted def fit(self, X, y): X = np.asarray(X, dtype=float) # The mean and the spread come from the training rows and nowhere else. # Reuse these exact numbers on anything that arrives later, or a new # row gets judged by a ruler it helped to build (§2c). self.mu_ = X.mean(axis=0) self.sd_ = X.std(axis=0) + 1e-12 self.X_ = (X - self.mu_) / self.sd_ self.y_ = np.asarray(y) self.classes_ = np.unique(self.y_) return self # that is the entire training procedure
Two lines of statistics and one copy. Compare this with fit in the logistic regression chapter, which runs a loop two thousand times. Here the cost has not been avoided, it has been moved to predict, and §4e is the invoice.
The + 1e-12 matters more than it looks. A column where every row is identical has a spread of zero, and dividing by it turns the whole feature matrix into nan without a single warning.
2
All the distances, in one expression
module 2adef _distances(self, X): Z = (np.asarray(X, dtype=float) - self.mu_) / self.sd_ # Z[:, None, :] is (queries, 1, features), self.X_ is (stored, features). # Broadcasting expands both to (queries, stored, features), so every gap # is computed at once. Fast, and it allocates that whole block, so it # is also how you run a laptop out of memory on a real dataset. gaps = np.abs(Z[:, None, :] - self.X_[None, :, :]) return (gaps ** self.p).sum(axis=-1) ** (1 / self.p)
One expression, and it is the only arithmetic in the class. Note what is not here: no shortcut, no early exit, no cleverness. Every query is compared against every stored row, because nothing has been summarised that would let us skip any of them.
For the Euclidean case you can drop the outer root entirely. Sorting by squared distance gives the same order as sorting by distance, and a square root per pair adds up. Keep the root only when you intend to show the number to a person.
3
The vote, with the tie rule written down
module 2cdef predict_proba(self, X): D = self._distances(X) # argpartition beats a full sort: it only guarantees the k smallest are # at the front, and does not bother ordering the other n - k. idx = np.argpartition(D, self.k - 1, axis=1)[:, :self.k] near_d = np.take_along_axis(D, idx, axis=1) # Weight by nearness, so a neighbour twice as close counts twice. # This is what makes a large k safe, and it makes exact ties vanish. w = 1.0 / (near_d + 1e-12) if self.weighted else np.ones_like(near_d) out = np.zeros((len(idx), len(self.classes_))) for j, c in enumerate(self.classes_): out[:, j] = (w * (self.y_[idx] == c)).sum(axis=1) return out / out.sum(axis=1, keepdims=True) def predict(self, X): # argmax breaks a tie by taking the lowest class index, which means # alphabetical order. That is arbitrary, so it gets said out loud # here rather than discovered later (§2c). return self.classes_[self.predict_proba(X).argmax(axis=1)] def explain(self, x): # The one thing this model can do that nothing else on the site can: # hand back the actual rows the answer was copied from (§4f). d = self._distances([x])[0] order = np.argsort(d)[:self.k] return [(self.y_[i], self.X_[i] * self.sd_ + self.mu_, d[i]) for i in order]
The proportions coming out of predict_proba look like probabilities and are not. Three votes out of five is 0.6, and it says nothing about how confident anything is: the same 0.6 arrives whether those five neighbours were touching the query or on the other side of the plot. Treat it as a vote share, and if you need a real probability, calibrate it against held-out data first.
explain is four lines and it is the reason to reach for KNN when you have a choice. It rescales the neighbours back into centimetres and grams on the way out, because standard deviations are for the algorithm and the person reading the report needs the units they measured in.
4
Choosing k without cheating
module 3bfrom sklearn.model_selection import cross_val_score, train_test_split from sklearn.neighbors import KNeighborsClassifier from sklearn.preprocessing import StandardScaler from sklearn.pipeline import make_pipeline X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.25, stratify=y, random_state=0) # The scaler goes INSIDE the pipeline. Inside, it is refitted on the four # training folds of every split. Outside, it would have already seen the # fold being scored, and every number below would come back flattering. for k in range(1, 40, 2): pipe = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=k, weights="distance")) s = cross_val_score(pipe, X_train, y_train, cv=5) print(f"k={k:2d} {s.mean():.3f} ± {s.std():.3f}") # Print the whole curve, not the winner. If k=5 scores 0.92 and k=9 # scores 0.91, those are the same result and the larger k is steadier. # The test set gets touched once, after k is settled, and never again.
weights="distance" is the single most valuable argument in the call, and it is off by default. It fixes ties, it makes a large k behave, and it costs nothing.
The ± s.std() is not decoration either. Five folds of a small dataset produce noisy scores, and a difference of 0.01 between two values of k is very often nothing at all. Picking the winner of a coin toss and reporting it as a tuned hyperparameter is how a model looks better on a slide than it does in production.
5
The bug you will write, once
§2c · scaling# Same data, same k, same library. One line moved. # WRONG: scaled before the split, so the scaler has already measured # the mean and spread of the rows it is about to be scored on. X_all = StandardScaler().fit_transform(X) Xtr, Xte, ytr, yte = train_test_split(X_all, y, random_state=0) # held-out accuracy: 0.94 <-- and nothing warns you # RIGHT: split first, and let the pipeline fit the scaler on train only. Xtr, Xte, ytr, yte = train_test_split(X, y, random_state=0) pipe = make_pipeline(StandardScaler(), KNeighborsClassifier(5)) # held-out accuracy: 0.91 <-- the number you can actually report # And the one that is not a bug at all, just a forgotten line: # no scaler anywhere. Width in centimetres against weight in grams # means weight counts about forty times as much as width, and the # classifier has quietly become "sort the fruit by weight" (§2c).
Three points of accuracy is a small lie, which is what makes it dangerous: it is too small to notice and large enough to win an argument. The leak scales with how much preprocessing you do outside the pipeline, and on a dataset with imputation and encoding as well as scaling it stops being three points.
The rule that avoids all of it: nothing learns anything from a row it will later be scored on. Put every step that learns something inside the pipeline, and the split enforces the rule for you.
← All algorithms
Shared · classification
Evaluation and comparison
The problem it solves: you have three classifiers that all say they are about 93 percent accurate, and you have to pick one. This page is the two things that question needs. First, how to score a classifier so the number means something, because accuracy on its own routinely does not. Then a straight comparison of K-nearest neighbours, the decision tree and logistic regression across the things that actually decide the choice.
Why evaluation comes first: the comparison table has a row for probability quality, and that row is meaningless until you have seen a threshold move. So the scoring comes first and the table pays it off. If you only want the table, the second tab is right there.
Shared by
K-nearest neighboursdecision treeslogistic regressionanything else that predicts a label
Running example for the whole evaluation section: a screening test for a condition, run on 600 people. Each person arrives as two numbers, fasting glucose in mg/dL and BMI, and either has the condition or does not. About one in six do, which is the ordinary state of screening data and the reason accuracy is about to embarrass itself. Seventy percent of the people are used for fitting and the rest are held back for scoring. Everything is generated in your browser from a seed, and the three classifiers are fitted live on it, so every number below came out of a model that actually ran.
01
Accuracy
A classifier makes a prediction for every person. Some of those predictions are right. Accuracy is the share that are right, and nothing more complicated than that.
accuracy = correct predictions ÷ total predictions
Below is a screening population split evenly, half with the condition and half without, and a logistic regression fitted on it. Each dot is one person. The line is where the model changes its mind.
has the condition
does not
the model's boundary
…accuracy on people it has never seen
…correct out of the held-out group
…
One number, easy to explain to anybody, and it goes up when the model gets better. It is the first thing every library prints and the first thing every tutorial reports, and on data shaped like this it is a perfectly reasonable summary.
Hold onto that. It is about to stop working, and it matters that you accepted it as reasonable first, because the failure is not that accuracy is a silly idea. The failure is that it is a reasonable idea applied to data it does not suit, which is a much harder thing to notice.
02
Why accuracy breaks
Two separate failures. The first is famous and easy to demonstrate. The second is quieter, survives every fix for the first, and is the one that actually decides which model you should ship.
2a One class is much rarer than the other
The even split above was a convenience. Real screening data is not like that: most people do not have the condition. So take the same generator and slide the population from an even split towards the lopsided one you would actually meet.
Two models are running. One is the fitted logistic regression. The other is a model of studied stupidity: it always says "no condition", for everyone, forever. It has no features, no fitting and no thought in it at all.
The population at50% without the condition
What each model scores
…accuracy of the fitted model
…accuracy of "nobody has it"
…cases the fitted model actually catches
…
has the condition
does not
the fitted model
"nobody has it"
Take the slider to 99 and read the two numbers next to each other. The model that has learned nothing scores 99 percent. Not because it is good. Because 99 percent of the answers are "no", and it says "no".
This is the trap in one line: on lopsided data, accuracy measures the lopsidedness. A published accuracy figure with no class balance beside it is not a result, it is a number, and you cannot tell the two apart from the outside. Whenever you see one, the first question is always what the majority class would have scored on its own.
2b The two mistakes are not worth the same
Now the deeper problem, and the reason rebalancing your data does not fix everything. Accuracy adds the two kinds of mistake together as though they were interchangeable. They almost never are.
A screening test can be wrong in two directions. It can raise a false alarm, sending somebody healthy for a confirmatory test they did not need, which costs an appointment and a bad week. Or it can miss a case, telling somebody who has the condition that they are fine, which costs them the months before anybody looks again.
Here are two settings of the same fitted model on the same held-out people. They score within a percentage point of each other on accuracy. They are not remotely the same test.
Both lines start from the same accuracy. Only the mix of mistakes differs.
…cautious test, total cost
…strict test, total cost
…
| Test | Accuracy | False alarms | Missed cases |
|---|---|---|---|
| Cautiousflags anything doubtful | … | … | … |
| Strictflags only the obvious | … | … | … |
The strict test is the more accurate of the two, by one person. It also misses several times as many cases. Push the slider one notch past 1, which is to say assume that missing a case is worse than a false alarm at all, and the ranking flips and never flips back.
Note what this failure does not depend on. It is not about imbalance: it would happen on a perfectly even split. Rebalancing your classes does not touch it. Accuracy is throwing away the distinction between the two mistakes at the moment it adds them up, and once they are added there is no way to get them back out.
So we need a score that refuses to add them up. That is the next step, and it is the last new object on this page. Everything after it is a reading taken off it.
03
The confusion matrix
The fix is to stop summarising and start counting. There are four things that can happen to one person, so count all four separately and never add them together again. Laid out as a two-by-two grid, that is the confusion matrix.
Below is the running dataset: … people held back from fitting, of whom … have the condition. Every one of them is a dot, placed left to right by the probability the model gives them. People who actually have the condition sit above the line, people who do not sit below it.
Drag the vertical threshold. Everything to the right of it the model is calling positive. The four quadrants that makes are the four cells of the matrix, which is the entire idea.
has the condition
does not
your threshold
Drag anywhere on the plot to move the threshold. It is the same threshold in step 4, so what you set here is what you will find there.
model says positive
model says negative
actually positive
…TRUE POSITIVEhad it, and was flagged
…FALSE NEGATIVEhad it, and was sent home
actually negative
…FALSE POSITIVEdid not have it, and was flagged anyway
…TRUE NEGATIVEdid not have it, and was correctly cleared
…
Two naming traps, because beginners swap these constantly and so does everybody else. The first word says whether the model was right, and the second word says what the model predicted. A false negative is a negative prediction that was false, so it is a missed case. A true negative is a negative prediction that was true, so it is a healthy person correctly cleared. Nothing in the word "negative" refers to the person's actual state, only to what the model said about them.
Now read the diagonal. Accuracy is the two green cells added together and divided by all four, which is why it can be so high here: the true negative cell is enormous, because most people do not have the condition, and it drowns out everything else. The matrix does not hide that. It puts the number you should be worried about, the missed cases, in a box of its own where it cannot be averaged away.
Press "new people" and the counts all change. What does not change is the shape of the problem: a fat true negative cell, and a missed-case cell that stubbornly refuses to reach zero unless you accept a pile of false alarms. That shape is a property of the data, not of the draw.
04
Precision and recall
Two numbers, both read straight off the four cells you already have. They are not new information. They are the two questions worth asking about that grid, and each one deliberately ignores a different cell.
precision= true positives ÷ everything the model flagged // TP ÷ (TP + FP)
recall = true positives ÷ everyone who actually has it // TP ÷ (TP + FN)
recall = true positives ÷ everyone who actually has it // TP ÷ (TP + FN)
- Precision looks at the flagged column. Of everybody this test sent for a second appointment, what share genuinely needed one? Low precision means the clinic is full of people who were never at risk. Precision is the cost of false alarms.
- Recall looks at the positive row. Of everybody who actually has the condition, what share did the test find? Low recall means people went home ill and unaware. Recall is the cost of misses.
The mnemonic worth keeping, because this is where everybody gets lost: precision is judged by the model's own claims, recall is judged by reality. Precision asks whether the model's flags were honest. Recall asks whether reality was covered.
Same widget, same threshold, same people. Now watch the two numbers as you drag.
has the condition
does not
your threshold
precision…
recall…
accuracy…
…
Drag it all the way right. Precision goes to a perfect 1.00, because the only people still being flagged are the ones the model is certain about, and it is right about them. Recall collapses, because almost nobody is being flagged at all. A test that is never wrong when it speaks, and almost never speaks.
They move in opposite directions, always, and that is not a defect of this model or this dataset. Raising the threshold can only ever remove flags. Removing flags can only ever make the flagged group purer and smaller. Purer is higher precision, smaller is lower recall. There is no setting where both are at their maximum, which is what makes this a genuine tradeoff rather than a tuning problem with a right answer waiting to be found.
Drag the threshold from one end to the other and every pair of values you pass through traces a curve. That curve is the whole tradeoff in one picture.
…precision and recall at your threshold
…precision a coin flip would get
Every point on that line is one threshold. The marker is yours. Sliding along the curve is free, in the sense that it costs no data and no retraining: it is a decision, not a result.
The flat dashed line is the precision you would get by flagging everybody, which is just the share of people who have the condition. Any curve that sags to it has stopped being useful, and on lopsided data that baseline is low enough that a bad model can look impressive against nothing.
05
F1, and the average that does not work
Two numbers are harder to rank than one, so people want them combined. The obvious way to combine them is to take the average, and the obvious way is wrong in a way that is easiest to see by breaking it on purpose.
Set precision to 1.00 and recall to 0.00 with the sliders below. That is a real model, and a famously useless one: it flags exactly one person, it happens to be right about them, and it misses every other case in the dataset.
The surface is F1 across every combination. It falls away to nothing along both edges, which is the behaviour the plain average refuses to have.
…the plain average
…F1
…
F1 = 2 × (precision × recall) ÷ (precision + recall) // the harmonic mean
The plain average says 0.5 for a model that finds nothing. F1 says 0. That contrast is the entire argument for the harmonic mean and it needs no algebra: the harmonic mean is dragged down by the smaller of its two inputs, so it cannot be rescued by the larger one. A model has to be decent at both to score at all.
Then the honest part, which most explanations skip. F1 assumes precision and recall matter equally. In this running example they plainly do not: a false alarm is an unnecessary appointment and a missed case is an undetected condition, and treating those as interchangeable is exactly the mistake step 2b was about. F1 has quietly reintroduced it, one level up.
The knob for that is F-beta, where beta says how many times more you care about recall than precision. Beta of 2 weights recall four times as heavily and is the usual choice for screening. Beta of 0.5 leans towards precision, for when a false alarm is the expensive one, such as a fraud flag that freezes somebody's account. F1 is simply F-beta with beta set to 1, and that setting is a decision somebody made, not a default that came from the mathematics.
06
Thresholds, ROC, and where the three classifiers differ
Look back at what every step since the matrix has depended on. The four cells needed a threshold. Precision and recall needed a threshold. The precision-recall curve was a threshold being dragged. None of it is possible unless the model hands you a probability rather than a verdict.
This is where the three classifiers stop being interchangeable, and it has nothing to do with which is more accurate. They differ in how many different probabilities they are capable of saying. That count is the number of usable thresholds you have, and it is decided by the mechanism, not by the data.
6a How many answers can each one give?
- Logistic regression puts a weighted sum through a sigmoid, and a sum of continuous numbers is a continuous number. Every person gets their own probability, so with … held-out people there are … distinct answers and just as many places to put a threshold. The dial is a real dial.
- KNN counts how many of the k neighbours are positive and divides by k. With k = 5 the only answers that exist are 0, 0.2, 0.4, 0.6, 0.8 and 1. Six values, whatever the dataset, whatever the person. Move the threshold from 0.61 to 0.79 and nothing happens at all, because no person is sitting in that gap. The only way to buy resolution is to raise k.
- A decision tree reports the mix of training rows in whichever leaf you land in. Grow it deep enough and every leaf ends up holding rows of a single class, so the mix is 0 or 1 and nothing else. The model has opinions but no doubt, and there is nothing left to threshold.
6b The ROC curve, and three mechanisms drawing it
Sweep the threshold from one end to the other and plot, for each setting, the share of real cases caught against the share of healthy people falsely flagged. That is the ROC curve. Up and to the left is good: catching everybody while alarming nobody. The diagonal is what a coin flip gets.
All three curves below come out of models fitted live on the same held-out people. The shapes are not drawn, they are consequences.
logistic regression
K-nearest neighbours
decision tree
a coin flip
Up the side is recall, the share of real cases caught. Along the bottom is the share of healthy people falsely flagged. Every dot is one place a threshold could go, so counting the dots counts the resolution of the model.
…AUC, logistic regression
…AUC, KNN
…AUC, tree
…answers logistic regression has
…answers KNN has
…answers the tree has
…
Three things to do to those sliders.
- Leave k at 5 and count the corners. The KNN curve is a staircase with six steps, one per possible answer, and between the corners there is nothing. Now raise k. The steps multiply, the staircase smooths into something that looks like a curve, and the AUC usually improves as a side effect. You are not making KNN smarter, you are giving it more things it is allowed to say.
- Take the tree to depth 16. The curve stops being a curve. It becomes two straight segments meeting at a single corner, because the model only ever emits 0 or 1 and a threshold anywhere between them does exactly the same thing. One operating point, take it or leave it. Many people have met this in the wild as "why is my tree's AUC so bad" and never found out that the answer had nothing to do with how well it classifies.
- Now walk the depth back down. The curve opens up as leaves start holding mixed rows again, and it peaks somewhere in the middle. Keep going to depth 1 and it collapses a second time, for the opposite reason: a stump has two leaves, so it has two answers. What a tree can say is limited by how many of its leaves hold genuinely mixed rows, and that is squeezed from both ends.
AUC is the area under the curve, and it summarises every threshold at once. Read it as the probability that the model gives a randomly chosen positive person a higher score than a randomly chosen negative one. 0.5 is a coin flip and 1.0 is perfect ordering. Because it never commits to a threshold, it is the fairer number for comparing models that nobody has tuned yet, which is exactly the situation you are in when you are still choosing between them.
It is also the number to stop using the moment you have chosen. AUC averages over thresholds you would never ship, including the ones that flag ninety percent of a healthy population. Once the model is picked, go back to the matrix and set the threshold on purpose.
And notice the trap this whole step has been walking you towards. A deep tree can have a poor AUC and a perfectly good accuracy at the same time, on the same predictions. Neither number is lying. They are answering different questions, and the tree's answer is bad only to the question about ranking, which is the question you are asking when you plan to tune a threshold later.
01
The three, side by side
One single decision tree, not a forest and not boosting. Plain logistic regression, not a neural network. The extensions are real and they are the next section, but they are not what you are choosing between when you are choosing between these three.
1a The table
Thirteen rows in three groups: how each one thinks, what each one needs from your data, and what each one costs you.
| K-nearest neighbours | Decision tree | Logistic regression | |
|---|---|---|---|
| Group 1 · How they think | |||
| Mental modelin one line | No model. It keeps the training set and takes a majority vote of the k closest examples. | A learned sequence of yes or no questions, one feature at a time, ending at a leaf. | A weighted sum of the features, squashed through a sigmoid into a probability. |
| Shape of the boundary | Arbitrary and local. Follows the data anywhere, and jagged at small k. | Axis-aligned rectangles only. A diagonal has to be built as a staircase. | One straight line. Curves only if you build the features by hand. |
| What "why did it predict that" looks like | The neighbouring examples themselves. Often the most convincing to a non-specialist. | A rule path from root to leaf. The only one a stakeholder can follow end to end. | Signed coefficients and odds ratios. Global and stable, but scale the features first. |
| What it assumes about your data | Nearby points share a label, and every feature counts equally unless you weight them. | The classes can be separated by thresholds on individual features. | The log-odds are roughly linear in the features. |
| Group 2 · What it needs from your data | |||
| Feature scaling | Mandatory. Without it the widest-ranging column becomes the only column. | Not needed. A split only cares about the order of the values. | Needed in practice: convergence, comparable coefficients, and the penalty. |
| Categorical features | Awkward. Distance between categories is undefined, and one-hot adds columns it hates. | A natural fit, though scikit-learn's tree still wants them encoded as numbers. | One-hot works well. Watch for collinearity from a redundant dummy. |
| Tolerance for many columns | Degrades badly. Distances concentrate and "nearest" stops picking anybody out. The curse of dimensionality. | Moderate. Fragile once columns grow relative to rows. | Excellent. Still the standard baseline for wide sparse text. |
| Data size sweet spot | Small to medium. Prediction cost grows with the training set. | Small to large. | Small to very large. Trains on millions of rows with SGD. |
| Outliers and noisy labels | Sensitive at k = 1, much steadier as k grows, since a wrong label gets outvoted. | Will isolate one noisy row into its own pure leaf. Prune, or set a minimum leaf size. | Fairly robust, but a perfectly separable outlier sends coefficients to infinity. |
| Group 3 · What you pay | |||
| Training cost against prediction cost | Nothing to train, expensive to predict. Every query scans the whole training set. | Moderate to train, near-instant to predict. | Moderate to train, instant to predict. One dot product. |
| Main overfitting knob | k. Small k is high variance, large k is high bias. | max_depth, min_samples_leaf, and pruning. |
Regularisation strength, and the choice between L1 and L2. |
| Probability qualitysee evaluation, step 6 | Coarse. At k = 5 the only possible outputs are 0, 0.2, 0.4, 0.6, 0.8 and 1. | Overconfident. Pure leaves report a flat 0 or 1, so there is nothing to threshold. | Calibrated by construction. The winner whenever you need a number to threshold on. |
| The price you pay | You pay at prediction time forever, and the training set ships with the model. | Instability. Change a few rows and you can get a visibly different tree. | Expressiveness. Any nonlinearity has to be built by hand. |
1b The two rows that are missing on purpose
There is no accuracy row, and there is not going to be one. No honest general answer exists. Which of the three wins on a given dataset depends entirely on that dataset, and anybody who tells you otherwise is describing their last project rather than yours. All three are cheap enough to simply fit and find out, and that afternoon is worth more than any table row could be. If you want one prior to start from: on small tabular data with a handful of informative columns they usually land within a couple of points of each other, and the rows above are what decides it.
There is no "handles missing values" row either. Whether a tree copes with a NaN is a property of the library and its version rather than of the algorithm, and the commonly repeated claim about it goes out of date quietly. If it matters to you, check the behaviour of the version you have installed and write down which version you checked, because the answer you find on a forum is probably about a different one.
1c Three things the table cannot say in one line
Scaling is not a nicety for KNN, it is the single most common way beginners break it. Glucose runs to about 200 and BMI to about 40. Left unscaled, a difference of five glucose points weighs the same as five BMI points, which is most of the useful range of BMI against a rounding error in glucose. The model is not being told that glucose matters more. It is being told that BMI does not exist. Nothing warns you, the code runs, and the accuracy is merely mediocre rather than obviously broken.
KNN inverts the usual cost pattern, and that is what rules it out in production. Every other model on this site is expensive once and cheap forever. KNN is free to fit and then charges you on every single request, for the life of the service. A model that takes an afternoon to train and answers in a microsecond is almost always the right trade against one that trains instantly and needs 200 ms per call. This, far more often than accuracy, is why KNN does not make it into a serving path.
A tree's readability is a real advantage and it is fragile. The rule path is genuinely legible at depth 4. At depth 15 it is a legal document, and the thing you were buying with a single tree has quietly gone. If readability is the reason you chose it, the depth limit is not a tuning parameter, it is the requirement.
TermsThe words the table leans on
- One-hot encoding. Turning a column of categories into one yes-or-no column per category, because most models can only do arithmetic. A colour column becomes three columns: is-red, is-green, is-blue.
- Collinearity. Two columns that carry the same information, so the model cannot tell which one earned the credit and splits it between them arbitrarily. One-hot causes it by accident: if you know is-red and is-green, you already know is-blue, so the third column is redundant and the usual fix is to drop one.
- The curse of dimensionality. Every column you add gives the space another direction to be empty in. Past roughly twenty columns, the nearest point and the farthest point are almost the same distance away, and any method built on "nearest" stops working. KNN §4a demonstrates it live.
- Calibration. Whether a stated probability means what it says. If you gather every case a model called 30 percent and about 30 percent of them turn out positive, it is calibrated. Logistic regression is calibrated by construction because that is precisely the quantity its loss is fitted to.
- L1 and L2. Two ways of penalising large coefficients. L2 shrinks them all smoothly. L1 pushes some to exactly zero, which drops those features entirely and gives you feature selection as a side effect.
- SGD. Stochastic gradient descent: the same downhill walk from linear regression, but each step is computed on a small random batch of rows rather than all of them. It is what lets a linear model train on data too large to hold in memory.
- AUC. The area under the ROC curve, defined in evaluation step 6. Read it as the chance that the model scores a random positive case above a random negative one.
02
Where each one leads next
Each of the three is the first step of a family. Knowing what the next step buys, and what it costs, is most of what separates picking a model from reaching for the one everybody mentions.
2a The tree, to a random forest or gradient boosting
Row 13 said a single tree is unstable: change a few rows and you get a visibly different tree. An ensemble fixes that directly, by growing many trees and combining them. The decision tree chapter ends by building both of these live, so the paragraphs below are the summary and that is the demonstration. A random forest grows them in parallel on random subsets of the rows and columns and averages the votes. Gradient boosting grows them one at a time, each new tree fitted to the errors the previous ones left behind. Both are more stable than one tree, both usually win on accuracy, and gradient boosting on tabular data is the reason it is the answer an assistant reaches for first.
Frame it as a trade, not a free upgrade. The single tree had exactly one thing the ensemble cannot have: you could read it. Three hundred trees voting is not a rule path, it is a number, and the feature importances that come out of it are a summary rather than an explanation. If you chose a tree because somebody has to be able to follow the decision, the ensemble takes that away and gives you accuracy in exchange. Sometimes that is the right trade. It is never a neutral one.
2b Logistic regression, to a neural network
The relationship here is exact, and it is the most useful intuition on this page: logistic regression is a neural network with no hidden layers and a sigmoid on the output. Not similar to one. That is what it is. Take the diagram of a small network, delete every layer in the middle, and what remains is the weighted sum and the squash from the logistic regression chapter.
Which tells you precisely what hidden layers buy. Row 2 said logistic regression can only draw a straight boundary, and that curves have to be built by hand as extra features. A hidden layer builds those features for you, by learning combinations of the inputs rather than being handed them. That is the entire upgrade: not more cleverness at the end, but learned features in the middle. Everything else, the loss, the gradient descent, the learning rate, is the machinery you already met.
One clarification, because it trips people constantly. With two classes the output is a single sigmoid. With more than two it is a softmax, which produces one number per class and forces them to add up to 1. Sigmoid squashes one number into a probability; softmax turns several numbers into a set of probabilities that compete with each other. Calling a multiclass output layer a sigmoid is a small error that reliably signals somebody has not looked.
2c KNN, to weighted distances and approximate search
Three directions. The first is to stop treating every neighbour and every feature equally: weight the votes by distance so a close neighbour counts for more, and weight the features so the columns that matter dominate the distance. The second is to stop scanning everything: a KD-tree or a ball-tree indexes the training set so whole branches can be skipped once they are provably too far away, which is free accuracy and a real speedup, and which quietly stops helping past about twenty columns for the reasons in row 7.
The third is where the algorithm ends up somewhere nobody expects. Give up on exactness and use approximate nearest neighbour search, which returns almost the right neighbours in a tiny fraction of the time. Those libraries are what vector databases are built out of, and vector search is the retrieval half of retrieval-augmented generation. When a chatbot looks something up in your documents before answering, the lookup is nearest neighbours on embedding vectors. The oldest and simplest algorithm on this site is running inside the newest tooling on your machine, with a learned distance and an approximate index bolted on, and the vote at the end unchanged.
03
Prompting AI
You now know enough to catch an assistant getting this wrong, and that is the skill worth carrying away from the page. None of what follows is general prompting advice. All of it is specific to picking and scoring a classifier.
3a Why the default answer is always the same
Ask any assistant "which classifier should I use" with no other context and you will get gradient boosting, or a random forest, nearly every time. That is not a bug and it is not laziness.
Think about where the text came from. The internet's writing about classifier choice is dominated by competition writeups and blog posts, and those have a shape: the winning model gets named, the leaderboard score gets quoted, and the constraints get left out, because in a competition there mostly are none. Nobody had a latency budget. Nobody had to explain a decision to a regulator. Nobody was shipping a frozen model into a clinic. So the training text says "boosting won" over and over, and never says "and it was ruled out on the next four projects for reasons that had nothing to do with accuracy".
It is a reasonable prior and a bad recommendation, and the fix is not a cleverer way of phrasing the question. The fix is supplying the constraints that make a generic answer impossible. If your prompt does not narrow the field, the reply cannot.
3b What has to be in the prompt
Seven things. Each one maps to a row of the table above or a step of the evaluation section, which is what makes that table worth having rather than trivia. Switch them on and watch the prompt rewrite itself.
The prompt as it stands
Which classifier should I use?
What you would get back
Gradient boosting
Nothing in that question narrows the field, so the answer comes from the prior: whatever wins competitions. It is not wrong so much as unaccountable, because there is no constraint in the prompt for it to be wrong about.
Constraints you actually have
Switch on a clinician has to justify it and we need a score, not a label together and the recommendation moves from gradient boosting to logistic regression. Nothing about the data changed. The constraints did the work, and they were true the whole time. They were simply not in the prompt.
3c What to check in the answer
Eight failure modes, and every one of them connects back to something earlier on this page. These are worth reading once and then recognising forever, because they show up in generated code far more often than in generated prose.
It recommends KNN for text, or for anything with a few hundred columns
- Myth
- More features give the distance more to work with.
- Reality
- Past about twenty columns, distances concentrate and the nearest point stops being meaningfully nearer than the farthest. On TF-IDF text this is not a degradation, it is a failure.
- Proof
- Table row 7, and the live demonstration in KNN §4a, where the ratio between the farthest and nearest neighbour falls from about thirty to under two.
The pipeline scales the whole dataset before splitting it
- Myth
- Scaling is preprocessing, so it can happen once at the top.
- Reality
- The scaler learns the mean and the spread. Fit it on everything and it has already read the rows it is about to be scored on, and the reported number comes back flattering. This is leakage, and it is one of the most common defects in generated code.
- Proof
- Split first, then put every step that learns something inside a
Pipeline. The rule is that nothing learns anything from a row it will later be scored on.
It skips scaling entirely for KNN or for regularised logistic regression
- Myth
- If it were needed, something would complain.
- Reality
- Nothing complains. The code runs and the model is quietly broken rather than obviously broken: the widest-ranging column becomes the whole distance, and the penalty punishes coefficients for the units they happen to be measured in.
- Proof
- Table row 5, and §1c above.
It reports accuracy on imbalanced data
- Myth
- 93 percent accuracy is a good result.
- Reality
- On a population where one in sixteen has the condition, predicting "nobody has it" scores about 94 percent. The number tells you the class balance and almost nothing else.
- Proof
- Evaluation step 2a. Ask for the confusion matrix and for what the majority class alone would have scored.
It calls train_test_split without stratify
- Myth
- A random split is a fair split.
- Reality
- It is fair on average and unreliable on a rare class. With 40 positives out of 600 an unlucky split can leave you scoring on eight of them, and the resulting metric swings by tens of points between runs for no reason connected to the model.
- Proof
- Pass
stratify=y, which keeps the class proportions in both halves. Then compare the run-to-run variation.
It calls predict when the problem needed predict_proba
- Myth
- The model returns the answer.
- Reality
predicthas already thrown the probability away and applied a threshold of 0.5 without mentioning it. On screening data 0.5 is almost never the threshold you want, and it is a decision about consequences that the library made on your behalf.- Proof
- Evaluation steps 3 and 6. Ask for probabilities and a stated threshold, with a reason for the number.
It says feature_importances_ tells you the direction of an effect
- Myth
- A high importance means the feature pushes the prediction up.
- Reality
- Importance is magnitude only. It says a feature was used a lot, not which way it pushed. Impurity-based importance is also biased towards columns with many distinct values, so a high-cardinality identifier can top the chart while carrying nothing.
- Proof
- Coefficients are the ones with a sign, and only after scaling. If you need direction from a tree, reach for permutation importance or SHAP rather than the attribute.
It describes the multiclass output layer as a sigmoid
- Myth
- Sigmoid is the classification activation.
- Reality
- Sigmoid squashes one number into one probability, which is right for two classes. More than two needs softmax, which produces one number per class and forces them to sum to 1 so the classes compete.
- Proof
- §2b above. It is a small error, and a reliable signal that the answer was assembled rather than checked.
3d Three questions that improve any answer
- "What would make this the wrong choice?" Forces out the constraint the answer skipped. An assistant that cannot name a situation where its recommendation fails has not made a recommendation, it has made a guess.
- "What are you assuming about my data?" Surfaces row 4 for whatever it just suggested. Half the time the assumption is one you can check in a minute and one you would have found out about in a week.
- "Show me the confusion matrix, not the accuracy." Reframes evaluation before a line of code gets written, which is the only point at which reframing it is cheap.
3e What not to hand over
Three decisions on this page are not technical, and an assistant has no basis for making any of them.
Which metric you optimise. Which of the two mistakes is worse. Where the threshold goes. Every one of those is a claim about consequences in the world, and the model does not know what a missed case costs you, who absorbs a false alarm, or which of the two your organisation will actually be held to account for. It will answer if you ask. The answer will be a plausible sentence assembled from other people's situations.
That is what the whole evaluation section was for. It was not teaching you formulas, all six of which fit on one line and none of which you need to memorise. It was building the judgment required to make those three calls yourself, in front of a matrix, with the numbers on the screen. An assistant can write the pipeline in twenty seconds and it will be good code. It cannot decide what counts as a good prediction.
← All algorithms
Shared · all of supervised learning
Bias and variance
The problem it solves: every chapter on this site ends with a dial. k in K-nearest neighbours, the depth of a tree, the penalty on a regression. They look like four unrelated settings and they are not. They are the same dial, and this page is what it does. Turn it one way and the model is too simple to be right. Turn it the other and it is too eager, and learns the noise along with the pattern. There is no setting where both problems go away, which is why it is a trade rather than a bug.
What to read first: any one chapter. This page names k, depth and lambda and assumes you have turned at least one of them and seen a score get worse. If you have read K-nearest neighbours, §3a there is this whole page compressed into a single integer, and module 4 comes back to it.
Turns up as
choosing kpruning a treehow much penaltyhow many featureswhen to stop trainingwhether more data would help
Running example for the whole page: two solar panels on a roof, logging what they make across the day. There is a true curve, the output the panels are physically capable of at each moment, which rises through the morning, peaks a little after one, and falls away. Nobody can see it. What the logger records is that curve plus whatever the weather did, because a cloud passing at half past eleven takes a bite out of the reading and no model that has ever been written can predict a cloud. Fifty readings a day, and the noise is about 55 W on a peak of around 810 W.
Why this example and not a real one: everything on this page needs the true curve, and on real data you never have it. Here it is generated, so the page can do the one thing a textbook can only assert: measure bias and variance directly, by fitting the same model to forty different days and looking at where the answers land. Module 2c is about what you do instead when the truth is not available, which is always.
01
Intuition: fit it again on a different day
Here is the move that makes the whole subject visible, and almost nobody does it, because in real life you only get one dataset and you fit your model once.
Fit it forty times. Forty different days, fifty readings each, same panels, same true curve, different clouds. Then draw all forty fitted curves on top of one another and look at what the model did.
1a Forty days, forty curves
the true curve, which nobody can see
the forty fitted models
their average
one day's readings
…how far the average sits from the truth
…how far the forty sit from each other
…
The slider is polynomial degree, the simplest flexibility knob there is: 1 is a straight line, 2 a single curve, 9 a curve that can change its mind nine times across the day. Module 4 shows that k, depth and lambda all behave exactly like this one.
Walk the slider from 1 to 9 slowly and watch the two failures arrive one after the other.
- At 1, all forty curves sit on top of each other. Change the day, change the clouds, change every single reading, and you get back more or less the same straight line. The model is completely stable and completely wrong: it is a line, the truth is a hump, and no amount of data will ever make a line into a hump. That is bias. It is the error you would still have if you had infinite data.
- At 9, the forty curves are all over the plot. Each one chases its own day's clouds, so no two agree, and most of them are doing something ridiculous in the early morning where the readings happen to be thin. On average they are about right, which is the frustrating part. Individually they are anything at all. That is variance. It is how much the answer depends on which day you happened to get.
You only ever get one of those forty curves. You do not get to see the spread, and nothing in the output tells you it exists.
1b One moment, forty answers
Pick a single moment in the day and ask all forty models what the panels are making. Forty numbers come back. Lay them on a line next to the truth and the two errors stop being metaphors.
Drag along the plot to move the moment. The slider above still controls how flexible the models are.
bias…
spread…
…
Bias is the gap between the truth and the middle of the cluster. Variance is the width of the cluster. They are separate quantities and a model can be bad at either one without being bad at the other, which is the entire reason they get separate names. Both bars are drawn against the same 320 W scale, so their lengths are comparable.
Set the model to degree 1 and drag through the day. The cluster stays tight and marches a long way from the truth, badly wrong in different directions at different hours. Set it to 9 and the cluster balloons, especially near the ends of the day where the readings run out. Somewhere in between there is a setting where the cluster is both near the truth and not too wide, and finding it is the whole job.
CuriousCould you just use the average of the forty?
You have probably already noticed that the amber average curve is better than almost every individual curve it is made of, especially at high degree. That is not a trick of the drawing. Averaging leaves the bias exactly where it was and divides the variance by roughly the number of models, so if variance is your problem, averaging is close to a free fix.
And yes, you can actually do it. You do not have forty days, but you can fake them by resampling the one day you have, fitting a model to each resample, and averaging the results. That is bagging, and a random forest is bagging applied to decision trees, with an extra twist that makes the individual trees disagree more so the averaging has more to work with. The tree chapter builds one at the end, and you can watch the averaging happen.
Which explains something that looks strange from the outside. A single deep tree is one of the highest-variance models in common use, and a forest of five hundred deep trees is one of the most reliable. Nothing was done about the bias. Five hundred noisy answers were averaged, and the noise mostly cancelled.
The price is the one named on the comparison page: you can read a single tree and you cannot read five hundred.
02
Where the error actually comes from
Take the squared error of a model, averaged over every day it could have been fitted on, and it splits into exactly three pieces. This is not a rule of thumb. It is an identity, and it is the reason the two words get used together.
expected error = bias² + variance + noise // and nothing else
- Bias² is how far the average model is from the truth. Being too simple to represent the shape.
- Variance is how far the individual models are from their own average. Being too sensitive to which day you got.
- Noise is the cloud. It belongs to the world rather than to the model, no model can touch it, and it does not move when you turn the dial.
2a The three pieces, measured
Because the truth is known here, all three can be computed rather than argued about. The chart stacks them, so the height of the whole stack is the error you would actually get.
bias²
variance
noise, the cloud
…bias², at this setting
…variance
…noise, always
…
Degree 1 runs off the top of the chart. Its bias² is roughly … times the whole stack at the best setting, and drawing it to scale would flatten everything else into a line. That it does not fit is the point.
Read it left to right and the handover is obvious. On the left the stack is nearly all coral: the model is wrong in a way that has nothing to do with luck. Coral collapses fast, the total comes down with it, and for a few settings in the middle the stack is almost all noise, which is as good as it gets. Then violet starts climbing and the total goes back up, this time for the opposite reason.
The grey band never moves. That is the part of this problem that is not about modelling at all. Fifty-five watts of cloud is in the data whatever you fit, and a model that reports an error below it on held-out readings has not beaten the weather, it has made a mistake somewhere in how it was scored.
2b The floor, and what it means to be near it
The best achievable error here is the noise on its own. In the units the panels are measured in, that is about … W of typical miss, forever, on the best model anybody will ever fit to this data. At the best setting on the chart the model gets to about … W.
That gap, the small difference between the floor and the best you managed, is the total of bias and variance you did not remove. It is the only part worth working on, and knowing how big it is stops you spending three weeks chasing an improvement that was never available.
Most people have never worked out their noise floor and have no idea whether they are three points away from it or thirty.
CuriousWhy is bias squared but variance is not?
Because they are already the same kind of thing, and writing it the other way would double-count.
Bias is a distance: the average prediction is 180 W below the truth. Squaring it turns a distance into an area, which is what has to happen for it to be added to an error that is itself squared. Variance is already an average of squared distances by definition, so it arrives pre-squared and needs nothing done to it.
If the notation annoys you, the honest fix is to take the square root of the whole sum at the end, which puts everything back into watts. That is exactly what RMSE is, and it is why the rest of this page quotes errors in watts rather than in watts squared. The identity only holds before the square root, which is the one inconvenient fact about it: you cannot add up three RMSEs and get the total.
CuriousCan you measure any of this on real data?
No, and this is the most important sentence on the page.
Splitting the error into bias and variance needs two things you will never have outside a simulation. It needs the true curve, to measure bias against, and if you had that you would not be fitting a model. And it needs many independent datasets, to measure variance across, and if you had those you would have concatenated them and used the lot.
So bias and variance are not a diagnostic. They are a vocabulary: a way of naming the two ways a model can be wrong, so that when you see a symptom you know which of the two it points at, and therefore which fixes are even worth trying.
What you can measure is the gap between training error and held-out error, which is a shadow of variance, and the height of the held-out error, which is a shadow of bias plus the noise floor. Two numbers you can get from any dataset in about a minute. Module 3 is entirely about reading them, and every practical instinct on this page comes from that pair rather than from the identity above.
03
Reading the gap, which is the part you can actually do
Two numbers, available on any dataset. Training error, scored on the readings the model was fitted to. Held-out error, scored on readings it has never seen. Everything practical about this page comes from looking at the two together, and almost nothing comes from looking at either one alone.
3a The two curves against flexibility
training error
held-out error
the gap
the noise floor
…training error, W
…held-out error, W
…the gap
…
This slider and the one in module 1 are the same dial: move either and every chart on this page follows, so you can scroll up and see what the forty curves looked like at any setting here.
Three readings, and between them they cover almost every situation you will meet.
- Both errors high, and close together. The model is not even fitting the data it was given. It is too simple, and the honest description is that it has not learned yet. Underfitting. More data will not help, because the model is already failing on data it can see.
- Training error low, held-out error much higher. The model has fitted the clouds. It looks superb on the readings it memorised and falls apart on new ones. Overfitting, and the width of the gap is your best available look at variance.
- Both errors low, and close to the floor. Stop. You are done, and the remaining error is weather.
The trap is quoting only the first number. Training error falls forever as you add flexibility, all the way to zero if you let it, so it can be driven to any value you like and it measures nothing except how hard you tried. On its own it is not a weak signal, it is an anti-signal: the lower it goes without the held-out number following, the worse things are getting.
3b The diagnostic that tells you what to do next
The chart above needs you to refit at every setting. There is a second chart that answers a different and more useful question, and it is the single most valuable thing on this page: hold the model fixed and vary how much data it gets. These are learning curves, and they tell you whether more data would help before you go and collect it.
training error
held-out error
the noise floor
…where the two curves settle
…held-out minus training, at 700
…
Both curves flat and high, sitting together. Bias. Collecting more data is money spent on nothing, and the model has to change instead. The gap can even come out slightly negative here, which is not a bug: a model this wrong misses the readings it was fitted to by about as much as it misses new ones.
A gap that is still closing as you go right. Variance. More data is genuinely the fix, and the curve tells you roughly how much more before it stops paying.
This is the chart to draw before anybody is asked to go and label another ten thousand rows. It costs one afternoon, it uses data you already have, and it answers a question that otherwise gets settled by whoever is most confident in the meeting.
CuriousWhy does training error keep falling, even when the model is getting worse?
Because the model is being scored on the answer sheet it was fitted to, and every extra degree of freedom is another chance to bend towards a point it has already seen.
Take it to the extreme and it stops being mysterious. Give a polynomial as many coefficients as there are readings and it will pass exactly through every single one of them, training error zero, a perfect score. It has not learned anything about sunlight. It has memorised fifty numbers, including the cloud at half past eleven, and it will say something absurd about any moment in between.
K-nearest neighbours does the same thing more honestly at k = 1, where the nearest reading to a stored reading is itself and the training error is exactly zero by construction. KNN §3a has that as a live demonstration, and it is the cleanest example on the site of a perfect score meaning nothing at all.
04
The same dial, in every chapter
Everything so far used polynomial degree, because it is the easiest flexibility knob to picture. Now the payoff. Here are four completely different models on the same solar readings, each with its own knob, each fitted live. They have nothing mechanically in common. A polynomial is a weighted sum, KNN stores rows and averages the nearest ones, a tree cuts the day into blocks, ridge is a polynomial with a penalty bolted on.
Look at the four charts before you read another word.
Polynomial degreemore flexible to the right →
KNN: k← more flexible to the left
Tree: maximum depthmore flexible to the right →
Ridge: penalty λ← more flexible to the left
training error
held-out error
the best setting on each
the noise floor
Four models, four knobs, one shape. In every panel the blue line walks steadily downhill as the model is allowed more freedom, and in every panel the teal line comes down and bottoms out. Three of the four then climb back up. The tree flattens instead, and for an honest reason: on fifty readings it runs out of points to split before it runs out of depth, so past its best setting the extra depth buys nothing and costs nothing. The best settings land at ….
Two of those dials run backwards, and this is the most common way people get confused about the whole subject. Degree and depth measure flexibility directly: bigger is more flexible. k and lambda measure restraint: bigger is simpler. A large k averages over half the day, and a large penalty crushes the coefficients towards zero. So the U in those two panels is mirrored, and "turn it up" means opposite things depending on which chapter you happen to be reading.
Worth naming the underlying quantity, since the knobs are all proxies for it. What actually varies is how many genuinely independent things the model is able to say. A degree-9 polynomial has ten coefficients. A KNN regressor with 50 readings and k = 2 has about 25 independent local averages. A depth-6 tree has up to 64 leaves, each free to take its own value. Flexibility is roughly a count of the answers a model can give, and every knob on this page is a way of setting that count.
Which is also why the evaluation page's step 6 turned out to be about the same quantity from a different direction. There, counting the distinct probabilities a classifier can emit told you the resolution of its ROC curve. Here, counting the distinct things a regressor can say tells you where it sits on this U. Same count, two consequences.
KNN, §3a
k = 1 scores a perfect 100 percent on the data it stored, and worse on unseen fruit than any larger k short of the extreme. The gap between those two numbers is this page's variance, in one integer.
Linear regression, §4b
Lambda, the penalty. Slide it up and the model stops chasing its own tail. That is variance being bought down at the cost of bias, priced explicitly.
Logistic regression, §4c
Perfect separation sends the weights to infinity. An unbounded model with nothing holding it back is the far right of this chart with the axis removed.
Evaluation, step 6
A tree grown until its leaves are pure. Every leaf is a memorised block, which is why its probabilities collapse and its ROC curve does too.
CuriousThen why do enormous neural networks work at all?
A fair question, and the U above genuinely does not describe them. A modern network can have more parameters than there are examples in its training set, which is the far right of every chart on this page, and by the argument so far it should be useless. It is not.
What people find when they keep pushing right is double descent. Held-out error rises exactly as promised, peaks around the point where the model has just enough capacity to fit the training data perfectly, and then, as capacity keeps growing past that point, falls again, sometimes below the first dip. The classic U is real and it is the first half of a wider curve nobody drew until recently.
The rough explanation is that once a model can interpolate the training data in many different ways, the fitting procedure gets to choose among them, and gradient descent has a quiet preference for the smoothest of those solutions. The capacity stops being spent on contortion and starts being spent on having somewhere gentle to land. That preference is itself a form of restraint, so it is not that regularisation stopped mattering, it is that it moved out of the penalty term and into the optimiser.
Two things this does not change. The noise floor is still the floor. And on the small tabular data every chapter of this site uses, you are firmly on the first descent, where the U is the whole story and pushing right just makes things worse.
05
What actually fixes each, and what does not
The reason the vocabulary earns its place: the two problems have almost opposite cures, so applying the wrong one is not merely wasteful, it makes things worse.
5a More data, and the thing it cannot touch
The most expensive fix in the industry, and the most often applied without checking. Here it is on three models, all fitted to the same panels.
degree 1, too simple
degree 5, about right
degree 9, too complex
the noise floor
…degree 1, held-out W
…degree 5
…degree 9
…
Drag it to the right end. Two of the three lines walk down onto the floor and stop there. The coral line does not move, because a straight line is a straight line whether you fit it to twelve readings or six hundred.
More data buys down variance and does nothing whatsoever about bias. The degree-9 model goes from hopeless to essentially optimal, because all that was ever wrong with it was that it did not have enough evidence to pin down its own flexibility. The degree-1 model is exactly as wrong at six hundred readings as at twelve, because what is wrong with it is not a shortage of evidence, it is that it is the wrong shape.
Which gives the rule worth carrying out of this page: before commissioning more data, draw the learning curve. If the two lines have already met, more data is a purchase with a guaranteed return of zero, and the honest thing to tell whoever is paying is that the model needs changing, not feeding.
5b The rest of the toolkit, sorted by what it treats
| If the problem is | Reach for | Because | What it costs |
|---|---|---|---|
| Too much bias · both errors high and sitting together | |||
| Underfitting | A more flexible model, or the same one with the dial turned up | The shape it can draw does not include the shape of the answer. | Variance, immediately. You are walking rightwards along the U on purpose. |
| Underfitting | Better features | The cheapest flexibility there is. A squared term, an interaction, a ratio somebody in the domain already knows matters. | Your time, and the risk of inventing features that only work on this sample. |
| Underfitting | Less penalty, less pruning, train for longer | The restraint you added is now the binding constraint rather than the noise. | Nothing, until it becomes the other problem. |
| Too much variance · training error low, held-out error far above it | |||
| Overfitting | More data | The only fix on this list that costs no accuracy anywhere. §5a. | Money and time, and it is often simply unavailable. |
| Overfitting | Regularisation: a penalty, a depth limit, a larger k | Restraint. It refuses the model the freedom it was using to chase noise. | Bias, deliberately. That is the trade, priced and visible on the dial. |
| Overfitting | Averaging several models: bagging, a random forest | Variance divides by roughly the number of models. Bias stays exactly where it was. | Compute, and the readability of a single model. |
| Overfitting | Fewer features | Every column is a direction the model can wander in, and a column carrying nothing is pure variance. | Bias, if you drop one that mattered. |
| Neither · held-out error sitting on the noise floor | |||
| You are finished | Stop, and go and work out what the error costs | The residual is weather. No model reaches under it and every hour spent trying is an hour lost. | Nothing, and most people never take it. |
Notice how the two halves mirror each other. Nearly every entry under one heading appears under the other with the sign flipped, which is what it means for this to be a trade rather than a problem with a solution.
CuriousHow do you actually choose the setting, without cheating?
Cross-validation, and then one honest number at the end.
Split the training data into five parts. For each candidate setting, fit on four and score on the fifth, rotating until each part has had a turn as the scorer, and average the five. Take the setting with the best average. That is the whole procedure, and it reuses your data five times over without ever scoring a model on a row it was fitted to.
The part people skip: the held-out set you used to choose is no longer held out. Try thirty settings against the same validation split and you will find the one that suits that split, which is overfitting one level up, with the same shape and none of the warning signs. The defence is a third split that gets looked at exactly once, after everything is settled, and whose number is the one you report.
And a rule of thumb that costs nothing: when two settings are within noise of each other, take the simpler one. Five folds of a small dataset produce scores that wobble by more than people expect, and a difference of half a watt between degree 5 and degree 6 is a coin toss. The simpler model will be steadier on data neither of you has seen.
06
Prompting AI: getting a diagnosis rather than a suggestion
This is the topic where an assistant is most likely to be confidently unhelpful, for a specific and predictable reason. Ask it why your model is not working and it will produce a list: try regularisation, try more features, try tuning the hyperparameters, try a different model, try more data. Every item on that list is a real technique. The list is still useless, because half of it treats bias and half of it treats variance, and applying the wrong half makes the model worse.
It produces the list because you gave it a symptom with no measurements attached, and a list is the only honest response to that. The fix is to hand over the two numbers.
"My model isn't performing well. How do I improve it?"
What you typically get back
Six suggestions in a numbered list, ordered by how often they appear in blog posts rather than by anything about your situation. Regularisation and "collect more data" will both be on it, which is notable, because those two treat the same problem and the first one treats it by making the model simpler while the second treats it by leaving the model alone. There is no way to follow both. And "try a more complex model" will be on the same list, which treats the opposite problem.
no training errorno held-out errorno noise floortreats both problems at onceno learning curve
"I am predicting solar output in watts from time of day. 50 readings, one feature, a polynomial of degree 9.
Training RMSE is 50 W. Held-out RMSE is 98 W. Repeated measurements at the same moment vary by about 55 W, so I believe the noise floor is near 55.
Tell me which of bias or variance dominates here and how you know, before you suggest anything. Then give me only the fixes that treat that one, and say what each will cost me on the other. If you think I need more data, tell me what the learning curve would have to look like for that to be true, so I can check before I go and collect it."
What changes
It cannot give you a list any more, because the numbers rule most of it out. Training error is close to the floor and held-out error is nearly double it, so the gap is doing the damage and the diagnosis is variance. That single sentence deletes "more features" and "a more complex model" from the reply before it is written. What is left is regularisation, a lower degree, averaging, or more data, and the last clause makes it check the learning curve rather than assume.
both errors givennoise floor stateddiagnosis before prescriptioncosts namedlearning curve demanded
Three habits, all cheap, all specific to this page.
- Always give both numbers, never one. A held-out score alone cannot be diagnosed and neither can a training score. The pair is the entire signal, and either one on its own invites a guess.
- Say what your noise floor is, or say that you do not know. Without it, nobody can tell whether you are near the limit or miles from it, and an assistant will cheerfully help you chase an improvement that does not exist.
- Ask for the diagnosis first and the fix second. The order matters more here than anywhere else on the site, because a fix proposed before a diagnosis has a roughly even chance of pointing the wrong way.
One thing to watch for in generated code specifically. An assistant asked to "plot the learning curve" will often produce a plot of loss against training epoch, which is a different chart with a similar name, and answers a different question: it tells you when to stop training, not whether more data would help. What §3b needs is error against training set size, which in scikit-learn is learning_curve rather than anything to do with epochs. Both are useful. They are not substitutes, and the phrase "learning curve" is used for both by people who mean only one.
07
Beyond AI: the bottom of the curve is not a decision
Every chart on this page has a lowest point, and it is tempting to treat finding it as the end of the work. It is not, for a reason no amount of cross-validation will surface: the lowest point minimises the average squared error, and almost nobody actually wants that.
Squared error says a miss of 200 W is sixteen times worse than a miss of 50. It says being over and being under are the same. For these panels that might be fine. If the forecast is being used to decide whether to draw from the battery, being 200 W over is a flat battery at nine in the evening and being 200 W under is a slightly conservative afternoon, and those are not the same event at all. The dial has a different best setting under that description, and no chart on this page knows it.
7b The judgment calls no search makes for you
Simple is not a consolation prize
This page frames simplicity as a cost: you accept bias to buy down variance. In a system that has to run for years, simplicity is also a benefit that never appears in any error metric. A simpler model is easier to explain when somebody challenges it, degrades more gracefully when the world shifts under it, and fails in ways somebody can recognise at three in the morning.
A model two watts worse and five times easier to reason about is very often the right choice, and the cross-validation score will never say so. That is a judgment about the organisation that has to live with it, not about the data.
What an error costs, in each direction
The bottom of the U is the bottom under squared error. If over-predicting and under-predicting have different consequences, the setting you want is somewhere else, and finding it means writing the cost down first.
How much of the answer sheet you have spent
Every setting you compare uses up a little of your validation set's honesty. Thirty comparisons against one split, and the winner is partly a winner of that split. Nothing warns you, and the final number is the one you will quote.
Whether tomorrow is drawn from today
All of this assumes the held-out readings come from the same world as the training ones. Panels degrade, trees grow, a neighbour builds an extension. When the world moves, the error rises for a reason that is neither bias nor variance, and this entire page is quiet about it.
Who absorbs the variance
A high-variance model is not equally unreliable for everyone. It is least reliable where the data was thinnest, which is usually wherever the smallest group was. The average error looks fine and the people at the edges carry the whole of it.
The useful way to hold all of this: bias and variance are not scores to optimise. They are the two questions to ask about any model that is not working well enough, in the right order, with two numbers you can get in a minute. Is it failing on the data it can already see, or only on data it cannot? Almost everything you should do next follows from which of those is true, and almost nothing follows from the error on its own.
Four cells. The first is the simulation this page is built on, and you can only run it when you already know the answer, which is never. The other three are the ones you will actually use, and they are short enough that there is no excuse for not running them before deciding what to do about a model.
1
Measuring bias and variance, when you happen to know the truth
modules 1 and 2import numpy as np from sklearn.pipeline import make_pipeline from sklearn.preprocessing import PolynomialFeatures from sklearn.linear_model import LinearRegression def bias_variance(make_model, truth, noise, n=50, draws=200, seed=0): """Only possible in a simulation: it needs the true function, and it needs many independent datasets. If you have either one on a real problem you are not in a position where you need this (§2b).""" rng = np.random.default_rng(seed) grid = np.linspace(5.5, 20.5, 81) preds = np.empty((draws, grid.size)) for d in range(draws): x = rng.uniform(5.4, 20.6, n) # a different day y = truth(x) + rng.normal(0, noise, n) # different clouds preds[d] = make_model().fit(x[:, None], y).predict(grid[:, None]) mean_pred = preds.mean(axis=0) bias2 = ((mean_pred - truth(grid)) ** 2).mean() variance = preds.var(axis=0).mean() return bias2, variance, noise ** 2 for deg in range(1, 10): b, v, e = bias_variance(lambda d=deg: make_pipeline( PolynomialFeatures(d), LinearRegression()), truth, 55) print(f"degree {deg}: bias² {b:8.0f} variance {v:8.0f} total {b+v+e:9.0f}")
preds.var(axis=0) is the whole of variance: the spread of the two hundred answers at each point, with no reference to the truth anywhere in the line. bias2 is the only place the truth appears, which is exactly why you cannot run this on real data.
The three numbers add to the expected squared error. Take the square root of the sum, not of the pieces, or nothing lines up (§2's first aside).
2
The U, without cheating to find it
modules 3a and 4from sklearn.model_selection import validation_curve, train_test_split X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0) degrees = np.arange(1, 10) train_scores, valid_scores = validation_curve( make_pipeline(PolynomialFeatures(), LinearRegression()), X_train, y_train, param_name="polynomialfeatures__degree", param_range=degrees, cv=5, scoring="neg_root_mean_squared_error") # sklearn returns "greater is better", so the sign comes back off train_rmse, valid_rmse = -train_scores.mean(1), -valid_scores.mean(1) for d, t, v in zip(degrees, train_rmse, valid_rmse): print(f"degree {d}: train {t:6.1f} validation {v:6.1f} gap {v-t:6.1f}") best = degrees[valid_rmse.argmin()] # and the test set is opened exactly once, here, after the choice is made
The dial swaps out and nothing else changes. param_name becomes kneighborsregressor__n_neighbors, or decisiontreeregressor__max_depth, or ridge__alpha with a log-spaced range, and you get the other three panels of module 4 from the same four lines.
cv=5 is what keeps it honest. Without it you would be choosing the degree against a single split, and the winner would partly be a winner of that split rather than of the problem.
3
The learning curve, which is the one worth running first
module 3bfrom sklearn.model_selection import learning_curve sizes, train_scores, valid_scores = learning_curve( model, X_train, y_train, train_sizes=np.linspace(0.1, 1.0, 10), cv=5, scoring="neg_root_mean_squared_error") train_rmse, valid_rmse = -train_scores.mean(1), -valid_scores.mean(1) # The whole reading is in the last few rows: where the two columns are # heading, and whether they are still moving towards each other. for n, t, v in zip(sizes, train_rmse, valid_rmse): print(f"n={n:5d} train {t:6.1f} validation {v:6.1f} gap {v-t:6.1f}") # NOT the same thing as loss against epoch, which answers a different # question (when to stop training) and shares the name (§6).
Run this before anybody is asked to collect more data. A gap that is still closing at the right-hand edge means more rows will pay. Two flat lines already touching means they will not, and the money should go on features or on a different model instead.
4
The diagnosis, written down so it stops being a vibe
modules 3 and 5def diagnose(train_rmse, valid_rmse, noise_floor=None, tol=0.15): """Two numbers in, a direction out. Deliberately blunt: the value of this is that it commits before you start trying things.""" gap = valid_rmse - train_rmse wide = gap > tol * valid_rmse if noise_floor and valid_rmse <= noise_floor * (1 + tol): return "done: held-out error is at the noise floor, stop here" if wide and train_rmse < valid_rmse / 2: return "variance: more data, more penalty, less depth, or average several models" if not wide: return "bias: better features or a more flexible model. More data will not help" return "some of both: tune the dial by cross-validation before changing anything else" print(diagnose(train_rmse=50, valid_rmse=98, noise_floor=55)) # variance: more data, more penalty, less depth, or average several models print(diagnose(train_rmse=240, valid_rmse=246, noise_floor=55)) # bias: better features or a more flexible model. More data will not help
Nine lines, and it is not really a piece of engineering. It is a way of forcing the diagnosis to happen before the fixing, which is the habit the whole page exists to install. Writing it down also makes the thresholds arguable: tol is a judgment, and having it as a named argument is better than having it as an instinct.
The noise_floor check goes first on purpose. It is the one branch that tells you to stop working, and it is the one nobody ever reaches, because almost nobody knows their floor.
← All algorithms
Supervised · classification
Decision trees
The problem it solves: you need a decision, and you need to be able to say why. A decision tree is a sequence of yes or no questions, each one asked of a single number or category, ending in a verdict. Nothing else on this site produces an explanation a person can read out loud and argue with. The interesting part is not how you follow a tree, which takes about ten seconds to learn. It is how the tree decides what to ask.
What to read first: nothing. This chapter stands on its own, and it leans on the two shared pages rather than repeating them. Evaluation and comparison is the scoring machinery module 4 hands off to, and it already fits a tree live. Bias and variance is the general shape of what the depth dial does, and module 5c here is one very literal instance of it.
Used for
credit and insurance decisionsmedical triage protocolschurn and fraud rulesanything that has to be auditablethe building block of every boosted model
Learning typeSupervised
Primary taskClassification
Also doesRegression
MechanismRecursive splitting
Running example for the whole chapter: a groundsman deciding each Saturday morning whether the match goes ahead. Three readings are taken at nine o'clock: whether it is raining, the wind speed in km/h, and the standing water on the pitch in millimetres. Three hundred Saturdays have been recorded, two hundred to learn from and a hundred kept back. About one in eight went the other way from what the readings suggested, because the call is made by a person, pitches drain, and weather turns, and no set of questions can recover a decision that was never in the numbers.
01
Intuition: walk it yourself
1a A tree somebody wrote by hand
Here is a tree somebody wrote by hand, the way a groundsman would if you asked them to put their reasoning on paper. Below it is one Saturday's readings.
Walk it. Read the question, look at the card, and click the branch that applies. You will reach a verdict in three clicks at most, and you will have done exactly what the algorithm does at prediction time, at exactly the same speed.
The path you have taken is lit. Everything else is still there, and this day will never visit it.
Raining…
Wind…
Standing water…
Three things are worth noticing about what you just did, because each one is a property of the whole family.
- You only looked at one reading at a time. Never two together, never a combination. Every question in the tree is about a single column, which is why no scaling was needed and why a wind speed in km/h and a depth in millimetres sat happily in the same model.
- Most of the tree was irrelevant. The tree asks four questions and you answered two or three of them. The rest of the tree existed and did nothing, which is what makes prediction so fast: the cost is the depth, not the size.
- The explanation came free. "It was raining, there was 9 mm of water, and the wind was under 30, so we played." That is not a summary of the model's reasoning. It is the reasoning, complete.
1b The same walk, drawn as territory
Now the part that makes trees click, and the reason this chapter has two pictures instead of one.
Drop the rain for a moment and keep the two numbers, so everything fits on a pair of axes. Every Saturday is a dot, placed by its wind speed and its standing water. Here is a tree on the left and that plot on the right, and they are the same object.
The tree
The same tree, as territory
the match went ahead
it was called off
where you are now
Every question cuts the map with one straight line, and the line is always parallel to an axis. Answering it throws away everything on the other side. Two questions in and you are standing in a small rectangle, and the tree's verdict for that rectangle is simply whichever colour is commonest inside it.
That is the whole geometry of a decision tree, and it is worth holding onto because two of this chapter's four failure modes are direct consequences of it. A tree cannot draw a line that is not parallel to an axis. It can only ever cut the rectangle it is currently standing in. And because each cut is made without looking ahead, an early cut it cannot take back shapes everything below it.
CuriousWhere did those questions come from?
The first tree came from a groundsman. The second one did not: it was fitted to the two hundred recorded Saturdays, and nobody chose either the features or the thresholds.
Which raises the only genuinely interesting question in the chapter. There are two columns and two hundred rows, so there are a few hundred distinct thresholds it could have asked about, and it has to pick one to ask first. It cannot try every possible whole tree, because the number of trees you can build from a few hundred candidate splits is larger than anything you would want to count.
So it does something cruder and almost always good enough: it asks which single question, right now, would tidy up this pile of Saturdays the most, takes that one, and then asks the same thing again on each half. Module 2 is entirely about what "tidy up" means, because that word is doing all of the work.
02
How it chooses a question
A pile of Saturdays, some played and some called off, and one question to ask about them. A good question separates them. A bad one leaves both halves as mixed as the pile you started with. To choose automatically, that has to become a number.
2a The one move
Drag the line. Everything to the left of it becomes one pile and everything to the right becomes another, and the bars show how mixed each pile is.
played
called off
your question
the pile…
left half…
right half…
…impurity removed by this question
…
The bars are impurity: zero when a pile is all one colour, largest when it is an even mix. A question is good exactly when the two halves it produces are purer than the pile it split, weighted by how many Saturdays ended up in each. That difference has a name, information gain, and maximising it is the entire fitting procedure.
gain = impurity(the pile) − [ share left × impurity(left) + share right × impurity(right) ]
The weighting matters more than it looks. A question that peels off three Saturdays into a perfectly pure sliver has done almost nothing, because the other hundred and ninety-seven are exactly as mixed as before, and the weights say so.
2b Two ways to measure mixed
Two functions do this job, they are the only two you will meet, and they answer the same question in slightly different handwriting. Both take the proportion of one class in a node and return a number that is zero at the extremes and largest in the middle.
Gini = 1 − p² − (1−p)² = 2p(1−p) for two classes, so it peaks at 0.5
Entropy = −p log₂ p − (1−p) log₂(1−p) // peaks at 1.0, measured in bits
Entropy = −p log₂ p − (1−p) log₂(1−p) // peaks at 1.0, measured in bits
Gini
entropy
your node
…Gini
…entropy, bits
…
Drag to either end and both fall to zero, which is what a pure node is: nothing left to learn here, every Saturday in this pile went the same way.
Gini has a reading worth knowing: it is the chance of getting it wrong if you picked a Saturday from this node at random and guessed its outcome by picking another Saturday from the same node at random. A node that is 90 percent played gives 2 × 0.9 × 0.1 = 0.18, so you would be wrong about one time in five.
Entropy comes from information theory and is measured in bits: it is how many yes-or-no questions you would need, on average, to learn the outcome of one Saturday from this node. An even split costs a full bit. A node that is 90 percent played costs 0.47 of a bit, because most of the time you already know.
The practical answer to which one to use is it does not matter very much, and anybody who tells you otherwise is selling something. They agree on the best split the overwhelming majority of the time. Entropy is slightly more eager near the ends, because its curve is steeper there, so it will occasionally pay more for carving out a small pure pocket. Gini has no logarithm in it and is therefore a little faster, which is why it is the default in scikit-learn.
On this data they happen to disagree, which is a useful thing to see rather than a problem. Gini's first question is wind above … km/h, and entropy's is wind above … km/h. Both are real thresholds in the rule the groundsman was actually following, both are defensible, and by depth four the two trees score the same on the hundred Saturdays neither of them saw.
2c The search, in full
There is no cleverness in how the best question is found. Every candidate threshold on every column is tried, the gain is computed for each, and the largest wins. Here is that search laid out for the root node, both columns at once.
splitting on wind
splitting on standing water
the winner
…the question it picks
…gain
…thresholds it tried
…
Each column is drawn across the full width on its own scale, so the two are comparable in height rather than in position. Both curves are lumpy rather than smooth, and the lumps are real: the gain changes only when the threshold crosses an actual recorded Saturday, so the curve is a staircase with one step per distinct value in the column.
Then the whole thing happens again on each half, and again on each half of those, on smaller and smaller piles, until something makes it stop. That is the algorithm. Module 3 writes it out and runs it.
Notice what this search is not doing. It never asks whether a slightly worse question now would allow a much better one later. It takes the best question available at this instant, commits, and never revisits it. That is called being greedy, it is the reason fitting a tree is fast enough to be practical, and it is also why the tree you get is very rarely the best tree that exists.
CuriousWhat happens when one outcome is rare?
This is where the choice of criterion genuinely starts to matter, and where both criteria quietly fail in the same way.
Suppose only one Saturday in twenty is called off. A node holding all two hundred Saturdays is then 95 percent played, and its Gini is 2 × 0.95 × 0.05 = 0.095, which is already close to pure. There is very little impurity left to remove, so every candidate split has a tiny gain, and a stopping rule phrased as "stop when the gain is below some minimum" will stop at the root. The tree predicts "played" for everything, scores 95 percent accuracy, and has learned nothing. It is the same trap as evaluation step 2a, arriving through a different door.
Four things to do about it, roughly in the order they are worth trying.
- Weight the classes, not the rows. This is the real fix. Multiply each called-off Saturday's contribution to the impurity by twenty, so the two classes carry equal total weight, and suddenly a node that is 95 percent played is as impure as an even split and every criterion starts looking for the minority again. In scikit-learn this is
class_weight="balanced", it is one argument, and it changes the tree more than any choice of criterion ever will. - Stop using accuracy immediately. Whatever you do to the tree, a 95 percent baseline means the number you are reading is measuring the class balance. The confusion matrix is the only honest scoreboard here.
- Take the probability, not the vote. A leaf holding 30 percent called-off is a leaf the tree labels "played", and that 30 percent is a genuinely useful number that the majority vote throws away. Read
predict_probaand set the threshold yourself against what a wrong call costs, which is the same argument the evaluation page makes about every classifier. - Then, and only then, consider entropy. Its curve is steeper near the ends, so it pays slightly more for a split that isolates a small pure group of the rare class. It is a real effect and it is small. If you reach for it before doing the three things above, you are tuning the least significant knob in the room.
The short version: "Gini or entropy" is almost never the question. "How are the two classes weighted" almost always is.
03
The algorithm: every step of the loop
Four lines, and one of them calls the other three again.
1grow(a pile of Saturdays):
2 if the pile is pure, or too small, or too deep: stop and make a leaf // §3b
3 try every threshold on every column, and keep the one with the most gain // §2c
4 split the pile in two on that question
5 grow(the left half) and grow(the right half)
2 if the pile is pure, or too small, or too deep: stop and make a leaf // §3b
3 try every threshold on every column, and keep the one with the most gain // §2c
4 split the pile in two on that question
5 grow(the left half) and grow(the right half)
Line 5 is the whole trick. Each half is a smaller version of the original problem, so the same procedure handles it, and the tree builds itself by doing one thing repeatedly on ever-smaller piles.
3a Watch it grow
Here is that loop running on the two hundred recorded Saturdays. Turn the dial and the tree grows one level at a time, and the map grows with it, because they are still the same object.
The tree it has learned
The territory that makes
…leaves
…accuracy on the 200 it learned from
…accuracy on the 100 it has never seen
…
played
called off
held-out Saturday
Take it up one level at a time and watch the two pictures move together. A new level in the tree is a new set of cuts on the map, always inside a rectangle that already exists, never across one. By the far end the map is a mosaic of slivers, several of which exist to accommodate a single Saturday that went the other way from its neighbours, and every one of those slivers is a memorised mistake rather than a rule.
3b What makes it stop
Line 2 of the loop is the only thing standing between you and a tree with one leaf per row. Left alone, the recursion runs until every pile is pure, which on noisy data means until every pile is a single Saturday. Four rules do the stopping, and they are the tree's entire set of dials.
- Maximum depth. The bluntest and the most used. It caps how many questions can be asked about any one Saturday, and it is the dial in the panel above.
- Minimum samples per leaf. Refuse to create a leaf holding fewer than, say, five rows. This one deals with the slivers directly and is usually a better instrument than depth, because it limits the tree where the problem actually is rather than uniformly.
- Minimum gain. Refuse a split that does not improve impurity by some amount. Useful, and the one to be careful with on rare classes, for the reason in §2c's aside.
- Pruning, after the fact. Grow the tree out fully, then work back up removing any split that does not earn its keep on data the tree did not see. This is the principled version and the one libraries mean by cost-complexity pruning, and it generally beats guessing a depth.
The first three stop the tree early, which is called pre-pruning and is cheap and a bit blind: a split with no gain today can still be the one that unlocks a good split tomorrow, and stopping early never finds out. Post-pruning does find out, at the cost of growing the whole tree first.
CuriousWhy not look further ahead than one question?
Because it is not affordable, and because the greedy version is usually close enough that nobody minds.
With two columns and two hundred rows there are a few hundred candidate splits at the root. To look two questions ahead you would evaluate every one of those against every candidate split in each of the two halves it produces, and the count starts multiplying immediately. Looking all the way ahead, which is to say finding the genuinely smallest tree that fits the data, is a known hard problem: nobody has an algorithm that does it quickly and nobody expects one.
The cost of being greedy is real and it has a name in this chapter's failure modes. An early split is never revisited, so a question that looked best at the root and was not can distort everything beneath it, and the only evidence you will ever see of the better tree you did not get is a slightly worse score. §5b is what that feels like from the outside.
The standard response is not a cleverer search. It is to grow many greedy trees on slightly different data and average them, which trades the explanation away and is the subject of the two asides at the end of this chapter.
04
Evaluation: scoring it, and picking the depth
You now have a tree for every depth from one to ten. Two questions are left, and only the second one is about trees.
The first is how to score a classifier at all, and the answer is not in this chapter, because it is the same answer for every classifier on the site and it has its own page. The second is how to use those scores to choose the depth, which is genuinely tree-specific and turns out to be harder and more interesting than it looks.
4a The scoreboard is the shared one
A tree predicts a label, so it is marked exactly like logistic regression or KNN: count the four things that can happen, then read precision and recall off the count. Below is this chapter's tree on the hundred Saturdays it has never seen, with the depth dial from §3a still in your hands.
tree says play
tree says called off
actually played
…TRUE POSITIVEplayed, and the tree said so
…FALSE NEGATIVEplayed, and the tree called it off
actually called off
…FALSE POSITIVEcalled off, and the tree sent them out
…TRUE NEGATIVEcalled off, and the tree agreed
…accuracy
…precision
…recall
…different probabilities it can emit
…
Watch the last metric rather than the first. A leaf reports the mix of training rows that landed in it, so that mix is the probability, and a tree grown to purity has nothing but 0s and 1s to report. Every other model on the site emits a smooth spread of probabilities. This one runs out of them as it gets deeper, which is the single most consequential difference between a tree and its neighbours, and it has almost nothing to do with accuracy.
4b Picking the depth, and being wrong about it
Now the hard question. The chart above uses a hundred Saturdays held back from fitting, and if you turn the dial until that number peaks you have done something subtly dishonest: you chose a setting by looking at the data you were going to report it on. Do that and the score stops being an estimate of anything.
The standard cure is k-fold cross-validation, and it is worth saying plainly what it does, because the name makes it sound harder than it is. Cut the two hundred training Saturdays into k blocks. Hide one block, fit on the rest, score on the hidden block. Do that k times so every block gets hidden exactly once, then average. No Saturday is ever scored by a tree that saw it, and every Saturday gets used for scoring once, which is the whole trick: you get k estimates out of one dataset.
cross-validated, with the spread across folds
the single held-out hundred
the truth, on 4,000 fresh Saturdays
Each row is one fold. The shaded block is hidden that time, the rest is what the tree is fitted on.
…depth cross-validation picks
…depth the single split picks
…depth that is actually best
…
The third line is the one you never get. Every other page on this site has to stop at an estimate. This chapter's Saturdays are generated, which means the rule underneath them is known and we can draw four thousand more and simply measure how good each tree really is. So for once the answer is on the chart, and it is worth sitting with what it says.
Cross-validation picks depth 4, and it does so at every fold count from two to nine. The single held-out hundred picks depth 3. The truth is that depth 3 is right, by about four points. The careful method lost to the casual one.
That is not an argument against cross-validation, and the reason is sitting on the chart next to the answer. Look at the whiskers, not the line: at the fold count you have set, the folds disagree by … points about depth 3 alone, and the bands for depths 3, 4 and 5 overlap almost entirely. Cross-validation did not tell you "4". It told you "somewhere around 3 to 5, and I cannot separate them on two hundred rows", and then the argmax threw that away. The single split had no whiskers to show you and was right by luck; had the hundred fallen differently it would have said 5 just as confidently.
Two things follow, and they are the reason this module exists rather than being a link to the shared page.
- Read the spread, then choose the simplest setting inside it. Given three depths you cannot tell apart, take the shallowest, because it is the one with the fewest slivers and the best chance of surviving contact with next season. That rule would have landed on 3 here, from cross-validation's own output, without ever seeing the truth.
- Every fold trains on less data than your final model will. With k = 5 each tree is fitted on 160 Saturdays and you then apply its verdict to a tree fitted on 200. The best depth for 160 rows is not always the best depth for 200, and on a dataset this small that gap is big enough to move the answer by one. Raising k shrinks the gap and shrinks each scoring block at the same time, which is why the whiskers get wilder as you push the slider right.
The honest summary of the whole module: on two hundred noisy rows, no method on this chart can tell depth 3 from depth 4, and the one that admits it is the one to use. The ceiling here is about 88 percent, because twelve percent of these labels were flipped on the way in and nothing can recover them. Any tree scoring near that is done, and any story about the difference between 80 and 82 percent is a story about the split.
05
Failure modes, and how to counter each
Two of these come straight out of the geometry in §1b, and the other two come from the greed in §2c. Knowing which is which tells you whether a fix is available.
5a It can only cut parallel to an axis
Here is a rule a tree finds genuinely awkward: call the match off when wind and water together are too much, in the ordinary way where a bit of each is as bad as a lot of one. That is a diagonal line, and a tree has no diagonal to draw with.
played
called off
the real rule
…leaves spent on one straight line
…accuracy on unseen Saturdays
…
It gets there. It just pays for a line that logistic regression would have drawn with two numbers, and every step of the staircase is a threshold fitted to the handful of Saturdays sitting near it.
The counters. Build the feature the tree cannot: if wind plus water is what matters, add a column that is wind plus water and the tree asks one question about it. Or use a model whose boundary is not axis-aligned, which is logistic regression for a straight line and KNN for an arbitrary one. Or accept the staircase, because with enough data it is a perfectly good approximation and this is only a real problem when data is short.
5b Move a few Saturdays and you get a different tree
The greedy search commits to the root before it knows anything else, and the root is chosen by a gain comparison that is often close. Two candidate questions separated by a hundredth of a point are, for practical purposes, tied, and which one wins is decided by whichever Saturdays happened to be in your sample.
…its first question
…accuracy on unseen Saturdays
…different roots seen so far
…
Each press drops ten percent of the Saturdays at random and refits from scratch. Same data source, same code, same settings.
Press it a dozen times. The accuracy barely moves and the tree keeps changing its mind about what to ask first. That matters more than it looks, because the tree's selling point is that a person can read it. A rule set that reorganises itself every time the data is refreshed is hard to defend to somebody who memorised last quarter's version, and "the model is unstable but its accuracy is stable" is a sentence that satisfies nobody outside the room.
The counters. Limit the depth, since shallow trees are far steadier than deep ones. Set a minimum leaf size, which stops the bottom of the tree reorganising over single rows. Or average many trees, which fixes the variance completely and throws the readability away, and is the first aside at the end of this chapter.
5c Left alone, it memorises
The dial from §3a against both accuracies at once. §4b already put the held-out curve on a chart in order to argue about choosing the depth; this is the same curve with the training score beside it, which is where the mechanism shows. It is also bias and variance in its most literal form, because here you can watch the slivers appear.
on the 200 it learned from
on the 100 it has not seen
the best depth
…best depth
…and what it scores
…a fully grown tree scores
…
The blue line reaching a perfect score is not an achievement, it is the definition of the problem: a tree deep enough will always get every training row right, because it can always ask one more question until each row sits alone.
The counters are §3b's four rules, and the one worth reaching for first is a minimum leaf size rather than a depth cap, because it targets the slivers directly. Choose whichever you use by cross-validation rather than by eye, which is what §4b is about, and remember that the held-out hundred above is drawing a picture rather than making a decision.
5d It is drawn to columns with many distinct values
A quieter failure, and the one most likely to be sitting in a model somebody already shipped. A column with two hundred distinct values offers two hundred candidate thresholds. A column with three offers two. The search takes the best gain it can find anywhere, so the column with more chances to get lucky wins more often than it deserves to.
Push that to the limit and it becomes obvious. Add a column holding the match's reference number, which is unique per Saturday and means nothing. There is always a threshold on it that separates the training data perfectly, the gain is enormous, and the tree will take it. It will then score flawlessly on the data it learned from and be worthless, because next Saturday's reference number is one it has never seen.
The counters. Drop identifiers and near-identifiers before fitting, and treat that as a rule rather than a judgment call. Be suspicious of any high-cardinality column near the root. And do not read feature_importances_ as a ranking of what matters, since impurity-based importance inherits this bias exactly: the column with the most candidate splits accumulates the most credit. Permutation importance asks a different and better question, which is how much the score falls when you scramble that column.
5e The judgment point
Readable is not the same as right
Every other model on this site is hard to argue with because you cannot see inside it. A tree is easy to argue with, and that turns out to cut both ways. A rule path reads like a reason, and people accept it as one: "declined because postcode was in this list and income was below that number" sounds like an explanation, and it is only a description of a route through a structure that was fitted to whatever happened before.
The tree did not decide that postcode should matter. Somebody put the column in the table, and the search took it because it separated the rows. A model that can recite its reasoning is not a model that has reasons, and the fluency is exactly what makes it persuasive to a committee that would have pushed back harder on a coefficient.
Which makes the readability a responsibility rather than a feature. If the tree is going to be shown to people, somebody has to read it, all of it, and be willing to say that a split is unacceptable even though it improved the score.
06
The same tree, predicting a number
Everything so far has predicted a label. Trees predict numbers with two small changes and nothing else, which is worth a page because it is the version buried inside every gradient boosting model you will ever use.
- A leaf returns the average of the training values that landed in it, instead of the majority label. Take the median instead and one wild value stops mattering.
- Impurity becomes variance. Gini and entropy are about label mixing and have nothing to say about numbers, so the split score becomes the squared error inside each half. A good question is one that leaves two groups whose values are tight around their own averages.
classification: pick the split that minimises share × Gini(each side)
regression: pick the split that minimises share × variance(each side)
regression: pick the split that minimises share × variance(each side)
The search, the recursion and the stopping rules are untouched. Which means the geometry is untouched too: a regression tree still carves the map into rectangles, and inside each rectangle it predicts one constant number. Its output is a staircase, and it cannot produce a value outside the range it was trained on, for the same reason KNN cannot: an average of stored answers is bounded by those answers.
That staircase is the honest reason a single regression tree is rarely used on its own. A straight line through the same data is smoother and needs two numbers. What the tree brings is that it will handle a threshold, an interaction and a sharp corner without being told any of them exist, and the bias and variance page fits one on its running example so you can see the staircase directly.
07
Prompting AI: making an assistant build it properly
Trees are three lines to fit and an assistant will get them running immediately. The trap is different from the other chapters: the code is right, the default settings are actively bad, and the output is so readable that nobody checks.
"Build a decision tree to predict whether the match is called off."
What you typically get back
Three lines that fit a DecisionTreeClassifier() with no arguments. That default has no depth limit and a minimum leaf size of one, so it grows until every leaf is pure: on this data that is thirty-nine leaves and a flawless training score. It will print that score proudly. It will then offer to plot the tree, which at full depth is a wall, and probably quote feature_importances_ as though it were a ranking of what matters.
no depth limitmin leaf size of 1scored on training dataclasses unweightedimportances misreadno stability check
"I have 300 Saturdays with raining (yes/no), wind_kmh, standing_water_mm and a called_off label, and about 1 in 8 of the labels is noisy. Write Python that:
1. splits into train and test first, stratified, and touches the test set once at the end,
2. chooses max_depth and min_samples_leaf together by 5-fold cross-validation, and prints the whole grid rather than only the winner,
3. also tries cost-complexity pruning and tells me which of the two did better,
4. reports the confusion matrix and a majority-class baseline, never accuracy alone,
5. refits on five different 90 percent samples and prints the root split of each, so I can see how stable the structure is,
6. prints the final tree as readable text with
export_text, not as a picture,
7. uses permutation importance rather than feature_importances_, and says in one sentence why."What changes
The model is the same three lines. Points 2 and 3 stop the defaults from choosing the depth for you, which on this data is the difference between 86 and 75 percent on unseen Saturdays. Point 5 is the one nobody asks for and the one that changes how you present the result, because it tells you up front whether the rule path you are about to show somebody survives a refresh of the data.
depth and leaf size tunedpruning comparedconfusion matrixstability measuredtree exported as textpermutation importance
Two instructions worth adding whatever else you ask for. Say what your class balance is, because an assistant will not ask, and on a rare outcome it will hand you a tree that predicts the majority everywhere and a score that looks excellent, for the reason in §2c's aside. And ask for the tree as text rather than as a plot: export_text gives you something you can read, diff against last month's, and paste into a document, and a rendered PNG of a thirty-nine leaf tree gives you a wall nobody will ever check.
08
Beyond AI: choosing the tool, owning the call
A single decision tree is rarely the most accurate thing you can fit, and it is very often the right thing to ship anyway. It is the only model on this site that a person with no statistics can follow end to end, check line by line, and object to a specific part of. When somebody has to be told why, in a room, that is not a nice-to-have.
| If your problem looks like this | Reach for | Because | The price you pay |
|---|---|---|---|
| Thresholds and interactions, mixed column types, and somebody has to be able to read the rule | A single pruned treethis chapter | No scaling, handles categories and numbers together, and the output is a rule path rather than a coefficient. | Unstable, only cuts parallel to the axes, and rarely the most accurate option. |
| The same shape of problem, and nobody has to read anything | Random forestor gradient boosting | Averaging or correcting many trees removes most of the instability and usually wins on score. | The readability, which was the reason to pick a tree in the first place. |
| A roughly straight boundary, and you want a probability you can threshold | Logistic regressiona chapter of its own | Calibrated by construction, stable, and a signed weight per feature. | One straight boundary, and every nonlinearity built by hand. |
| A strange boundary shape, few columns, and no time to fit anything | K-nearest neighboursa chapter of its own | No training at all, and the explanation is the neighbouring cases themselves. | Scales badly with columns and pays at prediction time forever. |
8b The judgment calls no prompt makes for you
Which columns are allowed to be asked about
The search takes whatever separates the rows. If a proxy for something you would never use on purpose is in the table, it will be at the root, and it will be in the rule path you show people.
How deep is too deep to defend
Depth is usually tuned for score. If the point of the tree is that somebody can follow it, the depth that a person can hold in their head is the binding constraint, and it is smaller than the one cross-validation picks.
Every threshold is somebody's edge case
"Wind above 30" means 30.1 and 29.9 are treated as different worlds. The data has no opinion about where that line goes, and the person standing at 29.9 will.
A rule path is not a reason
It describes the route, not the justification. The justification is whatever put those columns in the table, and that was a person.
8c What people build out of trees
Almost nobody ships one tree. Both of the things they ship instead start from the failure modes above, and they take opposite routes out of them.
CuriousRandom forests: grow many, let them vote
§5b said a tree reorganises itself when the data changes slightly. A forest takes that instability and uses it.
Fit a tree to a random sample of the rows, drawn with replacement so some Saturdays appear twice and some not at all. Do it again, and again. And at every split, let each tree choose from a random subset of the columns rather than all of them, so the trees are forced to disagree about what to ask rather than all finding the same root. Then let them vote.
What the forest decides
Accuracy on the unseen hundred
…the forest scores
…one fully grown tree scores
…one properly pruned tree scores
…
Watch the left picture rather than the number. With one tree the regions are jagged, with hard corners drawn around individual Saturdays. Add trees and the corners soften, because a corner that exists in one tree and not in the other forty-nine loses the vote. The averaging is removing variance, which is the mechanism bias and variance §1 describes and this is what it looks like on real regions.
And here is the honest result, which most write-ups skip. The forest comfortably beats a fully grown tree, which is what it is for. It does not beat a single tree whose depth was chosen properly on this data, because the real rule here is a handful of axis-aligned thresholds and one small tree can express it exactly. Forests earn their keep when there are many columns and no time to tune, which describes most real projects and not this one.
What you pay is the whole of §1: fifty trees voting is a number, not a rule path, and the thing you chose a tree for is gone.
CuriousBoosting: grow them one at a time, each fixing the last
The opposite idea. A forest grows its trees independently and averages them, so each one is a full attempt at the problem. Boosting grows them in sequence, and each new one is deliberately bad at almost everything except the cases the previous ones got wrong.
Start with every Saturday weighted equally and fit the smallest possible tree: one question, two leaves, a stump. It will be poor. Then increase the weight on every Saturday it got wrong and fit another stump to the reweighted pile, so the next question is forced to care about the hard cases. Repeat, and let the stumps vote with a say proportional to how well each one did.
The stumps so far, and what they add up to
Accuracy as rounds are added
…on the 200 it learned from
…on the unseen hundred
…this round's question
…
Dots are drawn at a size proportional to their current weight, so you can watch the algorithm's attention move. The amber line is the stump this round fitted, and the shaded regions are what all the rounds so far agree on.
Two things to do with the slider. Step through the first few rounds one at a time and watch the weights: after round one, a band of Saturdays swells because the first stump got them wrong, and round two's question lands somewhere that deals with exactly those. The sequence is not fitting the data, it is fitting what is left over.
Then take it to the far end. The training score keeps creeping up and the held-out score peaks around … rounds and then slips. Boosting overfits, slowly and later than a single deep tree, but it does overfit, and the number of rounds is a hyperparameter exactly like depth. The modern libraries with the familiar names are this idea with better mathematics underneath, and they still ask you how many rounds.
The trade is the same one twice over: both of these buy accuracy and stability with the readability that was the point of a tree. If you needed the explanation, you needed the single tree, and this whole section is about what you give up to leave it behind.
A decision tree from scratch. It is the only listing on this site where the whole algorithm is a function that calls itself, and the only one where the finished model can be printed and read.
1
Two ways to measure mixed
module 2bimport numpy as np def gini(y, w=None): # w is the per-row weight. Passing class weights here is the single # most effective thing you can do on a rare outcome (§2c). w = np.ones_like(y, dtype=float) if w is None else w if w.sum() == 0: return 0.0 p = (w * y).sum() / w.sum() return 2 * p * (1 - p) def entropy(y, w=None): w = np.ones_like(y, dtype=float) if w is None else w if w.sum() == 0: return 0.0 p = (w * y).sum() / w.sum() if p in (0.0, 1.0): return 0.0 # log(0) is -inf, and a pure node costs nothing return -(p * np.log2(p) + (1 - p) * np.log2(1 - p))
Both are zero at the extremes and largest at an even split, and that is the only property the algorithm uses. Everything else about them is interpretation: Gini is the chance of guessing wrong, entropy is bits.
The w argument is the part worth copying. Once the impurity takes weights, class balancing is one line at the call site rather than a resampling step you have to remember.
2
Try everything, keep the best
module 2cdef best_split(X, y, w, impurity=gini, min_leaf=1): n, n_features = X.shape base = impurity(y, w) best = None for f in range(n_features): order = np.argsort(X[:, f]) # sort once per feature, not per split xs, ys, ws = X[order, f], y[order], w[order] for i in range(min_leaf - 1, n - min_leaf): if xs[i] == xs[i + 1]: # no threshold sits between equal values continue left, right = slice(None, i + 1), slice(i + 1, None) wl, wr = ws[left].sum(), ws[right].sum() child = (wl * impurity(ys[left], ws[left]) + wr * impurity(ys[right], ws[right])) / (wl + wr) gain = base - child if best is None or gain > best[0]: best = (gain, f, (xs[i] + xs[i + 1]) / 2) return best # (gain, feature, threshold)
That is the whole of the intelligence. Two loops, no lookahead, no backtracking: the greedy choice from §2c, written out.
The xs[i] == xs[i+1] guard is not an optimisation, it is a correctness fix. Without it you will propose a threshold that splits two identical values apart, which produces a nonsensical gain and, on a column with repeats, an infinite descent.
3
The function that calls itself
module 3class Node: def __init__(self, **kw): self.__dict__.update(kw) def grow(X, y, w, depth=0, max_depth=4, min_leaf=5, min_gain=0.0, impurity=gini): p = (w * y).sum() / w.sum() # §3b, all four stopping rules, in the order they are cheapest to check if depth >= max_depth or len(y) < 2 * min_leaf or p in (0.0, 1.0): return Node(leaf=True, p=p, n=len(y)) found = best_split(X, y, w, impurity, min_leaf) if found is None or found[0] <= min_gain: return Node(leaf=True, p=p, n=len(y)) gain, f, t = found go_left = X[:, f] <= t return Node(leaf=False, f=f, t=t, gain=gain, n=len(y), L=grow(X[go_left], y[go_left], w[go_left], depth+1, max_depth, min_leaf, min_gain, impurity), R=grow(X[~go_left], y[~go_left], w[~go_left], depth+1, max_depth, min_leaf, min_gain, impurity))
Five lines of stopping rules, one line of searching, two recursive calls. Compare it with fit in the logistic regression chapter, which runs a loop two thousand times: there is no loop here at all, and no iteration count to tune. The tree stops when it runs out of questions worth asking.
min_leaf=5 rather than the library's 1 is a deliberate default. It is the single setting that does most to prevent §5c, and leaving it at one is how the wall of slivers gets built.
4
Predict, and print the reason
modules 1 and 8def predict_proba(node, x): while not node.leaf: node = node.L if x[node.f] <= node.t else node.R return node.p # the mix of the leaf, not a rounded verdict (§2c) def explain(node, x, names): """The thing no other model on this site can do: hand back the actual route, in words, for this one row.""" steps = [] while not node.leaf: left = x[node.f] <= node.t steps.append(f"{names[node.f]} = {x[node.f]:g} " f"{'<=' if left else '>'} {node.t:g}") node = node.L if left else node.R return steps, node.p, node.n # steps -> ['raining = 1 > 0.5', 'water_mm = 9 <= 12.05', 'wind_kmh = 34 > 30.05'] # p -> 0.18 (18% of the Saturdays in this leaf were played) # n -> 22 (and there were 22 of them, which is how much to trust it)
Returning n alongside p matters more than it looks. A leaf reporting 0.18 from twenty-two Saturdays and a leaf reporting 0.00 from one are printed identically by most code, and only one of them is worth acting on.
5
The defaults that bite, in the library
§5c and §5dfrom sklearn.tree import DecisionTreeClassifier, export_text from sklearn.model_selection import GridSearchCV from sklearn.inspection import permutation_importance # The default: no depth limit, min_samples_leaf=1. It grows until every # leaf is pure, scores 1.00 on the rows it learned, and 0.75 on new ones. bad = DecisionTreeClassifier().fit(X_train, y_train) grid = GridSearchCV( DecisionTreeClassifier(class_weight="balanced", random_state=0), {"max_depth": range(1, 11), "min_samples_leaf": [1, 3, 5, 10, 20]}, cv=5, scoring="balanced_accuracy").fit(X_train, y_train) print(export_text(grid.best_estimator_, feature_names=names)) # text, not a picture: you can diff it against last month's and paste it # into a document, which is the entire reason you chose a tree # best_params_ is an argmax over the fold means, so read the spread too: # §4b is about what gets thrown away between those scores and one answer print(grid.cv_results_["std_test_score"]) # NOT feature_importances_, which is impurity based and flatters any # column with many distinct values (§5d) imp = permutation_importance(grid.best_estimator_, X_test, y_test, n_repeats=20)
The gap between bad and the tuned model on this data is eleven points on unseen Saturdays, and the untuned one reports a perfect training score on its way there. Nothing warns you, and the tree is so readable that it looks trustworthy while it does it.
class_weight="balanced" is in the estimator rather than the grid on purpose. It is not a hyperparameter to search, it is a statement about what you want the model to care about, and that is a decision rather than a result.