Classic Machine Learning
The algorithms that still do most practical work: linear and logistic regression, decision trees, k-nearest neighbors and k-means clustering, each built from scratch in a few lines of Python.
When classic machine learning is the right choice
Deep learning gets the headlines, but much of the machine learning running in businesses is "classic": algorithms from statistics and early machine learning that predate the deep-learning boom. They remain the first choice when:
- The data is tabular. Rows and columns from a database or spreadsheet: ticket fields, server metrics, customer records. On this kind of data, tree-based methods are hard to beat.
- The dataset is small. A few hundred or a few thousand examples. Neural networks need much more data to avoid memorizing it.
- Decisions must be explained. A straight line or a short tree can be read and checked by a person, an auditor or a regulator. A network with millions of weights cannot, at least not directly.
- Cost and speed matter. These models train in seconds on a laptop and need no GPU.
Deep learning wins on images, audio and free text, where the useful features are hard to define by hand; see Deep learning. The terms used below (model, feature, label, training) are introduced on How machines learn.
Linear regression
Linear regression predicts
a number by fitting a straight line (or, with several features, a
flat surface) through the training data: y = a·x + b,
where the slope a and the intercept b are the
model parameters. "Fitting" means choosing a and
b so that the line is as close as possible to all points.
The usual measure is least squares: make the sum of the
squared vertical distances between the points and the line as small
as possible. For a single feature there is a direct formula, so no
trial and error is needed.
Example: a house's daily heating energy use against the average outdoor temperature. Colder days use more energy, so we expect a line sloping down.
# Average outdoor temperature (°C) and heating energy use of one house (kWh) per day.
temps = [-2, 0, 3, 5, 8, 10, 12, 15]
kwh = [39, 36, 30, 29, 23, 22, 18, 13]
# Least squares: the line y = a*x + b with the smallest sum of squared errors.
mean_x = sum(temps) / len(temps)
mean_y = sum(kwh) / len(kwh)
a = (sum((x - mean_x) * (y - mean_y) for x, y in zip(temps, kwh))
/ sum((x - mean_x) ** 2 for x in temps))
b = mean_y - a * mean_x
print(f"learned line: kWh = {a:.2f} * temp + {b:.2f}")
errors = []
for x, y in zip(temps, kwh):
predicted = a * x + b
errors.append(abs(y - predicted))
print(f"{x:3} °C actual {y:2} predicted {predicted:5.1f}")
print(f"mean absolute error: {sum(errors) / len(errors):.2f} kWh")
print(f"forecast for 6 °C: {a * 6 + b:.1f} kWh")learned line: kWh = -1.49 * temp + 35.76 -2 °C actual 39 predicted 38.7 0 °C actual 36 predicted 35.8 3 °C actual 30 predicted 31.3 5 °C actual 29 predicted 28.3 8 °C actual 23 predicted 23.8 10 °C actual 22 predicted 20.8 12 °C actual 18 predicted 17.9 15 °C actual 13 predicted 13.4 mean absolute error: 0.62 kWh forecast for 6 °C: 26.8 kWh
The model learned two numbers. Read them: every degree warmer saves about 1.49 kWh a day, and at 0 °C the house uses about 35.8 kWh. That readability is the strength of linear regression. The mean absolute error says that predictions are off by about 0.6 kWh on average on these days. (On the training days, so it is an optimistic number; a fair estimate needs held-back data, see Evaluating models.)
Classification and logistic regression
Classification is the supervised-learning task of predicting which category from a fixed set an input belongs to: phishing or normal, network or account or hardware. With two categories it is binary classification; with more, multiclass.
Logistic regression is, despite its name, a classification algorithm. It computes a weighted sum of the features, just like linear regression, and then squeezes the result through the sigmoid (logistic) function, which turns any number into a value between 0 and 1 that is read as a probability:
| Weighted sum z | sigmoid(z) |
|---|---|
| −4 | 0.02 |
| −2 | 0.12 |
| 0 | 0.50 |
| 2 | 0.88 |
| 4 | 0.98 |
Suppose a trained phishing model has learned
z = 0.6 · links + 0.2 · capitals% − 5. An email with 2
links and 3% capitals gets z = 1.2 + 0.6 − 5 = −3.2, a
probability of 0.04: normal. One with 7 links and 18% capitals gets
z = 4.2 + 3.6 − 5 = 2.8, a probability of 0.94: phishing.
The default decision threshold is 0.5, but you can move it: a
security team might flag everything above 0.3, accepting more false
alarms to miss fewer attacks.
Training finds the weights (here 0.6, 0.2 and −5) with gradient descent or a related method: start somewhere, measure how wrong the probabilities are, and nudge each weight in the direction that reduces the error. That procedure is the same one that trains neural networks; it is explained step by step on Neural networks. In fact, a single artificial neuron with a sigmoid is a logistic regression. Like linear regression, the weights are readable: a positive weight pushes toward "phishing", a negative one away from it; compare the sizes of weights only when the features are on the same scale.
Decision trees and random forests
A decision tree is a model that predicts by asking a sequence of simple questions about the features, each answer leading to the next question, until it reaches a leaf with the prediction. It looks like a flowchart, and it can be used for both classification and regression.
The difference from the hand-written router on What is AI? is where the questions come from. To learn a tree, the algorithm looks at every feature and every possible split, and picks the question that separates the labels best (the groups after the split are as "pure" as possible, measured with a statistic such as Gini impurity or information gain). It then repeats this for each branch until the groups are pure enough or the tree reaches a maximum depth.
Trees are easy to explain and handle mixed data (numbers and categories) well. Their weakness is that a deep tree keeps splitting until it has memorized the training data, noise included. Limiting the depth helps; a forest helps more.
Random forests
A random forest is an ensemble of many decision trees, each trained on a random sample of the training rows and considering a random subset of the features at each split; for a prediction, the trees vote (classification) or their answers are averaged (regression). Each tree on its own is somewhat wrong in its own way, and the errors partly cancel out — the same reason a crowd's average guess is often better than most individual guesses. Random forests and the related gradient-boosted trees are among the strongest general-purpose methods for tabular data. The price is explainability: you can no longer read the model as one flowchart.
k-nearest neighbors
k-nearest neighbors (k-NN) classifies a new item by finding the k most similar items in the labeled training data and taking a majority vote of their labels. "Similar" usually means a small distance between feature values, measured as a straight line. It hardly trains at all: the "model" is the stored examples, and all the work happens at prediction time.
Example: emails described by two features (number of links, and the percentage of words in capitals), labeled by the security team. Three new emails are classified by their three nearest neighbors.
import math
from collections import Counter
# Emails described by two features, labeled by the security team:
# (number of links, share of words in capitals as %)
emails = [
((1, 2), "normal"), ((0, 1), "normal"), ((2, 4), "normal"),
((3, 3), "normal"), ((1, 6), "normal"), ((2, 1), "normal"),
((7, 18), "phishing"), ((9, 25), "phishing"), ((6, 13), "phishing"),
((8, 9), "phishing"), ((5, 20), "phishing"),
]
def classify(new, k=3):
# Sort all known emails by straight-line distance to the new one.
by_distance = sorted(emails, key=lambda e: math.dist(e[0], new))
nearest = by_distance[:k]
for features, label in nearest:
print(f" neighbor {features} {label:8} "
f"distance {math.dist(features, new):.2f}")
votes = Counter(label for _, label in nearest)
return votes.most_common(1)[0][0]
for new in [(2, 3), (6, 15), (4, 8)]:
print(f"new email {new}:")
print(" ->", classify(new))new email (2, 3): neighbor (2, 4) normal distance 1.00 neighbor (3, 3) normal distance 1.00 neighbor (1, 2) normal distance 1.41 -> normal new email (6, 15): neighbor (6, 13) phishing distance 2.00 neighbor (7, 18) phishing distance 3.16 neighbor (5, 20) phishing distance 5.10 -> phishing new email (4, 8): neighbor (1, 6) normal distance 3.61 neighbor (8, 9) phishing distance 4.12 neighbor (2, 4) normal distance 4.47 -> normal
The first two are easy. The third, (4, 8), sits between the groups: two of its three nearest neighbors are normal, so it is classified as normal, by a vote of 2 to 1. A borderline vote like that is a good reason to send an email to a person rather than act automatically.
Two practical points. Choose an odd k for two classes so votes cannot tie. Distances can tie too: if two stored examples are equally far away at the edge of the top k, which one counts depends on how the implementation breaks ties (often simply the order of the training data), so that vote is less certain than it looks. And put features on a comparable scale first: here links run from 0 to 9 and capitals from 1 to 25, so the capitals feature dominates the distance. Real projects rescale features (for example to 0–1) before using k-NN. With large datasets k-NN becomes slow, because every prediction compares against every stored example; the same "find the nearest items" idea, made fast, powers semantic search over embeddings (see Embeddings and retrieval).
Clustering and k-means
Clustering is the unsupervised task of dividing unlabeled items into groups (clusters) so that items in the same group are more similar to each other than to items in other groups. Nobody says in advance what the groups mean; a person interprets them afterward. Uses in IT include grouping tickets to discover recurring problems, grouping servers by load pattern, and segmenting users by behavior.
k-means is the best-known clustering algorithm. You choose the number of clusters k; the algorithm then repeats two steps until nothing changes:
- Place k cluster centers (at the start, at random points).
- Assign: put each item in the cluster of its nearest center.
- Update: move each center to the average (mean) position of the items in its cluster.
- Repeat steps 2 and 3 until the assignments stop changing.
A tiny worked case: CPU load averages of six servers are 10, 12, 14, 60, 64 and 68%, and k = 2. Start the centers at 10 and 12. Assign: 10 goes to the first center; 12, 14, 60, 64 and 68 are at or closer to 12. Update: the centers move to 10 and (12 + 14 + 60 + 64 + 68) / 5 = 43.6. Assign again: 12 and 14 are now closer to 10, while 60, 64 and 68 are closer to 43.6. Update: the centers move to 12 and 64. One more round changes nothing, so the algorithm stops with a "quiet" cluster around 12% and a "busy" one around 64%.
k-means is fast and simple, but you must pick k yourself, the result can depend on the random starting centers (so it is usually run several times and the best result kept), and it assumes roughly round clusters of similar size. Like k-NN, it needs features on comparable scales.
Choosing an algorithm
Start with the question you are answering, then try the simplest model that fits it, and only add complexity when the simple one is measurably not good enough.
| You want to… | Task | Start with | Then try |
|---|---|---|---|
| Predict a number (energy use, ticket volume) | Regression | Linear regression | Random forest, gradient-boosted trees |
| Predict a category, with an explanation | Classification | Logistic regression, small decision tree | Random forest |
| Classify using similarity to known cases | Classification | k-nearest neighbors | Any of the above, if k-NN is too slow |
| Find groups in unlabeled data | Clustering | k-means | Density-based or hierarchical clustering |
| Work with images, audio or free text | Various | A pretrained deep-learning model | Fine-tuning it on your data |
In practice: libraries
The examples on this page are written from scratch with only the
Python standard library, to show that there is nothing hidden. Real
projects use a library. In Python the standard one for classic
machine learning is scikit-learn, which offers all
the algorithms above behind the same small interface (create a
model, fit it on training data, predict
on new data) plus tools for splitting, scaling and evaluation. Cloud
platforms also offer automated machine learning (AutoML), which
tries several algorithms and settings for you; you still need to
understand the results.