Evaluating Models

Why accuracy alone misleads

Imagine a bank where 1 in every 100 card transactions is fraud. A "model" that answers not fraud to every transaction, without looking at anything, is right 99 times out of 100. Its accuracy is 99%, and it is completely useless: it never catches a single fraud case, which was the whole point.

This happens whenever the classes are imbalanced — one outcome is much rarer than the other. That is the normal situation in IT and security: phishing among all email, failing disks among all disks, intrusions among all log lines. The rare class is usually the one you care about. So a single headline number is never enough; you need to see which mistakes the model makes. How the data was split into training and test sets in the first place is covered in Data for AI; the models themselves are on Classic Machine Learning.

The confusion matrix

A confusion matrix is a table that counts, for a classifier, how often each actual class was predicted as each class. For a yes/no question such as "is this email phishing?" it has four cells. By convention the class you are looking for (phishing) is called positive, the other negative — this says nothing about good or bad.

• True positive (TP): phishing, and the model flagged it.
• False negative (FN): phishing, but the model let it through — a missed case.
• False positive (FP): a normal email the model flagged — a false alarm.
• True negative (TN): a normal email, correctly let through.

With more than two classes — tickets labeled "network", "account" or "hardware" — the matrix simply gets more rows and columns: one per class. The diagonal still holds the correct answers, and the off-diagonal cells show exactly which classes the model confuses with each other, which is often more useful than any single score.

Accuracy, precision, recall and F1

Each metric below is a different question asked of the same four numbers. In words first, then as a fraction:

Accuracy is the share of all predictions that were correct. It is fine when the classes are roughly balanced and both kinds of mistake cost about the same, and misleading otherwise, as the fraud example showed.

Precision is the share of positive predictions that were actually positive. Low precision means many false alarms: users learn to ignore the warning banner, and the security team drowns in tickets. Precision also depends on how rare the positive class is: a filter that catches 95% of phishing and flags 2% of normal mail is right about only a third of what it flags when 1 in 100 emails is phishing (see Bayes' theorem).

Recall (also called sensitivity or true positive rate) is the share of actual positives the model found. Low recall means many misses: phishing reaches inboxes.

The F1 score is the harmonic mean of precision and recall. Unlike an ordinary average it is pulled down hard by the lower of the two: precision 1.00 with recall 0.10 averages to 0.55, but its F1 is only 0.18. It is useful as one summary number when both kinds of mistake matter.

Worked example: a phishing classifier

A phishing filter gives each of 20 emails a score between 0 and 1; the higher, the more suspicious. Six emails are really phishing. The code turns scores into yes/no decisions with a threshold, counts the four cells and computes the metrics — first at threshold 0.5, then at 0.3, and finally for a "model" that always says normal.

# 20 emails: 1 = phishing, 0 = normal, plus the model's phishing score
actual = [1, 0, 0, 1, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1, 0, 0]
scores = [0.95, 0.10, 0.80, 0.61, 0.05, 0.40, 0.30, 0.15, 0.55, 0.20,
          0.88, 0.35, 0.08, 0.12, 0.72, 0.52, 0.25, 0.45, 0.18, 0.02]


def evaluate(predicted):
    pairs = list(zip(actual, predicted))
    tp = pairs.count((1, 1))   # phishing, flagged
    fn = pairs.count((1, 0))   # phishing, missed
    fp = pairs.count((0, 1))   # normal, flagged (false alarm)
    tn = pairs.count((0, 0))   # normal, let through
    accuracy = (tp + tn) / len(pairs)
    precision = tp / (tp + fp) if tp + fp else 0.0
    recall = tp / (tp + fn) if tp + fn else 0.0
    f1 = (2 * precision * recall / (precision + recall)
          if precision + recall else 0.0)
    print(f"  TP={tp} FN={fn} FP={fp} TN={tn}")
    print(f"  accuracy={accuracy:.2f} precision={precision:.2f} "
          f"recall={recall:.2f} F1={f1:.2f}")


for threshold in (0.5, 0.3):
    print(f"Threshold {threshold}:")
    evaluate([1 if s >= threshold else 0 for s in scores])

print("Baseline (always 'normal'):")
evaluate([0] * len(actual))

Threshold 0.5:
  TP=4 FN=2 FP=3 TN=11
  accuracy=0.75 precision=0.57 recall=0.67 F1=0.62
Threshold 0.3:
  TP=6 FN=0 FP=5 TN=9
  accuracy=0.75 precision=0.55 recall=1.00 F1=0.71
Baseline (always 'normal'):
  TP=0 FN=6 FP=0 TN=14
  accuracy=0.70 precision=0.00 recall=0.00 F1=0.00

Three things stand out. The do-nothing baseline scores 0.70 accuracy — only five points below the real model — while catching zero phishing emails. Lowering the threshold to 0.3 leaves accuracy unchanged at 0.75, yet the model now catches all six phishing emails (recall 1.00) at the price of two extra false alarms. Accuracy could not see that difference at all; precision and recall could. (Precision is really undefined when nothing is flagged; the code reports 0.0 for that case, as scikit-learn does by default, with a warning.)

Precision versus recall: choosing a threshold

Most classifiers produce a score, not a verdict. The threshold turns the score into a decision, and it is a business choice, not a property of the model. Lower it and you flag more: recall goes up, precision usually goes down. Raise it and the reverse happens. You cannot maximize both at once with the same model; you choose where on the trade-off to sit.

The question to ask is: which mistake is more expensive?

• A missed phishing email (false negative) can mean stolen credentials and an incident. If the flagged email only gets a warning banner or lands in quarantine where a user can release it, a false alarm is cheap — favor recall.
• A false alarm (false positive) is expensive when the action is drastic: blocking a customer's payment, disabling an account, paging the on-call engineer at 03:00. Too many and people stop trusting the system — favor precision.

Medical screening, fraud detection and intrusion detection usually lean toward recall with a human checking the flagged cases; automatic blocking leans toward precision. To compare models over all thresholds at once, practitioners plot a ROC curve (receiver operating characteristic: true positive rate against false positive rate) or a precision–recall curve, and summarize it as the area under the curve (AUC). For the ROC curve, 1.0 is perfect and 0.5 is no better than guessing.

Overfitting and underfitting

A model is useful only if it works on data it has never seen. Two things can go wrong, and they are mirror images.

Underfitting means the model is too simple to capture the pattern in the data: a straight line through data that clearly curves. It does badly on the training data and on new data. Fixes: a more expressive model, better features, or more training.

Overfitting means the model learned the training data too well — including its noise and accidents — so it scores excellently on the training data and clearly worse on new data. It memorized instead of generalizing, like a student who learned last year's exam answers by heart. Fixes: more (and more varied) data, a simpler model, stopping training earlier, or regularization (a penalty on complexity).

The signature to look for is the gap between training error and test error. Both high: underfitting. Training low, test much higher: overfitting. Both low and close together: what you want. This is why you never judge a model on the data it was trained on, and why the test set must stay untouched until the very end.

Cross-validation

With a small dataset, one train/test split is a gamble: by luck the test part might contain only easy cases, or only hard ones. Cross-validation evaluates a model several times on different splits of the same data and averages the results. In k-fold cross-validation, every example is used for testing exactly once.

The usual form is k-fold cross-validation: shuffle the data, cut it into k equal parts (folds), then train k times — each time on k − 1 folds, testing on the remaining one. With k = 5, every round trains on 80% of the data and tests on 20%. The spread of the five scores is informative too: if they range from 0.60 to 0.95, the model's quality depends a lot on which data it sees.

The example below splits 15 helpdesk tickets into five folds and, in each round, evaluates the simplest possible model (the baseline described next) on the held-out fold. Ticket numbers are list positions; every ticket appears in exactly one test fold.

import random
from collections import Counter
from statistics import mean

# 15 helpdesk tickets, labeled by category (ticket number = list index)
labels = ["network", "account", "account", "hardware", "account",
          "network", "account", "account", "hardware", "network",
          "account", "account", "network", "account", "hardware"]

k = 5
indexes = list(range(len(labels)))
random.seed(7)
random.shuffle(indexes)              # mix first, so folds are not in date order
folds = [sorted(indexes[i::k]) for i in range(k)]

scores = []
for number, test in enumerate(folds, start=1):
    train = [i for i in indexes if i not in test]
    # Baseline model: always predict the most common label in the training part
    majority = Counter(labels[i] for i in train).most_common(1)[0][0]
    correct = sum(labels[i] == majority for i in test)
    score = correct / len(test)
    scores.append(score)
    print(f"Fold {number}: test {test}  predict '{majority}'  "
          f"accuracy {score:.2f}")

print(f"Mean accuracy over {k} folds: {mean(scores):.2f}")

Fold 1: test [0, 3, 4]  predict 'account'  accuracy 0.33
Fold 2: test [10, 12, 13]  predict 'account'  accuracy 0.67
Fold 3: test [6, 9, 14]  predict 'account'  accuracy 0.33
Fold 4: test [2, 7, 8]  predict 'account'  accuracy 0.67
Fold 5: test [1, 5, 11]  predict 'account'  accuracy 0.67
Mean accuracy over 5 folds: 0.53

Single folds score 0.33 or 0.67 depending on luck; the mean, 0.53, is the honest figure (8 of the 15 tickets are "account"). Real projects use a library such as scikit-learn, which does the same splitting, training and averaging in a single function call — and can keep the class proportions equal in every fold (stratified k-fold), which matters for imbalanced data. For data with a time order, such as server metrics, folds must respect time — train on the past, test on the future — or the model gets to peek ahead.

Baseline models

A score means nothing without something to compare it with. A baseline model is a deliberately simple reference model — such as always predicting the most common class, or predicting the average value — that a real model must beat to be worth anything.

In the phishing example the always-normal baseline reached 0.70 accuracy; in the ticket example the always-"account" baseline reached 0.53. A ticket classifier with 0.60 accuracy sounds mediocre; compared to 0.53 it is barely better than doing nothing. Other useful baselines: a simple rule an expert would write ("flag email whose sender domain was registered this week"), yesterday's value for a forecast, or a logistic regression before trying a neural network. If the complex model only beats the simple one by a hair, the simple one is usually the better choice: cheaper, faster and easier to explain.

Evaluating generative AI

Everything above assumes there is one right answer to compare with. Software testers call this the oracle problem; one way around it is to check how the output must change when the input changes, as in metamorphic testing. For generative AI — a summary, an email draft, a piece of code — there are many acceptable answers, so evaluation is harder and uses several methods side by side:

• Human evaluation: people rate or compare outputs (which of two answers is better?). The most trustworthy method and the most expensive; raters must have clear criteria.
• Benchmarks: fixed sets of questions with known answers, scored automatically. Useful for comparing models, but they can leak into training data and say little about your own task — see Limits and Myths.
• Automatic text metrics that measure overlap with a reference text, such as BLEU (common for translation) and ROUGE (common for summaries). Cheap, but overlap is not the same as correctness.
• Task-based tests: your own set of realistic inputs with checks — does the generated code pass its unit tests, does the answer cite the right knowledge article, is the JSON valid? Often the most useful of all for an IT team.
• Model-as-judge: another language model grades the output against criteria. Fast and scalable, but it inherits that model's own biases and errors, so it is checked against human ratings.

The same principle applies as for classifiers: decide beforehand which mistakes matter (a made-up fact, a leaked secret, a wrong tone) and measure those, on examples that look like real use.