Data for AI

Garbage in, garbage out

A dataset is a structured collection of examples used to train or evaluate a model, typically one row per example with its features and, for supervised learning, a label. Ten thousand closed tickets with their final queue form a dataset; so do a million photos with captions.

The old computing phrase "garbage in, garbage out" applies to machine learning more strongly than to ordinary software. A normal program with bad input gives one bad answer. A model trained on bad data learns the badness and repeats it on every future input, with no error message. If the service desk routinely closed account tickets under "network" because that queue was less busy, a model trained on those tickets learns to do the same.

That is why practitioners often say that most of the work in a machine-learning project is data work — finding, cleaning, labeling and checking — and that a better dataset usually beats a cleverer algorithm.

Where data comes from

• Your own systems. Ticket histories, monitoring metrics, logs, CMDB records, sales. Often the best source, because it matches the situation the model will work in.
• Public datasets. Collections released for research or benchmarking (ImageNet for images, many text corpora). Check the license and whether the data resembles yours.
• Web data. Large language models are pretrained on text collected from the public web. That raises questions about copyright, consent and quality that are still being argued in courts and parliaments.
• Purchased or partner data. Useful, but you inherit the supplier's quality and legal problems.
• Generated data. Synthetic data, discussed below.

Whatever the source, record where each dataset came from, when, and under what terms. "Datasheets for datasets" (Gebru et al., 2018) is a well-known template for exactly that.

Data labeling

Data labeling (or annotation) is the process of attaching the correct answer to each example — the queue for a ticket, "phishing" or "normal" for an email, a box around each car in a photo — so the data can be used for supervised learning. Sometimes labels come for free: a closed ticket already records which queue solved it. Often people have to create them.

The people doing this are called annotators or labelers. They can be your own experts (a security analyst labeling phishing), crowd workers, or a specialized company. Three things make the difference between useful and useless labels:

• Guidelines. Written rules for every annotator: what counts as "account" versus "network"? What do you do with a ticket that is both? Without them, each annotator follows their own idea and the model learns the mix.
• Agreement. Give the same examples to two annotators and compare. If they agree on 88 of 100 tickets, the raw agreement is 88%; the remaining twelve show where the guidelines are unclear (statistics such as Cohen's kappa also correct for agreement by chance). Label disagreement limits how well a model can learn the task, and how well you can measure it: test labels that are only 88% consistent cannot show whether a model is better than that.
• Review. Spot-check a sample, resolve disagreements, and update the guidelines.

Labeling is also human work with real conditions attached; large labeling operations have been criticized for low pay and for exposing workers to disturbing content. It is part of the cost of AI even when it is invisible in the product.

Data quality

"Quality" breaks down into dimensions that you can check one by one. The names vary between frameworks; these are common:

The first six are the classic set from DAMA UK, checked rule by rule on real data in Dimensions of data quality; representativeness is added here because a model must work on cases it has not seen.

Timeliness has a twist in machine learning: the world changes after training. New products, new attack techniques, a new office — the data the model sees in use slowly drifts away from what it learned from. This is called data drift, and it is why models need monitoring and periodic retraining.

Profiling every column first (counts, distinct values, missing values, minimum and maximum) finds most of these problems; see data profiling.

Training, validation and test sets

You cannot judge a model on the examples it learned from; it may simply have memorized them. So before training, the labeled data is split into three parts that never overlap:

• The training set is the part of the data the model learns its parameters from. It is the largest part, often 60–80%.
• The validation set is held back from training and used during development to compare models and tune settings (hyperparameters), for example to choose between three algorithms or decide when to stop training.
• The test set is held back until the very end and used once to estimate how well the final model will perform on new data. If you keep adjusting the model until the test score looks good, the test set has quietly become a second validation set and its score is no longer honest.

Two details matter. First, shuffle before splitting: data is often stored in date or category order, and taking the first 70% would give a training set that looks different from the test set. Second, when one class is rare (phishing among normal emails, failures among healthy servers), a random split can leave a part with almost none of it. A stratified split prevents that by splitting each class separately. The example below shows both on 40 labeled emails, 12 of them phishing (30%).

import random
from collections import Counter

# 40 labeled emails: 28 normal, 12 phishing (ids are made up).
emails = [(f"mail-{i:02}", "phishing" if i % 10 in (0, 3, 7) else "normal")
          for i in range(1, 41)]


def split(rows):
    """Cut a list into 70% training, 15% validation, 15% test."""
    a = round(len(rows) * 0.70)
    b = a + round(len(rows) * 0.15)
    return rows[:a], rows[a:b], rows[b:]


def report(title, parts):
    print(title)
    for name, rows in zip(("training", "validation", "test"), parts):
        counts = Counter(label for _, label in rows)
        print(f"  {name:10} {len(rows):2} emails, "
              f"{counts['phishing']:2} phishing ({counts['phishing'] / len(rows):.0%})")


random.seed(42)          # same shuffle on every run
shuffled = emails[:]
random.shuffle(shuffled)  # never split in the original order
report("Random split:", split(shuffled))

# Stratified split: split each class separately, then combine,
# so every part gets the same share of phishing.
parts = ([], [], [])
for label in ("normal", "phishing"):
    group = [row for row in shuffled if row[1] == label]
    for part, piece in zip(parts, split(group)):
        part.extend(piece)
report("Stratified split:", parts)

Random split:
  training   28 emails, 10 phishing (36%)
  validation  6 emails,  2 phishing (33%)
  test        6 emails,  0 phishing (0%)
Stratified split:
  training   28 emails,  8 phishing (29%)
  validation  6 emails,  2 phishing (33%)
  test        6 emails,  2 phishing (33%)

The random split happened to put no phishing at all in the test set: a test score from that split would say nothing about how well the model catches phishing. The stratified split keeps all three parts close to 30%. The seed makes the shuffle repeatable, so colleagues get the same split. (Libraries such as scikit-learn have a ready-made stratified split; this is the idea behind it.)

Data leakage

Data leakage is when information that will not be available at prediction time, or that comes from the evaluation data, gets into training. The model then looks excellent in testing and disappoints in production. It is one of the most common reasons for "too good to be true" results. Typical forms:

• Future information. A model that predicts whether a ticket will breach its service level uses the field "resolution time" as a feature. That field is only known after the ticket is closed, so in real use it is always empty.
• Duplicates across splits. The same phishing email, sent to 200 employees, appears 200 times. Some copies end up in training and some in test, so the test checks memory, not generalization. Remove duplicates, or split by campaign, before splitting.
• Target leakage. A feature is really a consequence of the label. Predicting "phishing" with a feature "moved to quarantine folder" is circular: the quarantine happened because someone already decided it was phishing.
• Preprocessing on all data. Computing averages or scaling over the whole dataset before splitting lets a little information from the test set into training. Fit preprocessing on the training set only.

The question to ask of every feature: "At the moment the model has to make this prediction, would we really know this value?"

Bias in the data

Data bias is a systematic distortion in a dataset — in who or what is included, how it was measured, or what past decisions it records — that makes a model trained on it treat some groups or situations unfairly or less accurately. The model is not prejudiced in any human sense; it faithfully learns what the data shows. Three common sources:

• Sampling bias: the data does not represent the population the model will serve. A speech recognizer trained mostly on one accent does worse on others; a ticket classifier trained only on English tickets fails on Dutch ones. Why a larger sample does not fix this is explained under sampling bias.
• Historical bias: the data accurately records an unfair past. Reuters reported in 2018 that Amazon dropped an experimental recruiting tool after it learned, from ten years of mostly male CVs, to downgrade CVs that mentioned "women's".
• Measurement bias: the way data is collected differs between groups. If one office logs every incident and another only logs major ones, a model will conclude that the first office has far more problems.

Bias cannot be fixed only at the end with a better algorithm; it has to be looked for in the data, from the start. How to detect and reduce it, and the Dutch childcare-benefits scandal as a lesson, are on Responsible AI.

Synthetic data

Synthetic data is artificially generated data that imitates the statistical properties of real data without being copied from real records. It can be generated by rules, simulations or generative models. Uses:

• filling gaps for rare cases (unusual failures, rare attacks);
• testing and demos without exposing real customer records;
• sharing a realistic dataset with a supplier or researchers.

It has limits. Synthetic data can only reflect what the generator knows, so it may miss exactly the surprising cases that matter, and it can copy the bias of the real data it was modeled on. Synthetic data generated from personal data is also not automatically anonymous: if individual records can be reconstructed, privacy law still applies. Why removing names or aggregating is not enough is explained under anonymization and re-identification; for synthetic data in software tests, see test data management. Use it to supplement real data, and test on real data.

Privacy and the GDPR

Tickets, emails and logs are full of personal data: names, email addresses, IP addresses, sometimes health or HR information. Using them to train a model is processing under the General Data Protection Regulation (GDPR) (in Dutch: AVG), with the same principles as any other processing: a lawful basis, purpose limitation (data collected to solve tickets is not automatically fit for training), data minimization, and limited retention. How to judge whether a new purpose is compatible with the original one (GDPR Article 6(4)) is worked out for analytics in purpose limitation.

Practical steps: remove or mask fields the model does not need, pseudonymize where possible, restrict who can access the training data, and do a data protection impact assessment (DPIA) when the processing is likely to be high-risk. Remember that a model can sometimes reproduce fragments of its training data, so personal data in training can resurface in output. And before pasting tickets into an external AI service, check whether your organization allows it; see Using AI at work.