Data Governance and Privacy

What data governance is

Data governance is the set of decision rights, rules and routines that determine who may decide what about an organization's data: what it means, how good it must be, who may see and use it, and how long it is kept. It is not a tool and not a department. In a small organization it can be a page of agreements and a short meeting each quarter; what matters is that each question has an answer and a person behind it.

Kestrel needs it the moment the KPI pack moves from Lucía Romero's spreadsheet into a pipeline and a dashboard. In the spreadsheet, Lucía decided everything silently: which contracts count as active, what to do with the meter whose customer is missing, who received the file. In the new setup those decisions must be made explicitly, once, by someone with the authority to make them. The three questions governance answers at Kestrel:

• Meaning: what is an active contract, a new customer, churn? (See Defining metrics once.)
• Quality: which checks must pass before figures are published, and who fixes the source when they fail? (See Data Types and Data Quality.)
• Use: who may see which data, for which purpose, for how long? That is mostly the privacy part of this page.

Owners and stewards

A data owner is the person accountable for a dataset or data domain: a manager on the business side who decides on its definitions, quality requirements and access, and who answers for it. A data steward looks after the data day to day on the owner's behalf: maintaining definitions and catalog entries, following up quality issues, and answering "can I use this for…" questions. The people who run the systems the data lives on are often called custodians; they protect and operate the data but do not decide what it means.

Note what the owner is not: the person who built the pipeline, or the IT department. Owners sit where the meaning and the risk sit. When Margriet asks why September's kWh figure changed after a correction, Femke's team answers, not the data engineer. Access to the analytics store follows the need-to-know principle, with requests through the normal service request process: REQ-6134 asks for Margriet's access to the KPI dashboard, and Femke approves it.

Catalogs and lineage

The data catalog

A data catalog is an inventory of an organization's datasets with their metadata: description, owner, stewards, columns and their meaning, classification, freshness, quality status and where the data comes from. It is the place an analyst looks first, so that "is there already a table with active contracts?" has an answer that is not "ask Lucía". Many catalogs fill the technical part automatically by scanning databases and pipeline code; the descriptions and ownership are written by people.

# Catalog entry for one gold table (illustrative)
name: gold.kwh_per_tariff
description: >
  Electricity imported per tariff per day, in kWh, for the monthly KPI pack.
  Contracts without a known customer are excluded; see the silver layer.
owner: Femke Jansen (head of billing)
stewards: [Lucía Romero, Joost Bakker]
classification: internal          # aggregated, no personal data
contains_personal_data: false
built_from: [silver.readings, silver.contracts]
refreshed: daily, ready by 07:00 Europe/Amsterdam
quality_checks: [unique meter per day, no negative imports, known tariff]

Examples of catalog tools, as of October 2026: the open-source OpenMetadata (2.0.4), DataHub (1.6.0.3) and Unity Catalog (0.6.0), and the catalogs of the cloud platforms, such as Microsoft Purview and the AWS Glue Data Catalog. For a team Kestrel's size, the entries above in a version-controlled repository, next to the pipeline code, are a perfectly good start. The classification field is not decoration: it is what access rules and retention rules are keyed on.

Data lineage

Data lineage is the record of where data comes from and how it moves and changes: which sources, jobs and transformations produced a table, a column or a number in a report. It answers two questions every data team gets: "where does this figure come from?" (follow the lineage upstream) and "what breaks if we change this?" (follow it downstream).

Lineage is a directed acyclic graph: datasets are the nodes, and each one points to what it is built from. Following the edges the other way finds everything downstream of a change. On 1 August 2026 the price of Kestrel's variable tariff went from €0.2500 to €0.2600 per kWh. Which tables and reports need a second look?

from collections import deque

# Lineage: each dataset and the datasets it is built from
built_from = {
    "bronze.contracts": ["kbs.contracts"],
    "bronze.invoices": ["kbs.invoices"],
    "bronze.tariff_prices": ["kbs.tariff_prices"],
    "bronze.readings": ["meter-api.readings"],
    "silver.contracts": ["bronze.contracts", "kbs.customers"],
    "silver.readings": ["bronze.readings"],
    "gold.kwh_per_tariff": ["silver.readings", "silver.contracts"],
    "gold.estimated_cost": ["silver.readings", "silver.contracts", "bronze.tariff_prices"],
    "gold.billed_per_month": ["bronze.invoices"],
    "report.kpi_pack": ["gold.kwh_per_tariff", "gold.billed_per_month"],
    "report.cost_dashboard": ["gold.estimated_cost"],
}

# Turn it around: for each dataset, which datasets use it?
used_by = {}
for target, inputs in built_from.items():
    for source in inputs:
        used_by.setdefault(source, []).append(target)


def downstream(dataset):
    """Everything built, directly or indirectly, from this dataset."""
    found, queue = set(), deque([dataset])
    while queue:
        for child in used_by.get(queue.popleft(), []):
            if child not in found:
                found.add(child)
                queue.append(child)
    return sorted(found)


for changed in ["kbs.tariff_prices", "meter-api.readings"]:
    print(f"A change to {changed} affects:")
    for name in downstream(changed):
        print("   ", name)

A change to kbs.tariff_prices affects:
    bronze.tariff_prices
    gold.estimated_cost
    report.cost_dashboard
A change to meter-api.readings affects:
    bronze.readings
    gold.estimated_cost
    gold.kwh_per_tariff
    report.cost_dashboard
    report.kpi_pack
    silver.readings

The price change touches one gold table and one report; a change in the readings touches almost everything. Lineage is only as precise as its nodes: if the whole KBS export were one node, every table would look affected by every KBS change. Real lineage tools record this graph automatically from the pipeline runs (OpenLineage, at 1.53.0 as of October 2026, is an open standard for reporting it) and some go down to column level. The same graph serves privacy: to find every place a customer's data ended up when they ask for erasure, follow the lineage downstream from the customer tables.

Data contracts

The KBS export feeds the analytics pipeline, but nobody ever promised the analysts anything. If a KBS developer renames amount_eur or moves the export to 07:00, the pipeline breaks or, worse, loads wrong data. A data contract is an agreement between the producer of a dataset and its consumers that writes down the schema, the meaning of the fields, quality rules, delivery times and how changes are announced, in a form a machine can check. It is the data counterpart of an API contract (see API Design Between Services).

# Data contract for the nightly KBS export (simplified; real formats
# such as the Open Data Contract Standard have more structure)
id: kbs-billing-export
version: 1.1.0
status: active
producer: KBS (owner Joost Bakker)
consumers: [nightly analytics load, CHG-0772]
schema:
  invoices:
    invoice_id:  {type: string, required: true, unique: true}
    contract_id: {type: string, required: true}
    period:      {type: date, required: true}
    amount_eur:  {type: "decimal(10,2)", required: true}
    status:      {type: string, allowed: [paid, open, overdue]}
quality:
  - every invoices.contract_id exists in contracts.contract_id
  - row count within 5% of the previous night
service_level:
  delivered_by: "06:00 Europe/Amsterdam"
changes:
  breaking: new major version, announced 30 days ahead
  additive: new optional columns allowed without notice

The value is in the checks and the conversation, not in the file. The pipeline validates every delivery against the contract and rejects a delivery that breaks it, before anything is published; the producer's own tests can run the same checks before a release. The Open Data Contract Standard (ODCS), a Linux Foundation AI & Data project, defines a YAML format for this; its current version is 3.2.0 (8 September 2026).

The GDPR for analytics

Most of what Kestrel analyzes is personal data: customers, their addresses, their contracts and how much energy their household uses. The basics of the General Data Protection Regulation (GDPR) are covered in Security; four of its rules shape analytics work in particular.

Purpose limitation

Personal data must be "collected for specified, explicit and legitimate purposes and not further processed in a manner that is incompatible with those purposes" (Article 5(1)(b)). Kestrel collected consumption data to bill customers. Using it for something else is "further processing", and Article 6(4) lists what decides whether the new purpose is compatible: the link between the old and new purpose, the context in which the data was collected, the nature of the data, the possible consequences for the people concerned, and safeguards "which may include encryption or pseudonymisation".

The same Article 5(1)(b) says that further processing for "statistical purposes" is not considered incompatible, with safeguards. What counts as statistical is narrow. Recital 162 says the result of statistical processing "is not personal data, but aggregate data", and that it is "not used in support of measures or decisions regarding any particular natural person". For Kestrel that draws a clear line:

If such a score were used to make decisions with legal or similarly significant effects without a human involved, such as refusing a contract, Article 22 would apply as well: people have the right "not to be subject to a decision based solely on automated processing, including profiling" in that case. For any new use of customer data at scale, Femke asks the data protection officer early; a data protection impact assessment may be required.

Outside the EU the rules on automated decisions differ; see data protection and AI.

Minimization and storage limitation

Data must be "limited to what is necessary" for the purpose (data minimization, Article 5(1)(c)) and kept in identifiable form "for no longer than is necessary" (storage limitation, Article 5(1)(e)). For an analytics store that means concrete design choices: do not copy names, email addresses or street addresses into it when the KPI pack needs only segment, city and tariff; replace customer numbers with pseudonyms; aggregate 15-minute readings to days as soon as the purpose allows; and delete on schedule (see Retention below). Article 89(1) adds that when a statistical purpose can be achieved with data that no longer identifies people, it must be achieved that way.

Smart-meter data in the Netherlands

Detailed consumption data is among the most revealing data an energy supplier holds. A household's 15-minute profile shows when people wake up, leave, come home, go on holiday and go to bed; research published as early as 2010 inferred occupancy and sleep times from it with ordinary statistical methods. Dutch rules reflect that. As of October 2026:

• The Energiewet has applied since 1 January 2026, replacing the Electricity Act 1998 and the Gas Act. It lets grid operators collect meter data no more finely than one value per quarter of an hour, and read it at most once a day. The detailed rules on data exchange have not been made yet, so the earlier rules of the regulator, the Authority for Consumers and Markets (ACM), still apply under transitional provisions.
• Under those rules, according to the ACM, an energy supplier receives monthly readings by default. Daily, hourly and 15-minute readings are passed to the supplier only with the customer's consent. The smart meter records electricity per quarter of an hour and gas per hour.
• A draft regulation on data exchange under the Energiewet would let suppliers with dynamic contracts receive the data the contract needs without separate consent. Its consultation closed on 4 October 2026; it is not in force.
• Grid operators have used 15-minute data since 1 May 2026 to allocate energy between suppliers. They aggregate it immediately and, according to grid operator Liander, keep the quarter-hour values for at most 10 days.
• Customers can have remote reading of their meter switched off ("administratief uit") free of charge.

For Kestrel's analysts this has a statistical consequence as well as a legal one. Dynamic contracts are billed per hour, so Kestrel asks for consent when they are signed; most other customers deliver only monthly readings. An analysis of "peak-hour use by tariff" can therefore include only the customers who gave consent, and they are not a random sample: they chose dynamic pricing or opted in, and probably watch their use more closely than others. Results from that group do not describe all customers (see sampling bias on Statistics for Analysts). The sample data and the pipeline examples in this topic simplify this: they show daily readings for every meter. Under the current Dutch rules, Kestrel would hold readings finer than monthly only for the customers who consented.

Pseudonymization and anonymization

Pseudonymization replaces identifiers with tokens; the security page shows how to do it with a keyed hash and why a plain hash of an email address is not enough. In Kestrel's analytics store, customer_id is replaced by such a token during the load. The key stays with Joost's pipeline team, separate from the analysts, so the analysts can count and join per customer without being able to see who the customer is.

Is that pseudonymized data still personal data? For Kestrel, certainly: Kestrel holds the key and can re-identify everyone. The Court of Justice of the EU added a nuance in case C-413/23 P (EDPS v SRB, 4 September 2025), ruling under the EU institutions' version of the GDPR, with the same definitions: pseudonymized data is not personal data "in all cases and for every person". For a recipient who has no realistic means to re-identify the people, it may not be; for the organization that pseudonymized it, it remains personal data. So a pseudonymized analytics store at Kestrel is still fully under the GDPR, with the extra protection being a safeguard, not an exemption. The European Data Protection Board (EDPB) adopted guidelines on pseudonymisation for public consultation in January 2025; as far as we can tell, no final version had been published by October 2026.

Anonymization

Anonymization is processing personal data so that the people it relates to can no longer be identified, by anyone, using all means reasonably likely to be used. Truly anonymous data is outside the GDPR: Recital 26 says the regulation "does not therefore concern the processing of such anonymous information, including for statistical or research purposes". The bar is high. The EDPB's draft guidelines on anonymisation (adopted on 7 July 2026, open for consultation until 30 October 2026) test anonymization against three risks: no record isolation (singling a person out), no linkage (connecting records about the same person across datasets) and no inference (deducing facts about a person).

Removing names and customer numbers is not anonymization. Neither is hashing them, nor aggregating, when the groups are small.

Re-identification

Re-identification is recovering who a record in a supposedly de-identified dataset belongs to, usually by combining the attributes that remain with other information. It works far more often than intuition suggests:

• Latanya Sweeney estimated in 2000, from 1990 United States census data, that 87% of the US population was likely unique on just three attributes: 5-digit ZIP code, gender and date of birth.
• In 2008, Arvind Narayanan and Vitaly Shmatikov identified known users in the "anonymized" Netflix Prize dataset by comparing a few of their film ratings with public reviews on the Internet Movie Database.
• A household's consumption curve is itself an identifier: its shape is nearly unique and stable from week to week.

A practical defense for published figures is small-cell suppression: never show a number based on fewer than k people. It follows the idea of k-anonymity (Sweeney, 2002), where each person must be indistinguishable from at least k − 1 others. Here a dashboard table of September kWh per segment and tariff suppresses cells with fewer than two customers:

-- September kWh per segment and tariff, as a dashboard cell would show it
WITH cells AS (
  SELECT cu.segment, c.tariff,
         count(DISTINCT cu.customer_id) AS customers,
         sum(r.kwh_import)              AS kwh
  FROM daily_readings r
  JOIN meters m USING (meter_id)
  JOIN contracts c USING (contract_id)
  JOIN customers cu USING (customer_id)
  GROUP BY cu.segment, c.tariff
)
SELECT segment, tariff, customers,
       CASE WHEN customers >= 2 THEN CAST(round(kwh) AS VARCHAR)
            ELSE 'suppressed' END AS kwh
FROM cells
ORDER BY segment, tariff;

┌───────────┬───────────┬───────────┬────────────┐
│  segment  │  tariff   │ customers │    kwh     │
│  varchar  │  varchar  │   int64   │  varchar   │
├───────────┼───────────┼───────────┼────────────┤
│ business  │ day_night │         1 │ suppressed │
│ business  │ dynamic   │         1 │ suppressed │
│ business  │ fixed     │         1 │ suppressed │
│ household │ day_night │         2 │ 633        │
│ household │ dynamic   │         2 │ 502        │
│ household │ fixed     │         2 │ 611        │
│ household │ variable  │         2 │ 376        │
└───────────┴───────────┴───────────┴────────────┘

Without suppression, "business, dynamic: 1,555 kWh" would simply be the bakery's consumption. The sample has only eleven customers with readings, so the example uses k = 2 to leave something to show; real policies use thresholds of 5, 10 or more, set by the data owner with the data protection officer. Even k = 2 is weak: each of the two households in a cell can subtract its own use and learn the other's exactly. Suppression also has to hold across tables, because a hidden cell can often be recovered by subtracting the visible cells from a published total.

Retention

A data retention policy states, for each category of data, how long it is kept, why, and what happens afterward (deletion or anonymization), and is enforced automatically. It puts storage limitation into practice and is the data counterpart of backup retention and log retention. Periods come from two sources:

• The law. Dutch tax law (Article 52(4) of the General Tax Act, the AWR) requires businesses to keep their records for seven years, so KBS keeps invoices for seven years. That duty applies to the administration, not to every copy: the analytics store keeps invoices only as long as the analyses need them. Kestrel chose to match the seven years so that multi-year comparisons line up with the books. How the seven years are counted is for the finance department to confirm.
• The purpose. No law requires detailed readings in an analytics store at all, so they may only be kept as long as the purpose needs them. Kestrel's policy keeps daily readings for 24 months, matching the two years of consumption history customers are shown, and quarter-hour readings of consenting customers for 13 months, enough to compare a month with the same month a year earlier. These are Kestrel's own choices, approved by Femke Jansen after advice from the data protection officer.

With data partitioned by month (see Files in a lake), enforcing the policy means deleting whole partitions that are too old. A sketch of the nightly check, on 6 October 2026:

from datetime import date

# Kestrel's retention policy for the analytics store (months to keep)
POLICY = {
    "quarter_hour_readings": 13,   # only customers who consented
    "daily_readings": 24,
    "invoices": 7 * 12,            # Kestrel's choice: matches the seven-year books
}


def months_between(earlier, later):
    return (later.year - earlier.year) * 12 + later.month - earlier.month


def expired(dataset, partition_month, today):
    """True when a monthly partition is older than the policy allows."""
    return months_between(partition_month, today) > POLICY[dataset]


today = date(2026, 10, 6)
partitions = [
    ("quarter_hour_readings", date(2025, 8, 1)),
    ("quarter_hour_readings", date(2025, 9, 1)),
    ("daily_readings", date(2024, 9, 1)),
    ("daily_readings", date(2024, 10, 1)),
    ("invoices", date(2019, 9, 1)),
    ("invoices", date(2019, 10, 1)),
]
for dataset, month in partitions:
    action = "delete" if expired(dataset, month, today) else "keep"
    print(f"{dataset:22} {month:%Y-%m}  {action}")

quarter_hour_readings  2025-08  delete
quarter_hour_readings  2025-09  keep
daily_readings         2024-09  delete
daily_readings         2024-10  keep
invoices               2019-09  delete
invoices               2019-10  keep

Deleting a partition is not the end of it:

• Copies: exports, the KPI-pack spreadsheets of previous months, test environments and backups hold the same data. The retention schedule covers them too, as described in Retention and deletion.
• Table history: in an open table format, deleted rows remain readable through time travel until the old snapshots are expired (see Time travel).
• Aggregates may outlive their inputs if they are truly anonymous: the monthly kWh per tariff can stay after the daily readings behind it are gone. Small cells must have been suppressed for that to hold.
• Logs of the deletion job itself are the evidence that the policy is followed; keep them, without the deleted data in them.