Statistics for Analysts

Populations and samples

Kestrel Energy has about 400,000 customers. Most questions an analyst gets are about all of them ("how much does a household use per month?", "would customers switch on usage alerts?"), but the data at hand often covers only some of them. Statistics is the toolkit for reasoning from the part to the whole, and for saying honestly how far that reasoning reaches.

The population is the complete set of people or things a question is about: all household customers, all portal visitors this month, all meter readings ever taken. A sample is the subset you actually observed and measured. A number that describes the population (the true average consumption) is usually unknown; the same number computed from a sample is an estimate of it, and a different sample would give a different estimate. Whether to divide by n or n − 1 for the standard deviation follows from this distinction, see sample or population?.

How much do estimates move from sample to sample? The example builds a made-up population of 100,000 households with a realistic, right-skewed spread of September usage, draws 1,000 random samples of 12 households and 1,000 of 400, and shows the range that holds the middle 95% of the sample means:

import random
import statistics as st

random.seed(2026)
# A made-up population: September kWh of 100,000 households (right-skewed)
population = [random.lognormvariate(5.6, 0.35) for _ in range(100_000)]
print(f"population mean {st.mean(population):.1f} kWh")

for n in (12, 400):
    means = [st.mean(random.sample(population, n)) for _ in range(1_000)]
    cuts = st.quantiles(means, n=40)          # 2.5%, 5%, ..., 97.5%
    print(f"n = {n:3}: 95% of sample means between {cuts[0]:.1f} and {cuts[-1]:.1f}")

population mean 286.6 kWh
n =  12: 95% of sample means between 235.9 and 348.1
n = 400: 95% of sample means between 276.8 and 296.9

With 12 households, the sample mean can easily be 50 kWh off. With 400 it is rarely more than 10 kWh off. The spread of the estimate shrinks with the square root of the sample size: four times as many observations halve the uncertainty. That is why "average consumption per customer" computed from the twelve customers in a small sample is a weak basis for any decision, however neatly it is formatted in the KPI pack.

Sampling bias: who is in the data?

The square-root rule only holds for random samples. A sampling bias is a systematic difference between the sample and the population caused by how the sample was selected, so that some members of the population were more likely to end up in it than others. More data does not cure it: a large biased sample gives a precise estimate of the wrong thing. Forms you will meet:

• Convenience samples. A satisfaction survey shown on the customer portal reaches customers who use the portal: younger, more online, often more engaged. Customers who only call the service desk are missing.
• Non-response. Of the customers who see the survey, the very satisfied and the very angry answer most. A 9% response rate says nothing about the other 91%.
• Survivorship. Kestrel's analytics copy holds current contracts. Customers who left before September are absent, so "customers on a dynamic tariff are very satisfied" may only mean that the unhappy ones are gone.
• Time windows. September usage is not a year's usage: no heating, longer days for solar panels. A sample from one month is a sample of that month.

The defense is to ask, for every dataset, "who or what could not have ended up in here?", and to say it in the report. The same question matters for training data, see bias in the data.

Confidence intervals

An estimate without its uncertainty invites false precision. A confidence interval is a range computed from a sample by a method that, over many repeated samples, captures the true population value a stated share of the time, usually 95%. Its width shows how precise the estimate is.

For a mean, the interval is the sample mean plus or minus a multiplier times the standard error (the sample standard deviation divided by √n). For large samples the multiplier is 1.96; for small ones it comes from the t distribution and is larger, to account for the extra uncertainty in the standard deviation itself. The KPI pack's average September consumption of the eight households in the sample:

import math
import statistics as st

# September kWh of the eight households in the Kestrel sample data
kwh = [334.990, 296.708, 271.212, 208.233, 336.270, 168.004, 275.960, 231.104]

n = len(kwh)
mean = st.mean(kwh)
se = st.stdev(kwh) / math.sqrt(n)        # standard error of the mean
t = 2.365                                # t table: 95%, n - 1 = 7 degrees of freedom
print(f"mean {mean:.1f} kWh, standard error {se:.1f}")
print(f"95% confidence interval: {mean - t * se:.1f} to {mean + t * se:.1f} kWh")

mean 265.3 kWh, standard error 21.1
95% confidence interval: 215.4 to 315.2 kWh

"265 kWh" sounds exact; "somewhere between 215 and 315 kWh" is what eight households can actually tell you. (Four of the eight households' readings also contain known errors, an outlier, a negative value, a duplicate and a missing day, see Exploratory Data Analysis; uncertainty from sampling comes on top of that.)

What "95%" means

The 95% belongs to the method, not to one interval. If you repeated the study many times and built a 95% interval each time, about 95% of those intervals would contain the true value. Any single interval either contains it or not; you just do not know which. The simulation below can check, because it draws samples from a population whose true mean is known:

import math
import random
import statistics as st

random.seed(7)
TRUE_MEAN = math.exp(5.6 + 0.35**2 / 2)    # known only because we chose the population
z = st.NormalDist().inv_cdf(0.975)         # 1.96
print(f"true mean {TRUE_MEAN:.1f} kWh")

hits = 0
for i in range(1, 21):
    sample = [random.lognormvariate(5.6, 0.35) for _ in range(40)]
    m = st.mean(sample)
    half = z * st.stdev(sample) / math.sqrt(len(sample))
    covers = m - half <= TRUE_MEAN <= m + half
    hits += covers
    print(f"{i:2}  {m - half:5.1f} to {m + half:5.1f}  {'contains' if covers else 'MISSES'}")
print(hits, "of 20 intervals contain the true mean")

true mean 287.5 kWh
 1  226.9 to 277.7  MISSES
 2  251.8 to 329.4  contains
 3  245.5 to 314.2  contains
 4  244.0 to 296.9  contains
 5  265.8 to 329.8  contains
 6  257.6 to 319.0  contains
 7  246.1 to 312.6  contains
 8  236.1 to 286.1  MISSES
 9  249.8 to 320.9  contains
10  254.3 to 325.7  contains
11  266.9 to 338.6  contains
12  244.9 to 311.3  contains
13  234.4 to 300.8  contains
14  267.8 to 337.3  contains
15  277.7 to 354.5  contains
16  256.9 to 312.2  contains
17  269.5 to 322.8  contains
18  246.8 to 309.5  contains
19  251.3 to 306.7  contains
20  256.1 to 336.2  contains
18 of 20 intervals contain the true mean

Hypothesis tests and p-values

Some questions are yes-or-no: did the new dashboard change anything? A hypothesis test answers them by contradiction. You state a null hypothesis, usually "there is no difference", and compute how unusual the observed data would be if it were true. Very unusual data count as evidence against the null hypothesis.

That measure of unusualness is the p-value: the probability of getting a result at least as extreme as the one observed, assuming the null hypothesis and the rest of the statistical model are true. A small p-value means the data fit poorly with "no difference".

A result is called statistically significant when its p-value falls below a threshold chosen in advance, the significance level (alpha), most often 0.05. The 0.05 is a convention, not a law of nature. Ronald Fisher, who popularized it in 1925, wrote that the value for which P = .05 "is 1.96 or nearly 2 ; it is convenient to take this point as a limit in judging whether a deviation is to be considered significant or not."

P-values are among the most misread numbers in business. In 2016 the American Statistical Association published six principles to correct that; three matter daily:

P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.

Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.

A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.

Source: Wasserstein and Lazar, "The ASA Statement on p-Values: Context, Process, and Purpose", The American Statistician 70(2), 2016. The probability of a hypothesis given the data is a different, Bayesian question (see Bayes' theorem).

A/B tests

An A/B test is a controlled experiment in which users are randomly assigned to two versions of something (a page, an email, a feature), A being the current version and B the change, and a predefined metric is compared between the groups. Random assignment is what makes it powerful: the groups differ only by chance and by the change, so a difference beyond chance can be attributed to the change, which no correlation in historical data can do.

The portal team wants customers to switch on usage alerts. Isabel Santos, the Product Owner, and Ravi Kumar proposed a "your usage this month" chart on the dashboard. Currently about 8% of visitors switch alerts on.

Plan before you start

Before the test started they wrote down:

• The metric: share of dashboard visitors who switch on a usage alert in the same visit. One primary metric, chosen in advance.
• The smallest effect worth detecting: from 8% to 10%, the rise the team hopes the chart brings, and the basis for the sample size.
• The sample size, computed from that effect, a significance level of 0.05 and 80% power (an 80% chance of detecting the effect if it is real):

import math
from statistics import NormalDist

# Plan: detect a rise from 8% to 10% with alpha 0.05 (two-sided) and 80% power
p1, p2 = 0.08, 0.10
z_alpha = NormalDist().inv_cdf(1 - 0.05 / 2)
z_power = NormalDist().inv_cdf(0.80)
p_bar = (p1 + p2) / 2
n = ((z_alpha * math.sqrt(2 * p_bar * (1 - p_bar))
      + z_power * math.sqrt(p1 * (1 - p1) + p2 * (1 - p2))) / (p2 - p1)) ** 2
print(f"visitors needed per variant: {math.ceil(n)}")

visitors needed per variant: 3213

They rounded up to 3,300 visitors per variant, which the portal reaches in about two weeks, and agreed to run whole weeks so weekday and weekend visitors are both included. The decision rule: ship B if the difference is significant at 0.05, then weigh the low end of the confidence interval against the cost of keeping the chart.

Analyze it once, at the end

After the planned number of visitors, the counts are compared with a z-test for two proportions:

import math
from statistics import NormalDist

visitors_a, alerts_a = 3300, 264     # A: current dashboard
visitors_b, alerts_b = 3300, 343     # B: dashboard with "your usage this month"

rate_a, rate_b = alerts_a / visitors_a, alerts_b / visitors_b
diff = rate_b - rate_a

# z-test for two proportions (pooled under the null hypothesis "no difference")
pooled = (alerts_a + alerts_b) / (visitors_a + visitors_b)
se_null = math.sqrt(pooled * (1 - pooled) * (1 / visitors_a + 1 / visitors_b))
z = diff / se_null
p_value = 2 * (1 - NormalDist().cdf(abs(z)))

# 95% confidence interval for the difference (unpooled standard error)
se = math.sqrt(rate_a * (1 - rate_a) / visitors_a + rate_b * (1 - rate_b) / visitors_b)
low, high = diff - 1.96 * se, diff + 1.96 * se

print(f"A {rate_a:.1%}  B {rate_b:.1%}  difference {100 * diff:+.1f} percentage points")
print(f"z = {z:.2f}, p = {p_value:.4f}")
print(f"95% CI for the difference: {100 * low:+.1f} to {100 * high:+.1f} percentage points")

A 8.0%  B 10.4%  difference +2.4 percentage points
z = 3.36, p = 0.0008
95% CI for the difference: +1.0 to +3.8 percentage points

The p-value is far below 0.05: a difference this large would be very unusual if the chart made no difference. The interval says more: the improvement is plausibly anywhere from 1.0 to 3.8 percentage points. Its lower end is below the 2 points the team hoped for, so B is very probably better, but possibly by less than planned. Write "percentage points" for a difference between two percentages: from 8.0% to 10.4% is +2.4 percentage points, or a 30% relative increase, and mixing the two is a common way to exaggerate. Because variant B adds a chart for customers, it must also be readable by everyone, see Honest and Accessible Visualization.

Peeking and multiple comparisons

The 5% false-positive rate of a test only holds if you test once, as planned. Two everyday habits break it. Peeking: checking the result every day and stopping as soon as it is significant. Multiple comparisons: testing many metrics (alerts, logins, clicks, calls, per segment, per device) and reporting the one that came out significant. The simulation runs 1,000 A/A tests, in which both variants are identical, so every "significant" result is false:

import math
import random
from statistics import NormalDist

def p_value(a, b, n):
    """Two-sided z-test for two proportions with n visitors each."""
    pooled = (a + b) / (2 * n)
    if pooled in (0, 1):
        return 1.0
    z = (b - a) / n / math.sqrt(pooled * (1 - pooled) * 2 / n)
    return 2 * (1 - NormalDist().cdf(abs(z)))

random.seed(42)
RUNS, LOOKS, STEP, RATE = 1000, 10, 200, 0.08   # A/A test: both variants 8%
end_only = stopped_early = 0
for _ in range(RUNS):
    a = b = 0
    significant_at_some_look = False
    for look in range(1, LOOKS + 1):
        a += sum(random.random() < RATE for _ in range(STEP))
        b += sum(random.random() < RATE for _ in range(STEP))
        if p_value(a, b, look * STEP) < 0.05:
            significant_at_some_look = True
    end_only += p_value(a, b, LOOKS * STEP) < 0.05
    stopped_early += significant_at_some_look

print(f"false positives, one test at the end:  {end_only / RUNS:.1%}")
print(f"false positives, stop at first p < .05: {stopped_early / RUNS:.1%}")
print(f"20 independent metrics, at least one 'significant': {1 - 0.95 ** 20:.1%}")

false positives, one test at the end:  5.2%
false positives, stop at first p < .05: 19.9%
20 independent metrics, at least one 'significant': 64.2%

Testing once at the end gives about the promised 5%. Looking ten times and stopping at the first significant look gives about 20%: a "winner" one time in five where there is nothing to find. And with twenty independent metrics the chance that at least one passes 0.05 by luck is 1 − 0.95²⁰ ≈ 64% (see independent events and "at least one"). The fixes: fix the sample size and the primary metric in advance, treat secondary metrics as ideas for the next test, and if you must monitor continuously, use a method designed for it (sequential testing) or a stricter threshold.

Significant is not the same as important

With enough data, every difference becomes significant. Across all 400,000 Kestrel customers, a difference of 2 kWh a month between two groups of 200,000 (less than 1% of a household's use) would have a tiny p-value and no consequence for anyone. The opposite also happens: with 12 customers, a large and real difference can fail to reach significance. "Not significant" means "not demonstrated with this data", not "no effect".

So report the effect size, the size of the difference in units people understand (kWh, euros, percentage points), with its confidence interval, and let the decision weigh it against cost. The chart in variant B costs some development time and a little maintenance; even the low end of the interval, one extra alert per hundred visitors, may well be worth that. Whether it is, is a business judgment, which the statistics inform but do not make.

Mistakes that make numbers lie

None of these needs advanced mathematics to avoid, only the habit of asking how a number was produced. Turning the results into charts that do not mislead is the subject of Choosing the Right Chart and Honest and Accessible Visualization.