Performance and Load Testing

What performance testing is

Performance testing is the non-functional testing that measures how fast, how much and how steadily a system works under a defined workload: response times, throughput, error rate and resource use. It answers questions that a functional test never asks. The cost page shows the right amount, but does it still do that in two seconds when 5,000 customers open it on a cold Monday morning? Does meter-api keep accepting readings when every meter reports in the same minute? What breaks first, and does it come back on its own?

Four numbers describe almost every result:

• Latency, the time one request takes, reported as percentiles (p95, p99), not as an average.
• Throughput, the requests (or readings, or logins) handled per second.
• The error rate, the share of requests that failed. A fast system that refuses a quarter of its requests is not fast.
• Saturation of the resources behind it: processor, memory, database connections, queues. They explain why the first three changed (see utilization and saturation).

None of these numbers mean anything on their own. "p95 is 40 ms" is a fact about 10 users, or about 1,000; about one meter ID that sits in a cache, or about 400,000 that do not. Every result on this page is therefore reported together with the load that produced it. Performance testing is where testing meets capacity planning: the capacity plan says how much load to expect, the test measures how much one unit can take, and the service level objectives say what "fast enough" means.

Kinds of performance test

The names are common usage, not a formal standard, and teams use them a little differently. What matters is the question each one answers. Three of them are explained on the capacity planning page: the load test, stress test and soak test. Two more complete the set:

• A spike test raises the load sharply within seconds, holds it briefly and drops it again. It asks: does the system survive a sudden rush, and does it recover when the rush is over? For Kestrel that rush is real: when the meters' clocks are synchronized, readings arrive in the first minute of each quarter hour, and a push notification about new prices sends thousands of customers to the portal at once.
• A breakpoint test increases the load slowly and steadily until the system stops meeting its objectives or fails. Where a stress test checks how the system behaves beyond the expected peak, a breakpoint test finds where the limit is; k6's documentation describes it as a test that gradually increases load "to identify the capacity limits of the system".

A smoke test here is the performance version of the smoke test on the test levels page: a tiny run that proves the script, the test data and the environment work before you spend an hour on the real thing.

A first k6 script

k6 (Grafana k6) is an open-source load testing tool: you write the test as a JavaScript file (TypeScript works too since version 1.0), and k6's engine runs it with as many simulated users as you ask for. As of October 2026 the current version is 2.3.0. Version 1.0 (May 2025) brought semantic versioning and a new end-of-test summary; version 2.0 (May 2026) removed several old options, so tutorials that use --no-summary, k6 login or the externally-controlled executor are out of date. This is the smallest useful script:

import http from 'k6/http';
import { check, sleep } from 'k6';

export default function () {
  const res = http.get('http://localhost:8080/api/meters/KE-000123/readings');
  check(res, {
    'status is 200': (r) => r.status === 200,
    'has 24 readings': (r) => r.json('readings').length === 24,
  });
  sleep(1);
}

k6 calls the exported default function over and over; one call is an iteration. A check is a named condition on a response. Unlike an assertion in a unit test, a failing check does not stop anything: k6 counts passes and failures and keeps going, because under load you want to know how many requests went wrong. sleep(1) pauses for a second, like a person reading the page. Run it:

$ k6 run --no-color scripts/first.js

         /\      Grafana   /‾‾/  
    /\  /  \     |\  __   /  /   
   /  \/    \    | |/ /  /   ‾‾\ 
  /          \   |   (  |  (‾)  |
 / __________ \  |_|\_\  \_____/ 


     execution: local
        script: scripts/first.js
        output: -

     scenarios: (100.00%) 1 scenario, 1 max VUs, 10m30s max duration (incl. graceful stop):
              * default: 1 iterations for each of 1 VUs (maxDuration: 10m0s, gracefulStop: 30s)


running (00m01.0s), 1/1 VUs, 0 complete and 0 interrupted iterations
default   [   0% ] 1 VUs  00m01.0s/10m0s  0/1 iters, 1 per VU


  █ TOTAL RESULTS 

    checks_total.......: 2       1.931018/s
    checks_succeeded...: 100.00% 2 out of 2
    checks_failed......: 0.00%   0 out of 2

    ✓ status is 200
    ✓ has 24 readings

    HTTP
    http_req_duration..............: avg=34ms  min=34ms  med=34ms  max=34ms  p(90)=34ms  p(95)=34ms 
      { expected_response:true }...: avg=34ms  min=34ms  med=34ms  max=34ms  p(90)=34ms  p(95)=34ms 
    http_req_failed................: 0.00% 0 out of 1
    http_reqs......................: 1     0.965509/s

    EXECUTION
    iteration_duration.............: avg=1.03s min=1.03s med=1.03s max=1.03s p(90)=1.03s p(95)=1.03s
    iterations.....................: 1     0.965509/s
    vus............................: 1     min=1      max=1
    vus_max........................: 1     min=1      max=1

    NETWORK
    data_received..................: 846 B 817 B/s
    data_sent......................: 100 B 97 B/s




running (00m01.0s), 0/1 VUs, 1 complete and 0 interrupted iterations
default ✓ [ 100% ] 1 VUs  00m01.0s/10m0s  1/1 iters, 1 per VU

How to read the summary:

• Without options, k6 runs one virtual user for one iteration, which is exactly what the scenario line says. That is a smoke test.
• checks_succeeded and the ✓ lines show the checks.
• http_req_duration is the time from sending the request to receiving the whole response. The line below it, { expected_response:true }, is the same metric for successful responses only (status 200–399 by default). Keep that line in mind; it becomes important in a spike.
• http_req_failed is the error rate; http_reqs the number of requests and the rate per second.
• iteration_duration is 1.03 s: 34 ms for the request plus the one-second sleep.

Virtual users and iterations

A virtual user (VU) is one simulated client in a load test: a loop that runs the test function again and again, each VU with its own state such as cookies and connections. More load means more VUs, a longer run, or less time between iterations. The simplest way to set them is an options block in the script:

export const options = {
  vus: 10,
  duration: '15s',
};

The same can be given on the command line, which overrides the script: k6 run --vus 10 --duration 15s scripts/options.js. For a ramp, stages lists targets over time; the threshold example below uses it.

Other tools

k6 is one of several good open-source tools. The ideas on this page (workload models, think time, percentiles, thresholds) apply to all of them. Versions as of October 2026:

Think time: users are not requests

Think time is the pause a test script inserts between a virtual user's actions, to model a person reading a page or deciding what to click. It is the most underestimated setting in load testing, because it decides how many requests per second a given number of users produces. Three runs of the same script with 10 VUs, differing only in the sleep() after each request:

Recorded with k6 2.3 against the stand-in API, 10 VUs each time. The same "10 users" produced anything from 10 to 325 requests per second. The rule behind it: in a test with a fixed number of users, requests per second ≈ users ÷ (response time + think time).

# Closed model: each virtual user waits for the answer, thinks, then sends again
def requests_per_second(vus, response_s, think_s):
    return vus / (response_s + think_s)


response = 0.031  # seconds: the test API's average answer with 10 users
for think in (1.0, 0.1, 0.0):
    rate = requests_per_second(10, response, think)
    print(f"10 VUs, think time {think:3.1f} s -> {rate:5.1f} requests/s")

10 VUs, think time 1.0 s ->   9.7 requests/s
10 VUs, think time 0.1 s ->  76.3 requests/s
10 VUs, think time 0.0 s -> 322.6 requests/s

The calculation matches the recordings to within a few percent. So "the portal handles 500 users" is not a result. "500 users with 30 seconds of think time, 17 requests per second, p95 180 ms, 0.1% errors" is. Take think time from real behavior, such as the time between page views in the portal's access logs, and vary it a little (sleep(20 + Math.random() * 20)) so users do not march in step.

Thresholds: pass or fail

A threshold in k6 is a pass or fail criterion on a metric, written in the script: for example p(95)<200 on http_req_duration, or rate<0.01 on http_req_failed. If any threshold is crossed, the whole test fails. (Not to be confused with a monitoring threshold, which raises an alert in production.) This script ramps up to 20 users with stages and sets three thresholds:

import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  stages: [
    { duration: '10s', target: 20 }, // ramp up to 20 users
    { duration: '10s', target: 20 }, // stay at 20 users
    { duration: '5s', target: 0 },   // ramp down
  ],
  thresholds: {
    http_req_failed: ['rate<0.01'],                  // less than 1% errors
    http_req_duration: ['p(95)<200', 'p(99)<400'],   // milliseconds
    checks: ['rate>0.99'],
  },
};

const BASE_URL = 'http://localhost:8080';

export default function () {
  const meterId = `KE-${100000 + Math.floor(Math.random() * 900000)}`;
  const res = http.get(`${BASE_URL}/api/meters/${meterId}/readings`);
  check(res, {
    'status is 200': (r) => r.status === 200,
    'body has readings': (r) => r.body.includes('"readings"'),
  });
  sleep(1);
}

$ k6 run --no-color --summary-export=summary.json scripts/load.js

         /\      Grafana   /‾‾/  
    /\  /  \     |\  __   /  /   
   /  \/    \    | |/ /  /   ‾‾\ 
  /          \   |   (  |  (‾)  |
 / __________ \  |_|\_\  \_____/ 


     execution: local
        script: scripts/load.js
        output: -

     scenarios: (100.00%) 1 scenario, 20 max VUs, 55s max duration (incl. graceful stop):
              * default: Up to 20 looping VUs for 25s over 3 stages (gracefulRampDown: 30s, gracefulStop: 30s)


running (01.0s), 02/20 VUs, 0 complete and 0 interrupted iterations
default   [   4% ] 02/20 VUs  01.0s/25.0s

running (02.0s), 04/20 VUs, 2 complete and 0 interrupted iterations
default   [   8% ] 04/20 VUs  02.0s/25.0s

…

running (25.0s), 02/20 VUs, 349 complete and 0 interrupted iterations
default   [ 100% ] 02/20 VUs  25.0s/25.0s


  █ THRESHOLDS 

    checks
    ✓ 'rate>0.99' rate=100.00%

    http_req_duration
    ✓ 'p(95)<200' p(95)=39.97ms
    ✓ 'p(99)<400' p(99)=40.65ms

    http_req_failed
    ✓ 'rate<0.01' rate=0.00%


  █ TOTAL RESULTS 

    checks_total.......: 702     27.211932/s
    checks_succeeded...: 100.00% 702 out of 702
    checks_failed......: 0.00%   0 out of 702

    ✓ status is 200
    ✓ body has readings

    HTTP
    http_req_duration..............: avg=30.81ms min=20.56ms med=30.77ms max=41.75ms p(90)=38.69ms p(95)=39.97ms
      { expected_response:true }...: avg=30.81ms min=20.56ms med=30.77ms max=41.75ms p(90)=38.69ms p(95)=39.97ms
    http_req_failed................: 0.00%  0 out of 351
    http_reqs......................: 351    13.605966/s

    EXECUTION
    iteration_duration.............: avg=1.03s   min=1.02s   med=1.03s   max=1.04s   p(90)=1.03s   p(95)=1.04s  
    iterations.....................: 351    13.605966/s
    vus............................: 2      min=2        max=20
    vus_max........................: 20     min=20       max=20

    NETWORK
    data_received..................: 297 kB 12 kB/s
    data_sent......................: 35 kB  1.4 kB/s




running (25.8s), 00/20 VUs, 351 complete and 0 interrupted iterations
default ✓ [ 100% ] 00/20 VUs  25s

All three thresholds pass, with their measured values next to them. Note what 20 users with sleep(1) mean for this API: each VU waits 97% of the time, so on average less than one request was in flight, and latency did not change at all. Now the same test with a p95 limit the API cannot meet:

$ diff scripts/load.js scripts/load-strict.js

12c12
<     http_req_duration: ['p(95)<200', 'p(99)<400'],   // milliseconds
---
>     http_req_duration: ['p(95)<35', 'p(99)<400'],   // p(95) 35 ms: too strict for this API

$ k6 run --no-color --quiet scripts/load-strict.js

  █ THRESHOLDS 

    checks
    ✓ 'rate>0.99' rate=100.00%

    http_req_duration
    ✗ 'p(95)<35' p(95)=40.04ms
    ✓ 'p(99)<400' p(99)=41.73ms

    http_req_failed
    ✓ 'rate<0.01' rate=0.00%


  █ TOTAL RESULTS 

    checks_total.......: 700     27.124403/s
    checks_succeeded...: 100.00% 700 out of 700
    checks_failed......: 0.00%   0 out of 700

    ✓ status is 200
    ✓ body has readings

    HTTP
    http_req_duration..............: avg=31.19ms min=20.6ms med=31.2ms max=44.6ms p(90)=38.96ms p(95)=40.04ms
      { expected_response:true }...: avg=31.19ms min=20.6ms med=31.2ms max=44.6ms p(90)=38.96ms p(95)=40.04ms
    http_req_failed................: 0.00%  0 out of 350
    http_reqs......................: 350    13.562201/s

    EXECUTION
    iteration_duration.............: avg=1.03s   min=1.02s  med=1.03s  max=1.04s  p(90)=1.04s   p(95)=1.04s  
    iterations.....................: 350    13.562201/s
    vus............................: 2      min=2        max=20
    vus_max........................: 20     min=20       max=20

    NETWORK
    data_received..................: 296 kB 12 kB/s
    data_sent......................: 35 kB  1.4 kB/s



time="2026-10-07T00:28:37+02:00" level=error msg="thresholds on metrics 'http_req_duration' have been crossed"

The ✗ marks the crossed threshold, the error line names the metric, and k6 exits with exit code 99. That is all a pipeline needs: any non-zero exit status fails the job, so a performance check becomes a quality gate without extra scripting. The test still ran to the end; to stop early when a threshold fails, k6 has an object form with abortOnFail: true.

Closed and open workload models

Every test so far used a fixed group of virtual users, each waiting for its answer before sending the next request. That is a closed workload model. It has a property that matters a lot: when the system slows down, the users wait longer, so they send fewer requests. The test backs off exactly when real customers would not. Customers do not wait for each other; new ones keep arriving.

An open workload model generates new requests (or iterations) at a fixed arrival rate, regardless of how fast earlier ones are answered. If the system slows down, requests pile up in flight, just as they would in production. In k6 the two arrival-rate executors, constant-arrival-rate and ramping-arrival-rate, are open; the other four (shared-iterations, per-vu-iterations, constant-vus, ramping-vus) are closed. This script runs both models one after the other against the same API, without think time, and uses four thresholds that always pass only to make the summary show each scenario separately:

import http from 'k6/http';
import { check } from 'k6';

export const options = {
  scenarios: {
    // closed model: a fixed group of users; a user sends the next request
    // only when the previous one has come back
    closed_model: {
      executor: 'ramping-vus',
      startVUs: 0,
      stages: [
        { duration: '5s', target: 30 },
        { duration: '7s', target: 30 },
      ],
      gracefulRampDown: '0s',
    },
    // open model: new requests arrive at a fixed rate,
    // whether or not earlier requests have finished
    open_model: {
      executor: 'constant-arrival-rate',
      startTime: '12s', // after closed_model has finished
      rate: 250,
      timeUnit: '1s',
      duration: '12s',
      preAllocatedVUs: 10,
      maxVUs: 50,
    },
  },
  // these thresholds always pass; they are here so that the summary
  // shows the numbers for each scenario separately
  thresholds: {
    'http_reqs{scenario:closed_model}': ['count>0'],
    'http_reqs{scenario:open_model}': ['count>0'],
    'http_req_duration{scenario:closed_model}': ['p(95)<1000'],
    'http_req_duration{scenario:open_model}': ['p(95)<1000'],
  },
};

const BASE_URL = 'http://localhost:8080';

// no sleep(): in the open model the executor decides when the next iteration starts
export default function () {
  const meterId = `KE-${100000 + Math.floor(Math.random() * 900000)}`;
  const res = http.get(`${BASE_URL}/api/meters/${meterId}/readings`);
  check(res, { 'status is 200': (r) => r.status === 200 });
}

$ k6 run --no-color scripts/scenarios.js

         /\      Grafana   /‾‾/  
    /\  /  \     |\  __   /  /   
   /  \/    \    | |/ /  /   ‾‾\ 
  /          \   |   (  |  (‾)  |
 / __________ \  |_|\_\  \_____/ 


     execution: local
        script: scripts/scenarios.js
        output: -

     scenarios: (100.00%) 2 scenarios, 80 max VUs, 54s max duration (incl. graceful stop):
              * closed_model: Up to 30 looping VUs for 12s over 2 stages (gracefulRampDown: 0s, gracefulStop: 30s)
              * open_model: 250.00 iterations/s for 12s (maxVUs: 10-50, startTime: 12s, gracefulStop: 30s)


running (01.0s), 05/40 VUs, 76 complete and 0 interrupted iterations
closed_model   [   8% ] 05/30 VUs  01.0s/12.0s
open_model   • [   0% ] waiting    11.0s

running (02.0s), 11/40 VUs, 341 complete and 0 interrupted iterations
closed_model   [  17% ] 11/30 VUs  02.0s/12.0s
open_model   • [   0% ] waiting    10.0s

…

running (24.0s), 08/48 VUs, 6120 complete and 0 interrupted iterations
closed_model ✓ [ 100% ] 00/30 VUs  12s       
open_model     [ 100% ] 09/18 VUs  12.0s/12s  250.00 iters/s


  █ THRESHOLDS 

    http_req_duration{scenario:closed_model}
    ✓ 'p(95)<1000' p(95)=118.77ms

    http_req_duration{scenario:open_model}
    ✓ 'p(95)<1000' p(95)=39.8ms

    http_reqs{scenario:closed_model}
    ✓ 'count>0' count=3141

    http_reqs{scenario:open_model}
    ✓ 'count>0' count=2991


  █ TOTAL RESULTS 

    checks_total.......: 6132   255.127453/s
    checks_succeeded...: 99.91% 6127 out of 6132
    checks_failed......: 0.08%  5 out of 6132

    ✗ status is 200
      ↳  99% — ✓ 6127 / ✗ 5

    HTTP
    http_req_duration..............: avg=61.37ms min=2.39ms  med=38.83ms  max=128.2ms  p(90)=113.72ms p(95)=116.8ms 
      { expected_response:true }...: avg=61.42ms min=20.35ms med=38.85ms  max=128.2ms  p(90)=113.72ms p(95)=116.8ms 
      { scenario:closed_model }....: avg=90.32ms min=20.43ms med=104.11ms max=128.2ms  p(90)=116.72ms p(95)=118.77ms
      { scenario:open_model }......: avg=30.98ms min=2.39ms  med=30.69ms  max=119.88ms p(90)=38.73ms  p(95)=39.8ms  
    http_req_failed................: 0.08%  5 out of 6132
    http_reqs......................: 6132   255.127453/s
      { scenario:closed_model }....: 3141   130.68417/s
      { scenario:open_model }......: 2991   124.443283/s

    EXECUTION
    dropped_iterations.............: 10     0.416059/s
    iteration_duration.............: avg=61.53ms min=2.95ms  med=38.97ms  max=128.39ms p(90)=113.84ms p(95)=116.94ms
    iterations.....................: 6132   255.127453/s
    vus............................: 8      min=5         max=30
    vus_max........................: 48     min=40        max=48

    NETWORK
    data_received..................: 5.2 MB 216 kB/s
    data_sent......................: 613 kB 26 kB/s




running (24.0s), 00/48 VUs, 6132 complete and 0 interrupted iterations
closed_model ✓ [ 100% ] 00/30 VUs  12s 
open_model   ✓ [ 100% ] 00/18 VUs  12s  250.00 iters/s

What the two halves show:

• Closed (ramping-vus, 30 users): once the ramp-up was done, 30 requests were always in flight, the API slowed down (p95 118.77 ms against 39.8 ms) and the request rate settled where the API could keep up: 3,141 requests in 12 seconds, about 262 per second. The test did not decide the load; the system did.
• Open (constant-arrival-rate, 250 per second): k6 started 250 iterations every second, whatever the response times, and needed only about 8 busy VUs to do it (the progress line showed 09/18 VUs: 9 busy, 18 allocated). 2,991 requests in 12 seconds.
• The 5 failed requests and 10 dropped_iterations come from the first second of the open phase, while k6 was still starting extra VUs: a short burst put more than 30 requests in flight and the API refused 5 of them. Dropped iterations mean the generator could not start an iteration on time; in a real test, more than a handful means the load generator, not the system, is the bottleneck.
• The per-scenario rates in the summary (130.68417/s) are divided by the whole 24-second run, not by the 12 seconds each scenario ran. Divide the counts yourself.

Coordinated omission

The closed model's backing off has a name. Coordinated omission is the measurement error that arises when the load generator waits for slow responses before sending the next request: it never sends the requests that would have arrived during the slow period, so it never measures them, and the reported latency is better than what users experienced. The term comes from Gil Tene's talks on measuring latency; k6's documentation mentions it when it explains why arrival-rate executors exist. Imagine the API stalls for two seconds: 30 closed-model VUs each record one slow request, while in reality hundreds of customers would have arrived in those two seconds and all waited. Use an open model whenever the question is "what happens when N customers per second arrive", which for a public portal or a fleet of meters is nearly always.

Reading the results

Report percentiles, not averages. The math page shows why p95 and p99 matter more than the mean, and the capacity page why the tail latency grows first under load. Load testing adds a trap of its own: errors that make latency look good. This spike test simulates a push notification at 19:00 that sends everyone to the portal at once:

import http from 'k6/http';
import { check, sleep } from 'k6';

// 19:00 on a cold evening: a push notification sends everyone to the portal at once
export const options = {
  stages: [
    { duration: '5s', target: 5 },   // normal evening traffic
    { duration: '2s', target: 50 },  // spike
    { duration: '8s', target: 50 },  // stay at the peak
    { duration: '2s', target: 5 },   // back to normal
    { duration: '8s', target: 5 },   // does the API recover?
  ],
  thresholds: {
    http_req_failed: ['rate<0.01'],
    http_req_duration: ['p(95)<300'],
    checks: ['rate>0.99'],
  },
};

const BASE_URL = 'http://localhost:8080';

export default function () {
  const meterId = `KE-${100000 + Math.floor(Math.random() * 900000)}`;
  const res = http.get(`${BASE_URL}/api/meters/${meterId}/readings`);
  check(res, {
    'status is 200': (r) => r.status === 200,
    'not shed (503)': (r) => r.status !== 503,
  });
  sleep(0.05); // impatient users: almost no think time
}

$ k6 run --no-color scripts/spike.js

…

     scenarios: (100.00%) 1 scenario, 50 max VUs, 55s max duration (incl. graceful stop):
              * default: Up to 50 looping VUs for 25s over 5 stages (gracefulRampDown: 30s, gracefulStop: 30s)


…

running (05.0s), 04/50 VUs, 151 complete and 0 interrupted iterations
default   [  20% ] 04/50 VUs  05.0s/25.0s

running (06.0s), 27/50 VUs, 330 complete and 0 interrupted iterations
default   [  24% ] 27/50 VUs  06.0s/25.0s

running (07.0s), 49/50 VUs, 632 complete and 0 interrupted iterations
default   [  28% ] 49/50 VUs  07.0s/25.0s

running (08.0s), 50/50 VUs, 1048 complete and 0 interrupted iterations
default   [  32% ] 50/50 VUs  08.0s/25.0s

…

running (15.0s), 50/50 VUs, 3983 complete and 0 interrupted iterations
default   [  60% ] 50/50 VUs  15.0s/25.0s

running (16.0s), 30/50 VUs, 4309 complete and 0 interrupted iterations
default   [  64% ] 30/50 VUs  16.0s/25.0s

running (17.0s), 06/50 VUs, 4531 complete and 0 interrupted iterations
default   [  68% ] 06/50 VUs  17.0s/25.0s

running (18.0s), 05/50 VUs, 4595 complete and 0 interrupted iterations
default   [  72% ] 05/50 VUs  18.0s/25.0s

…

running (25.0s), 05/50 VUs, 5027 complete and 0 interrupted iterations
default   [ 100% ] 05/50 VUs  25.0s/25.0s


  █ THRESHOLDS 

    checks
    ✗ 'rate>0.99' rate=74.78%

    http_req_duration
    ✓ 'p(95)<300' p(95)=115.4ms

    http_req_failed
    ✗ 'rate<0.01' rate=25.21%


  █ TOTAL RESULTS 

    checks_total.......: 10064  401.769306/s
    checks_succeeded...: 74.78% 7526 out of 10064
    checks_failed......: 25.21% 2538 out of 10064

    ✗ status is 200
      ↳  74% — ✓ 3763 / ✗ 1269
    ✗ not shed (503)
      ↳  74% — ✓ 3763 / ✗ 1269

    HTTP
    http_req_duration..............: avg=61.63ms min=2.2ms   med=79.7ms   max=129.08ms p(90)=112.11ms p(95)=115.4ms 
      { expected_response:true }...: avg=81.55ms min=20.45ms med=98.65ms  max=129.08ms p(90)=113.68ms p(95)=116.38ms
    http_req_failed................: 25.21% 1269 out of 5032
    http_reqs......................: 5032   200.884653/s

    EXECUTION
    iteration_duration.............: avg=112.3ms min=52.35ms med=130.38ms max=179.61ms p(90)=162.79ms p(95)=166.04ms
    iterations.....................: 5032   200.884653/s
    vus............................: 5      min=1            max=50
    vus_max........................: 50     min=50           max=50

    NETWORK
    data_received..................: 3.4 MB 138 kB/s
    data_sent......................: 503 kB 20 kB/s




running (25.0s), 00/50 VUs, 5032 complete and 0 interrupted iterations
default ✓ [ 100% ] 00/50 VUs  25s
time="2026-10-07T00:33:52+02:00" level=error msg="thresholds on metrics 'checks, http_req_failed' have been crossed"

The progress lines show the spike: 4 VUs at 5 seconds, 49 at 7, 50 until 15, then back to 5. The summary holds the lesson:

• The latency threshold passed (p95 115.4 ms, well under 300) while a quarter of all requests failed. The refused requests came back in about 2 ms (min=2.2ms) and pulled the overall numbers down: the average over successful responses only ({ expected_response:true }) is 81.55 ms, against 61.63 ms over everything.
• Only the error-rate and check thresholds caught the problem. Always pair a latency threshold with an error-rate threshold.
• default ✓ [ 100% ] on the last progress line only means the scenario ran to the end; it says nothing about thresholds. The exit code (99) does.
• At 5 VUs after the spike, every request succeeded again: the API recovered as soon as the load dropped. Refusing work it cannot do (load shedding) protected it; a queue that kept growing would not have recovered so fast.

The same effect in a few lines of Python, with made-up but typical numbers:

import math
import statistics


def percentile(values, p):
    """Nearest-rank percentile: the smallest value with at least p% at or below it."""
    ordered = sorted(values)
    rank = math.ceil(p / 100 * len(ordered))
    return ordered[rank - 1]


# 1,000 requests from a spike test, in milliseconds:
# 750 got an answer (between 20 and 132 ms), 250 were refused at once with 503
ok = [20 + (i * 37) % 113 for i in range(750)]
refused = [2] * 250
everything = ok + refused

for label, values in [("all requests", everything), ("successful only", ok)]:
    print(f"{label:16} mean {statistics.mean(values):5.1f} ms   "
          f"median {statistics.median(values):5.1f} ms   p95 {percentile(values, 95):3d} ms")
print(f"error rate: {len(refused) / len(everything):.0%}")

all requests     mean  57.6 ms   median  57.0 ms   p95 125 ms
successful only  mean  76.2 ms   median  76.0 ms   p95 127 ms
error rate: 25%

Tags, groups and exports

Real tests call several endpoints. Give each request a name tag, so all /api/meters/KE-…/readings URLs are counted as one entry instead of one per meter ID, and set thresholds per name. A group() collects the requests of one user action:

import http from 'k6/http';
import { check, group, sleep } from 'k6';

export const options = {
  vus: 10,
  duration: '15s',
  thresholds: {
    // one threshold per request type, selected by tag
    'http_req_duration{name:GET /api/meters/{id}/readings}': ['p(95)<150'],
    'http_req_duration{name:POST /api/readings}': ['p(95)<100'],
    // and one for a whole group
    'http_req_duration{group:::customer opens dashboard}': ['p(95)<150'],
  },
};

const BASE_URL = 'http://localhost:8080';

export default function () {
  const meterId = `KE-${100000 + Math.floor(Math.random() * 900000)}`;

  group('customer opens dashboard', () => {
    // without the name tag every meter id would be its own URL in the results
    const res = http.get(`${BASE_URL}/api/meters/${meterId}/readings`, {
      tags: { name: 'GET /api/meters/{id}/readings' },
    });
    check(res, { 'readings 200': (r) => r.status === 200 });
    sleep(1);
  });

  group('meter submits reading', () => {
    const body = JSON.stringify({ meter_id: meterId, kwh: 0.42, at: '2026-10-06T19:00:00Z' });
    const res = http.post(`${BASE_URL}/api/readings`, body, {
      headers: { 'Content-Type': 'application/json' },
      tags: { name: 'POST /api/readings' },
    });
    check(res, { 'reading accepted (202)': (r) => r.status === 202 });
  });

  sleep(1);
}

$ k6 run --no-color --quiet --summary-mode=full scripts/portal.js

  █ THRESHOLDS 

    http_req_duration{group:::customer opens dashboard}
    ✓ 'p(95)<150' p(95)=39.83ms

    http_req_duration{name:GET /api/meters/{id}/readings}
    ✓ 'p(95)<150' p(95)=39.83ms

    http_req_duration{name:POST /api/readings}
    ✓ 'p(95)<100' p(95)=39.5ms


…

--summary-mode=full adds the timing breakdown (http_req_blocked, _connecting, _waiting and so on) and one section per group; the default compact mode leaves those out.

A winter-peak test for Kestrel

The Portal team wants proof, before December, that the cost page and meter-api survive the winter peak at the same time. The capacity plan gives the inputs: the portal peaks at 52 requests per second on a cold Monday morning in January, and 400,000 meters send one reading per 15 minutes each, 444 per second when spread evenly. The team adds its own assumption: test the portal at 1.5 times the forecast peak, because forecasts are wrong in both directions. With an open model, k6 needs enough VUs to hold the requests in flight; Little's Law gives that number from the rate and the response time:

import math

# Inputs from the capacity plan, plus the team's own safety margin (an assumption)
portal_peak = 52        # requests/s, a cold Monday morning in January
margin = 1.5            # test above the forecast, not exactly at it
readings_avg = 400_000 / (15 * 60)   # one reading per meter per 15 minutes

targets = {
    "portal GET readings": (portal_peak * margin, 0.300),  # rate, latency objective (s): p99 for the portal
    "meter POST readings": (readings_avg, 0.120),
}

for name, (rate, slow) in targets.items():
    busy = rate * slow                      # Little's Law: requests in flight
    pre = math.ceil(busy * 1.2)             # VUs to start with, 20% spare
    print(f"{name}: {rate:5.0f}/s, {busy:4.1f} in flight at {slow * 1000:.0f} ms "
          f"-> preAllocatedVUs {pre}, maxVUs {pre * 4}")

portal GET readings:    78/s, 23.4 in flight at 300 ms -> preAllocatedVUs 29, maxVUs 116
meter POST readings:   444/s, 53.3 in flight at 120 ms -> preAllocatedVUs 64, maxVUs 256

The script for the dedicated load-test environment combines both workloads in one run, with thresholds from the objectives. It was not run for this page (k6 checked that it parses, nothing more):

// winter-peak.js: meter traffic and portal customers at the same time
import http from 'k6/http';
import { check } from 'k6';

const BASE_URL = __ENV.BASE_URL || 'https://api-test.kestrel.example.com';
const HEADERS = {
  'Content-Type': 'application/json',
  Authorization: `Bearer ${__ENV.TEST_TOKEN}`, // a test-environment token, never in the script
};

export const options = {
  scenarios: {
    meters: {
      executor: 'constant-arrival-rate', // 400,000 meters, one reading per 15 minutes
      exec: 'submitReading',
      rate: 444,
      timeUnit: '1s',
      duration: '30m',
      preAllocatedVUs: 64,
      maxVUs: 256,
    },
    portal: {
      executor: 'ramping-arrival-rate', // customers checking their costs
      exec: 'openReadings',
      startRate: 10,
      timeUnit: '1s',
      preAllocatedVUs: 29,
      maxVUs: 116,
      stages: [
        { duration: '5m', target: 78 },  // a Monday morning ramp-up
        { duration: '20m', target: 78 }, // January peak + 50%
        { duration: '5m', target: 10 },
      ],
    },
  },
  thresholds: {
    'http_req_failed{scenario:meters}': ['rate<0.01'],
    'http_req_duration{scenario:meters}': ['p(95)<120'],
    'http_req_failed{scenario:portal}': ['rate<0.01'],
    'http_req_duration{scenario:portal}': ['p(99)<300'], // SLO: 99% within 300 ms
    dropped_iterations: ['count<10'], // the generator itself must keep up
  },
};

// 5,000 synthetic test meters, so caches see realistic variety
const meterId = () => `M-${90000 + Math.floor(Math.random() * 5000)}`;

export function submitReading() {
  const body = JSON.stringify({ meter_id: meterId(), kwh: 0.42, at: new Date().toISOString() });
  const res = http.post(`${BASE_URL}/v1/readings`, body, {
    headers: HEADERS,
    tags: { name: 'POST /v1/readings' },
  });
  check(res, { 'reading accepted': (r) => r.status === 202 });
}

export function openReadings() {
  const res = http.get(`${BASE_URL}/v1/meters/${meterId()}/readings`, {
    headers: HEADERS,
    tags: { name: 'GET /v1/meters/{meter_id}/readings' },
  });
  check(res, { 'readings 200': (r) => r.status === 200 });
}

The plan around the script matters as much as the script:

• Environment. A load-test environment sized and configured like production, with its own database, never db01. Results from a smaller environment do not scale up in a straight line: queues and connection pools hit their limits at different points.
• Data. 5,000 synthetic test meters and customers (never copies of production personal data), and a test token passed in as an environment variable.
• The generator. One machine running k6 can become the bottleneck itself. Watch its processor use during the run, and treat dropped_iterations as a warning that the generator, not the system, fell behind. Large tests run from several machines or a hosted service.
• Both sides. The k6 summary shows what clients saw; the dashboards of meter-api and the database show why. Note the start and end time of each run so the two can be compared.
• A soak, too. INC-22047 was a connection leak that a five-minute load test could not show; at production traffic it added six connections per minute and brought db01 within minutes of its connection limit before the rollback. The postmortem therefore added a two-hour soak test before each release (CHG-0748, owner Ravi Kumar). The same script with a two-hour middle stage does it, and because the k6 summary only shows totals at the end, the team watches the trend on the dashboards.
• Every merge, a little. A one-minute run with a few VUs and the same thresholds runs in the pipeline on every merge, as a performance regression check. The full test runs nightly and before each release.

Mistakes that make results meaningless

• Averages only. An average hides the slow tail, and fast failures pull it down. Report p95 and p99 with the error rate.
• "Users" without a rate. Say requests per second and think time, or use an arrival-rate executor and say the rate.
• The closed model for an open world. A fixed number of VUs hides the latency of a slow period (coordinated omission).
• Unrealistic data. One account, one meter, one search term: the test measures a cache.
• No warm-up. The first minute includes cold caches, connection setup and just-in-time compilation. Ramp up, and judge the steady part.
• The generator as the bottleneck. A laptop on Wi-Fi cannot load a production cluster. Check the generator's own resources and dropped_iterations.
• The wrong environment. A test system with half the database memory, or a test that skips the load balancer, measures something else.
• Ignoring errors because latency looks fine. See the spike test above.