Exploratory and Manual Testing

Scripted and exploratory testing

Manual testing means a person operates the software and judges what happens, instead of a program doing both. It comes in two styles, and most teams use both.

In scripted testing, tests are designed in advance and written down as test cases: steps, test data and an expected result. Whoever runs them follows the script. Scripted tests are repeatable and easy to hand over, and they suit checks that must be done the same way every time, such as a regulatory acceptance test. Their weakness is that they only look where their author thought to look, and a script that has passed a hundred times rarely finds anything new (the testing principle that "tests wear out").

Exploratory testing turns that around. The ISTQB (International Software Testing Qualifications Board) glossary describes it as an approach in which tests are designed and executed dynamically, based on the tester's knowledge, exploration of the test item and the results of previous tests. You try something, look at what happens, and let that decide your next test. The term first appeared in print in Cem Kaner's book Testing Computer Software (1988); the practice is as old as software.

Test charters

A test charter is the mission of one exploratory session: what to explore, and what kind of information or problems to look for. It is short on purpose. A charter does not list steps; it points the tester's attention and leaves the route to them. A widely used template comes from Elisabeth Hendrickson's book Explore It! (2013):

Explore <target>
with <resources>
to discover <information>

At Kestrel Energy, the Portal team is building the "your costs this month" page (Product Backlog item PBI-450), which splits a customer's costs per day into a day part and a night part. The night rule (night is before 07:00 and from 23:00) already caused one bug, and 25 October 2026 is a day with 100 quarter-hours instead of 96. Sofia Lindqvist, one of the Developers, writes this charter for herself:

Explore the cost page
with day_night and dynamic test contracts and the synthetic readings
for 24-26 October 2026 in the test environment
to discover billing surprises: totals, rounding and the day/night split
that a customer could not explain or that differ from the KBS invoice

Good charters are neither too broad nor too narrow. "Test the portal" gives no direction; "check that 23:00 is billed at the night price" is a single test case, not a session. A useful check: could a colleague tell, after 90 minutes, whether the session did what the charter asked?

Session-based test management

Exploratory testing has a reputation problem: to a manager it can look like someone clicking around. Jonathan Bach answered that in 2000 with session-based test management (SBTM), developed with his brother James Bach at Hewlett-Packard. The unit of work is not a test case but a session, which he defined as "an uninterrupted block of reviewable, chartered test effort":

• Chartered: every session has a charter.
• Uninterrupted: no email, chat or meetings during the session. In his team sessions lasted "ninety minutes, give or take"; about 45 minutes counted as a short session and about two hours as a long one. This is a timebox, not a stopwatch.
• Reviewable: the tester writes a session sheet that someone else can read, and discusses it in a short debrief.

The session sheet

The session sheet is a plain text form. Bach's original has the sections shown below. The task breakdown is the tester's estimate of how the session time was split: test design and execution (looking for problems), bug investigation and reporting, and session setup (preparing data, reading, writing the sheet). He called these the TBS metrics. A session that is mostly setup tells the team that the test environment is getting in the way. "Charter vs. opportunity" records how much time went to useful testing outside the charter, which is allowed when the tester stumbles on something important.

CHARTER
Explore the cost page with day_night and dynamic test contracts and the
synthetic readings for 24-26 October 2026 to discover billing surprises.
#AREAS
Portal | cost page | day/night split
Data   | DST day 25 October 2026
Oracle | KBS invoice preview

START
2026-10-07 14:00

TESTER
Sofia Lindqvist

TASK BREAKDOWN
#DURATION                          normal (90 min)
#TEST DESIGN AND EXECUTION         60
#BUG INVESTIGATION AND REPORTING   30
#SESSION SETUP                     10
#CHARTER VS. OPPORTUNITY           90/10

DATA FILES
synthetic-readings-2026-10-24..26.csv (test meters M-30001..M-30014)

TEST NOTES
- day_night, 24 Oct: day + night parts add up to the total on every meter
- 25 Oct: the day list shows 02:00-02:45 only once; daily kWh too low
- 23:00 and 06:59 billed at night price, 07:00 at day price (fix holds)
- compared M-30007 with the KBS invoice preview: night cost differs
- dynamic contract: hourly prices shown, but no explanation for the hour
  that occurs twice on 25 Oct (opportunity, outside the charter)

BUGS
#BUG 1042  reproduced: 25 Oct total counts 96 quarter-hours, not 100
#BUG 1051  night cost on cost page EUR 0.32, invoice EUR 0.36 (M-30007,
           24 Oct): cost page rounds per quarter-hour, KBS per day

ISSUES
#ISSUE 1  Chart legend says "Low"/"Normal", the invoice says
          "Night"/"Day". Which words do customers know? (ask Isabel)
#ISSUE 2  Test environment has no dynamic prices for 25 Oct 02:00-03:00
          +0100; needed for the next session

Notice the split between bugs (concerns about the product) and issues (questions or obstacles about the testing or the project). Both matter; only the bugs go into the defect tracker, written up as described on Test Cases and Bug Reports. The percentages are rough estimates, not timesheet data.

The debrief

Right after the session, Sofia and Lars de Boer, the team's Scrum Master, spend fifteen minutes on the sheet. SBTM's debrief agenda is PROOF:

• Past: what happened during the session?
• Results: what was achieved, what was found?
• Obstacles: what got in the way of good testing? (Here: no dynamic prices for the repeated hour.)
• Outlook: what still needs to be done? (A second session on dynamic contracts on 25 October.)
• Feelings: how does the tester feel about it? An uneasy "I am not sure the totals are right" is information too.

Lars does not need to be a tester to run this; any team member who asks good follow-up questions will do. In a team without testers, pairing developers for debriefs also spreads knowledge of the product.

Heuristics: where to look

A testing heuristic is a fallible rule of thumb that helps you decide what to test or whether something is a problem. "Fallible" is the point: a heuristic suggests, it does not prove. Checklists of heuristics are how experienced testers avoid relying on whatever comes to mind first.

SFDIPOT: the product elements

James Bach's Heuristic Test Strategy Model (version 6.3, 2024) lists seven kinds of product elements to consider, remembered as SFDIPOT. Older material uses SFDPOT ("San Francisco Depot"), which lacks the Interfaces element. Applied to the cost page:

Ten minutes with this table before a session produces more test ideas than an hour of improvising. It also works as a review question for an automated suite: which of these elements do our tests never touch? Other well-known heuristics are short "touring" ideas (follow the money, follow the data from input to report) and the boundaries and partitions from Test Design Techniques, which exploratory testers use constantly.

Test oracles: how you know it is wrong

Finding a problem has two parts: making it happen, and recognizing it. A test oracle is the source you use to decide what the right result is: a specification, an expected value in a test case, another system, a calculation by hand, a domain expert. The oracle problem is that for many results there is no easy way to know the right answer. What is the correct cost for an hour that occurs twice? No requirement said.

FEW HICCUPPS: consistency oracles

When there is no written answer, testers fall back on consistency: a product should be consistent with things we know. The list HICCUPPS goes back to James Bach; Michael Bolton extended it in 2012 to FEW HICCUPPS. Each letter is a reason to call something a problem:

Sofia's rounding finding came from the "comparable products" oracle. The cost page and the KBS (Kestrel Billing System) invoice both price the same day's night readings, but round at different moments. A few lines of Python reproduce the difference exactly, which turns "the numbers look slightly off" into a bug report with evidence:

from decimal import Decimal, ROUND_HALF_UP

NIGHT_PRICE = Decimal("0.2240")  # euro per kWh, day_night contract
CENT = Decimal("0.01")

# Night quarter-hours of meter M-30007 on 24 October: 00:00-06:59 and
# 23:00-23:59, 32 in total, each 0.05 kWh
readings = [Decimal("0.05")] * 32


def cents(amount):
    return amount.quantize(CENT, rounding=ROUND_HALF_UP)


# Portal (cost page): rounds every quarter-hour, then adds the cents
portal = sum(cents(kwh * NIGHT_PRICE) for kwh in readings)

# KBS (the invoice): adds the kWh of the day, then rounds once
total_kwh = sum(readings)
kbs = cents(total_kwh * NIGHT_PRICE)

print(f"Night use:        {total_kwh} kWh")
print(f"Cost page shows:  EUR {portal}")
print(f"Invoice shows:    EUR {kbs}")
print(f"Difference:       EUR {kbs - portal}")

Night use:        1.60 kWh
Cost page shows:  EUR 0.32
Invoice shows:    EUR 0.36
Difference:       EUR 0.04

Each quarter-hour costs €0.0112, which rounds down to €0.01, so the cost page loses €0.0012 thirty-two times. Which side is "right" is a business decision for Isabel Santos (Product Owner) and Joost Bakker (KBS owner), not for the tester. The tester's job is to make the inconsistency visible and explainable.

Tools that support exploration

Exploratory testing is manual in the sense that a person steers, not in the sense that no tools are allowed. Testers use whatever makes them faster or lets them see more:

• Browser developer tools to watch the API requests behind a page, change the window size or throttle the network (see Testing and Debugging APIs).
• Logs and dashboards of the test environment, to see the error that the page politely hides.
• Small scripts that generate test data or compute an expected answer, like the rounding check above.
• Note-taking with timestamps and screenshots, so that a surprise can be reproduced (see reproducing the problem).

Before her session, Sofia wanted to know exactly which quarter-hours a customer should see on 25 October, when the clocks in the Netherlands go back from 03:00 summer time to 02:00 winter time. The standard library answers it in a dozen lines:

from datetime import date, datetime, timedelta, timezone
from zoneinfo import ZoneInfo

AMSTERDAM = ZoneInfo("Europe/Amsterdam")
QUARTER = timedelta(minutes=15)


def quarter_hours(day):
    """Local start times of every quarter-hour of one calendar day."""
    start = datetime(day.year, day.month, day.day, tzinfo=AMSTERDAM)
    end = start + timedelta(days=1)  # midnight of the next day, local time
    moment = start.astimezone(timezone.utc)
    result = []
    while moment < end.astimezone(timezone.utc):
        result.append(moment.astimezone(AMSTERDAM))
        moment += QUARTER
    return result


for day in (date(2026, 10, 24), date(2026, 10, 25), date(2026, 10, 26)):
    print(day, len(quarter_hours(day)), "quarter-hours")

print()
print("25 October, around the clock change:")
for moment in quarter_hours(date(2026, 10, 25))[7:16]:
    print(" ", moment.strftime("%H:%M"), moment.strftime("%z"))

2026-10-24 96 quarter-hours
2026-10-25 100 quarter-hours
2026-10-26 96 quarter-hours

25 October, around the clock change:
  01:45 +0200
  02:00 +0200
  02:15 +0200
  02:30 +0200
  02:45 +0200
  02:00 +0100
  02:15 +0100
  02:30 +0100
  02:45 +0100

The script steps through the day in UTC (Coordinated Universal Time), where every quarter-hour is exactly 15 minutes, and only converts to local time for display. That is how the times 02:00 to 02:45 appear twice, with different UTC offsets. With this list in hand, the session could test the right thing: does the portal show 100 rows, and can a customer tell the two 02:15 rows apart?

Usability testing

Some problems only show up when the person at the keyboard is not on the team. Usability testing evaluates how easily real users can reach their goals with the product, usually by watching representative users do realistic tasks while they think aloud. The observer does not help or explain; every moment of hesitation is data.

For the cost page, Daniel Mensah (head of customer service) invites two of the pilot customers who already come to the Sprint Review. Each gets three tasks, for example "Find out how much of this month's costs were at the night rate" and "Find the day this month when you used the most electricity". Isabel Santos and Bram Peeters watch and take notes. Typical findings are not bugs in the strict sense: the customer looks for the night costs in the menu instead of the chart, or reads "Low" as "low usage" instead of "night rate" (the same issue Sofia raised).

• Test with the people who will use the product, not colleagues who know how it works.
• Small, frequent rounds after each change teach more than one big study at the end.
• Record what people do, not only what they say: people blame themselves for problems the design caused.
• Ask for consent before recording, and keep recordings and notes free of personal data that is not needed (see GDPR).

Usability is related to, but not the same as, accessibility: a page can be easy to use for most people and still impossible with a keyboard or a screen reader. That is covered in Accessibility Testing.

Manual or automated?

The useful question is not "manual or automated?" but "who is better at this particular job, a person or a program?" A program is better at repeating the same check exactly, fast and often. A person is better at noticing what nobody asked about.

The two feed each other. A bug found by exploration becomes an automated test, so it never needs a human again: the night-rate boundaries that once slipped through are now a parametrized pytest test (see Unit Testing with pytest), and BUG-1051 will get one as soon as the business decides which rounding is right. Automation in turn frees people's time for the exploration that finds the next one.