JSON and Other Data Formats

JSON in depth

JSON (JavaScript Object Notation) is the default data format of web APIs. It started as a subset of JavaScript's syntax for object literals, but it is language-independent: every mainstream language can read and write it. The current specification is RFC 8259 (2017), an Internet Standard that matches the ECMA-404 standard. Its whole grammar fits on one page, which is a large part of its success.

The six kinds of values

{
  "meter": "KE-55012",
  "kwh": 1234.5,
  "estimated": false,
  "note": null,
  "tariffs": ["day", "night"],
  "address": { "city": "Den Haag", "postcode": "2511 AA" }
}

That is all. There is no date type, no binary type, no comment syntax and no way to reference another part of the document. Everything else has to be agreed between sender and receiver — which is where most JSON problems come from.

Common JSON pitfalls

Syntax: comments, commas and quotes

• No comments. // note or /* … */ make the document invalid. Some tools accept "JSON with comments" (such as VS Code's settings files), but that is a different, non-standard format. If a configuration file needs comments, YAML or TOML is a better choice.
• No trailing commas. [1, 2, 3,] and {"a": 1,} are invalid, although JavaScript and Python accept them in their own code.
• Double quotes only, for strings and for names. A Python dictionary printed with print() uses single quotes and True/None, so it is not JSON — use json.dumps().

Numbers: big integers, floats and money

JSON itself puts no limit on numbers, but RFC 8259 warns that many implementations store them as IEEE 754 double-precision floats — JavaScript does this for every number. A double holds integers exactly only up to 2⁵³ − 1 (9,007,199,254,740,991). A 64-bit database ID such as 12345678901234567890 silently becomes 12345678901234567000 in a browser. The usual fix: send large IDs as strings.

Decimals have the opposite problem: 0.1 cannot be stored exactly in binary floating point, so sums drift (see floating-point numbers). For money, send an integer number of cents (8450) or a decimal string ("84.50") together with a currency code. Also note that NaN and Infinity are not valid JSON, although Python's json.dumps() writes them unless you pass allow_nan=False.

Dates and times: ISO 8601

Because JSON has no date type, APIs send dates as strings, and the format must be agreed. The standard choice is ISO 8601, the international standard for writing dates and times with the largest unit first: 2026-03-02 for a date, 2026-03-02T09:30:00Z or 2026-03-02T10:30:00+01:00 for a moment in time. Internet protocols usually use RFC 3339, a strict profile of ISO 8601. Its advantages: no confusion between 03/02 and 02/03, strings sort in time order, and every language can parse it.

Always include the time zone — Z for UTC, or an offset. 2026-03-02T09:30:00 without one is a "local time" nobody can interpret reliably. Many APIs simply send everything in UTC and let the client convert for display.

Null versus missing

{"note": null} and {} are different documents. In an update (PATCH), many APIs read a missing field as "leave it unchanged" and null as "clear this field" — JSON Merge Patch works this way. Some languages and libraries blur the two (both become "no value"), so an API should document which it means, and a client should not send null for fields it does not want to touch.

Text encoding: UTF-8 only

JSON exchanged between systems must be encoded as UTF-8, and a parser may reject anything else. Inside strings, any character may also be written as an escape such as \u00fc (ü). Both forms mean the same text. Mojibake such as "Zürich" in a JSON response means a program decoded UTF-8 bytes as Latin-1 or Windows-1252 somewhere along the way — the JSON was fine; the reader was wrong.

Serialization and parsing

A program works with objects in memory: dictionaries, lists, dates, class instances. A network connection or a file carries only bytes. Two conversions bridge the gap:

• Serialization turns in-memory data into a sequence of characters or bytes in an agreed format, so it can be sent or stored. Python's json.dumps() does this for JSON. It is also called encoding or marshaling.
• Parsing reads text or bytes, checks them against the format's grammar and builds data structures from them. json.loads() parses JSON; if the text breaks the grammar, the parser stops with an error that says where. Turning the parsed structures back into program objects is called deserialization.

Every API call does both, on both sides. In the outage API on What Is an API?, the server serializes a Python list into JSON text and the client parses it back. The example below shows the details that matter in practice: default= for types JSON does not know (here a datetime, written as ISO 8601), sort_keys for stable output, ensure_ascii for non-ASCII text, and a parse error.

import json
from datetime import datetime, timezone

reading = {
    "meter": "KE-55012",
    "city": "Den Haag",
    "kwh": 1234.5,
    "estimated": False,
    "note": None,
    "read_at": datetime(2026, 3, 2, 9, 30, tzinfo=timezone.utc),
}


def to_json(value):
    """Called by json.dumps for types it does not know."""
    if isinstance(value, datetime):
        return value.isoformat()  # ISO 8601 text
    raise TypeError(f"{type(value).__name__} is not JSON serializable")


# Serialize: Python objects -> JSON text
text = json.dumps(reading, default=to_json, sort_keys=True)
print(text)
print(json.dumps({"city": "Zürich"}))                      # ASCII-only escapes
print(json.dumps({"city": "Zürich"}, ensure_ascii=False))  # real UTF-8 text

# Parse: JSON text -> Python objects
data = json.loads(text)
print(type(data["kwh"]).__name__, type(data["read_at"]).__name__, data["note"])
print(datetime.fromisoformat(data["read_at"]))

# Broken JSON: single quotes, as in a pasted Python dict
try:
    json.loads("{'meter': 'KE-55012'}")
except json.JSONDecodeError as error:
    print("JSONDecodeError:", error)

{"city": "Den Haag", "estimated": false, "kwh": 1234.5, "meter": "KE-55012", "note": null, "read_at": "2026-03-02T09:30:00+00:00"}
{"city": "Z\u00fcrich"}
{"city": "Zürich"}
float str None
2026-03-02 09:30:00+00:00
JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)

• Without default=to_json, json.dumps() raises TypeError: Object of type datetime is not JSON serializable — the most common JSON error in Python.
• After parsing, read_at is a str, not a date. The receiver must know the field holds ISO 8601 and convert it itself: the format carries no types beyond its own six.
• ensure_ascii=True (the default) writes \u00fc; ensure_ascii=False writes the real character. Both are valid JSON for the same text.
• The parse error gives a line, column and character position. For a trailing comma the message is similar; newer Python versions word it more specifically.

Validating JSON with JSON Schema

Parsing tells you the text is valid JSON. It does not tell you that the data is right: that id is present, that status is one of three values. JSON Schema is a vocabulary, itself written in JSON, for describing what a valid document looks like: its type, required properties, allowed values, ranges and patterns. The current version is 2020-12. OpenAPI 3.1 and later use it to describe request and response bodies, and editors use it to check configuration files as you type.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://api.example.com/schemas/outage.json",
  "title": "Outage report",
  "type": "object",
  "required": ["id", "area", "status"],
  "properties": {
    "id": { "type": "integer", "minimum": 1 },
    "area": { "type": "string", "minLength": 1 },
    "status": {
      "type": "string",
      "enum": ["investigating", "repairing", "restored"]
    },
    "customers": { "type": "integer", "minimum": 0 }
  },
  "additionalProperties": false
}

$schema names the JSON Schema version; $id gives this schema its own identifier. required lists the properties that must be present, properties describes each one, and "additionalProperties": false rejects anything not listed — useful against typos and against clients slipping in fields they should not set.

In real projects you use a validator library (for Python, the third-party jsonschema package). To see what such a library does, here is a hand-written mini-validator for a subset of the keywords — type, required, properties, enum and additionalProperties — applied to three outage reports:

import json

SCHEMA = {
    "type": "object",
    "required": ["id", "area", "status"],
    "properties": {
        "id": {"type": "integer"},
        "area": {"type": "string"},
        "status": {"type": "string", "enum": ["investigating", "repairing", "restored"]},
        "customers": {"type": "integer"},
    },
    "additionalProperties": False,
}

TYPES = {
    "string": lambda v: isinstance(v, str),
    "boolean": lambda v: isinstance(v, bool),
    # bool is a subclass of int in Python, but true is not a JSON integer;
    # JSON Schema counts 2.0 as an integer, 2.5 not
    "integer": lambda v: (isinstance(v, int) and not isinstance(v, bool))
    or (isinstance(v, float) and v.is_integer()),
    "object": lambda v: isinstance(v, dict),
}


def validate(document, schema):
    """Check a small subset of JSON Schema; return a list of problems."""
    if not TYPES[schema["type"]](document):
        return [f"expected {schema['type']}"]
    problems = []
    for name in schema.get("required", []):
        if name not in document:
            problems.append(f"{name}: is required")
    for name, value in document.items():
        rule = schema["properties"].get(name)
        if rule is None:
            if schema.get("additionalProperties") is False:
                problems.append(f"{name}: is not allowed")
            continue
        if not TYPES[rule["type"]](value):
            problems.append(f"{name}: expected {rule['type']}, got {json.dumps(value)}")
        elif "enum" in rule and value not in rule["enum"]:
            problems.append(f"{name}: must be one of {rule['enum']}")
    return problems


reports = [
    '{"id": 101, "area": "Riverside", "status": "repairing", "customers": 240}',
    '{"id": "101", "area": "Riverside", "status": "fixed"}',
    '{"area": "Old Harbor", "status": "restored", "customers": true, "eta": "12:00"}',
]
for text in reports:
    problems = validate(json.loads(text), SCHEMA)
    print("valid" if not problems else "invalid")
    for problem in problems:
        print("  -", problem)

valid
invalid
  - id: expected integer, got "101"
  - status: must be one of ['investigating', 'repairing', 'restored']
invalid
  - id: is required
  - customers: expected integer, got true
  - eta: is not allowed

Note the subtle rules a validator must get right: in Python, True is an instance of int, but JSON true is not an integer; and JSON Schema counts 2.0 as an integer. A good validator also reports all problems at once, so a client can fix its request in one go. Validate at the edge — when data enters your system — and answer invalid requests with a clear 400 or 422; see input validation.

XML

XML (Extensible Markup Language) is a W3C markup language for structured data. Before JSON took over, it was the format of web services, and it is still everywhere in enterprise IT: SOAP messages, SAML assertions in single sign-on, RSS and Atom feeds, Office documents (a .docx file is a zip archive of XML files), Maven and .NET project files, and many device and application configurations.

An XML document is a tree of elements, each with a start and end tag. Elements can have attributes and contain text or other elements. Namespaces — a URI bound to a prefix or set as the default with xmlns — keep element names from different vocabularies apart.

Python's standard library reads XML with xml.etree.ElementTree:

import xml.etree.ElementTree as ET

document = """<?xml version="1.0" encoding="UTF-8"?>
<outages xmlns="https://api.example.com/ns/outages">
  <outage id="101" status="repairing">
    <area>Riverside</area>
    <customers>240</customers>
  </outage>
  <outage id="102" status="restored">
    <area>Old Harbor &amp; Docks</area>
    <customers>35</customers>
  </outage>
</outages>"""

ns = {"o": "https://api.example.com/ns/outages"}
root = ET.fromstring(document)
print(root.tag)  # the namespace is part of the name

for outage in root.findall("o:outage", ns):
    area = outage.find("o:area", ns).text  # element text
    customers = int(outage.find("o:customers", ns).text)  # text: convert it yourself
    print(outage.get("id"), outage.get("status"), area, customers)

try:
    ET.fromstring("<outage><area>Riverside</outage>")
except ET.ParseError as error:
    print("ParseError:", error)

{https://api.example.com/ns/outages}outages
101 repairing Riverside 240
102 restored Old Harbor & Docks 35
ParseError: mismatched tag: line 1, column 25

• With a default namespace, the element names include it in braces: {https://api.example.com/ns/outages}outages. Searching for a plain outage finds nothing — the most common surprise with XML namespaces. The ns dictionary maps a short prefix to the URI for searching.
• Everything in XML is text. 240 must be converted with int(); only a schema (XSD, XML Schema Definition) can say it is a number.
• Special characters are escaped as entities: & in the document becomes & after parsing.
• XML parsers are strict: a missing end tag stops the parse with a position, just like JSON.

Compared with JSON, XML is more verbose (every name appears twice) and has more ways to express the same thing (attribute or child element?). In return it has mature schemas, namespaces, digital signatures (XML Signature, used by SAML) and transformation tools.

YAML

YAML is the format of files that people write and read: CI/CD pipelines, Kubernetes manifests, Ansible playbooks, Docker Compose files and OpenAPI descriptions. Structure comes from indentation, comments start with #, and quotes are usually optional. The current specification is YAML 1.2.2 (2021), a revision of YAML 1.2. YAML 1.2 is designed as a superset of JSON, so most JSON documents are valid YAML too.

# Outage notification settings
service: outage-api
regions:
  - name: west
    notify: [sms, email]
  - name: east
    notify: [email]
retry:
  attempts: 3
  backoff_seconds: 5

YAML pitfalls

• Indentation is structure. Two spaces more or less moves a key to another parent, and the file may still be valid YAML — just with a different meaning. Tabs are not allowed for indentation.
• Unquoted values are guessed. YAML decides the type of a plain value from how it looks. version: 1.10 becomes the number 1.1, and a postcode or account number with a leading zero may lose it.
• The "Norway problem". YAML 1.1 treated yes, no, on, off, y and n (in several spellings) as booleans. A list of country codes with NO for Norway turned into false. YAML 1.2's core schema accepts only true and false (in any of three capitalizations: true, True, TRUE) as booleans, but parsers that follow YAML 1.1 rules are still widely used — PyYAML, for one (PyYAML leaves y and n as strings). More YAML 1.1 surprises, such as octal file modes and 22:22 read as a number, and how Ansible reads them: Configuration Languages.

country: NO
version: 1.10

{'country': False, 'version': 1.1}

The fix for all of these is the same: quote strings that could be read as something else (country: "NO", version: "1.10"), and validate the file against a schema in your pipeline.

CSV and Protocol Buffers

CSV

CSV (comma-separated values) is a plain table: one record per line, fields separated by commas, an optional header line. RFC 4180 describes the common form, including double quotes around fields that contain commas, quotes or line breaks. APIs offer it for exports and reports because spreadsheets open it directly. Its limits: everything is text, there is no nesting, and the delimiter varies — Excel in a Dutch or German locale writes semicolons, because the comma is the decimal separator there. Read CSV with a real CSV parser (see CSV files in Python), never with split(","). The traps of CSV in data work (types, leading zeros, dates, encodings) are on CSV, JSON and Parquet.

Protocol Buffers

Protocol Buffers (protobuf), developed at Google, is a binary serialization format with a schema written first, in a .proto file. A compiler generates code for each language from that schema; the generated code serializes messages into compact bytes that identify fields by number instead of by name. The result is smaller and faster to process than JSON, and both sides are guaranteed to agree on types — but you cannot make sense of a message without the schema: field numbers and wire types are visible, names and meaning are not. Protocol Buffers are the default format of gRPC; see GraphQL and gRPC.

The same outage report as a Protocol Buffers message definition (proto3 syntax):

syntax = "proto3";

message Outage {
  int32 id = 1;
  string area = 2;
  string status = 3;
  int32 customers = 4;
}

The numbers are field tags, not values. They are what travels over the wire, so they must never be changed or reused once the schema is in use — a new field gets a new number. That rule is what lets old and new versions of a service talk to each other.

The formats compared

For a new web API, JSON is the safe default. Choose another format when there is a concrete reason: an existing standard (SAML, SOAP), files that people maintain by hand (YAML), spreadsheet users (CSV), or measured performance needs between your own services (Protocol Buffers).