URLs, Encoding and Content Types

URL and URI

You will see two terms used side by side. A URI (Uniform Resource Identifier) is any string that identifies a resource in the standard scheme:rest syntax. A URL is a URI that also tells you how to reach the resource, such as https://www.example.com/outages. Identifiers that only name something, such as urn:isbn:9780131103627 (a book) or mailto:support@example.com, are URIs too. In daily web work the difference rarely matters, and the WHATWG URL Standard simply uses "URL" for everything.

Two standards

RFC 3986 (2005) defines the generic URI syntax and is what many server-side libraries follow. The WHATWG URL Standard is a living standard written to match how browsers actually parse addresses, including the sloppy ones people type; it aims to replace RFC 3986 for the web. The two agree on normal URLs but differ on edge cases: WHATWG, for example, treats a backslash in https:\\example.com as a slash. When a security check (say, "only allow URLs on our own domain") parses a URL differently from the component that finally uses it, attackers can slip through the gap. Parse once, with one well-tested library, and validate the parsed result rather than the raw string.

The parts of a URL

https://ops@api.example.com:8443/v1/outages?status=open#map
^^^^^                                                        scheme
        ^^^                                                  userinfo
            ^^^^^^^^^^^^^^^                                  host
                            ^^^^                             port
                                ^^^^^^^^^^^                  path
                                            ^^^^^^^^^^^      query
                                                        ^^^  fragment

Scheme

The URL scheme is the part before the first colon. It says how to interpret the rest and, for network schemes, which protocol to use: https, http, wss for secure WebSockets, ftp, mailto, file. It also implies a default port (443 for https, 80 for http), so the port is normally left out.

Userinfo

An optional user name (and, historically, password) before an @. Do not use it: credentials in a URL end up in browser history, proxy logs and screenshots, and RFC 3986 calls the user:password form deprecated. It is also a phishing trick: in https://login.example.com@203.0.113.50/ the real host is 203.0.113.50; everything before the @ is just a user name.

Host

A domain name or an IP address (IPv6 in square brackets: https://[2001:db8::10]/). Host names are case-insensitive.

Port

The TCP or UDP port, only needed when it differs from the scheme's default.

Path

Which resource on that server, as segments separated by /. Whether paths are case-sensitive depends on the server; on most Linux-based servers they are.

Query

The query string is the part after ? and before #. By convention it holds name=value pairs separated by &, used for searches, filters and paging: ?status=open&page=2. The URL standards do not require that format; it is what HTML forms produce and what web frameworks expect.

Fragment

The URL fragment is the part after #. It points to a place within the resource, such as a section heading, and is handled entirely by the client: the browser never sends it to the server. You will not find it in server logs, and a server cannot redirect based on it. Single-page applications use it for client-side navigation for exactly that reason.

Percent-encoding

A URL may only contain a limited set of ASCII characters, and some of those have a special job: / separates path segments, ? starts the query, & and = structure it, # starts the fragment. Percent-encoding is the way to put any other character, or a special character used as plain data, into a URL: the text is first converted to UTF-8 bytes, and each byte that needs it is written as % followed by its two hexadecimal digits. A space becomes %20; é is two bytes in UTF-8 and becomes %C3%A9.

Python's urllib.parse module splits URLs and handles the encoding in both directions:

from urllib.parse import parse_qs, quote, unquote, urlencode, urlsplit

url = "https://api.example.com:8443/v1/outages?postcode=2511%20CV&status=open#map"
parts = urlsplit(url)
print("scheme:  ", parts.scheme)
print("host:    ", parts.hostname)
print("port:    ", parts.port)
print("path:    ", parts.path)
print("query:   ", parts.query)
print("fragment:", parts.fragment)
print("decoded: ", parse_qs(parts.query))

# Percent-encoding: text -> UTF-8 bytes -> %XX for every byte that needs it
print(quote("Café & Bar"))                # é is two bytes: %C3%A9
print(quote("Den Haag/Centrum", safe=""))  # safe="" encodes / as well
print(unquote("Caf%C3%A9%20%26%20Bar"))

# Form encoding (application/x-www-form-urlencoded): a space becomes +
query = urlencode({"postcode": "2511 CV", "note": "1+1=2"})
print(query)
print(parse_qs(query))
print(parse_qs("note=1+1=2"))   # a + that was not encoded turns into a space

scheme:   https
host:     api.example.com
port:     8443
path:     /v1/outages
query:    postcode=2511%20CV&status=open
fragment: map
decoded:  {'postcode': ['2511 CV'], 'status': ['open']}
Caf%C3%A9%20%26%20Bar
Den%20Haag%2FCentrum
Café & Bar
postcode=2511+CV&note=1%2B1%3D2
{'postcode': ['2511 CV'], 'note': ['1+1=2']}
{'note': ['1 1=2']}

urlsplit does not decode anything: the query is returned exactly as written, and parse_qs decodes it into a dictionary of lists (a name can appear more than once). quote keeps / by default because it assumes you are encoding a whole path; pass safe="" when the value is a single path segment that may contain a slash.

Form encoding: where + means space

Form encoding (application/x-www-form-urlencoded) is the format in which an HTML form sends its fields, either in the query string of a GET or in the body of a POST: name=value pairs joined with &, percent-encoded, with one difference from ordinary percent-encoding: a space becomes +. That is why urlencode above produced postcode=2511+CV, and why a literal plus sign has to travel as %2B.

The last line of the example shows the classic bug: a client that builds a query by hand sends note=1+1=2, and the server, correctly applying form decoding, reads "1 1=2". Phone numbers such as +31 70 … and email addresses with a plus tag suffer the same fate. The fix is always the same: let a library build the query (urlencode, URLSearchParams in JavaScript) instead of gluing strings together.

International domain names

Host names in DNS are limited to ASCII letters, digits and hyphens. To allow names such as münchen.de, internationalized domain names (IDNs) are converted to an ASCII form with Punycode (RFC 3492) and the prefix xn--: münchen.de becomes xn--mnchen-3ya.de. Browsers show the readable form in the address bar, but DNS queries, certificates and server logs contain the xn-- form. Because letters from different scripts can look identical (a Cyrillic "а" and a Latin "a"), browsers have rules for when to display the Punycode form instead, to make look-alike phishing domains harder. In Python, "münchen.de".encode("idna") gives b'xn--mnchen-3ya.de', but note that this built-in codec implements the older IDNA 2003 rules; current rules (IDNA2008, RFC 5890 and following) differ for a few characters.

Media types and Content-Type

A body is just bytes. The receiver needs to be told what those bytes are, and the URL's file extension is no reliable guide. A media type (also called a MIME type, after the Multipurpose Internet Mail Extensions it comes from) is a standard label for a data format, written as type/subtype with optional parameters, such as text/html; charset=utf-8. The Content-Type header carries it in both requests and responses. Official types are registered with IANA (the Internet Assigned Numbers Authority).

A server that sends the wrong type causes confusing failures: an API returning JSON as text/html makes some clients refuse to parse it, and a script served as text/plain is blocked by browsers when the X-Content-Type-Options: nosniff header is set (a recommended security header that stops browsers from guessing types).

Character sets

For text, the bytes only make sense if you know the character encoding. The charset parameter of Content-Type states it. Today the answer should always be UTF-8: send text/html; charset=utf-8 and also put <meta charset="utf-8"> at the top of the HTML, so the page still decodes correctly when saved to disk. JSON has no charset parameter at all, because JSON exchanged between systems must be UTF-8 (RFC 8259); adding ; charset=utf-8 to application/json does no harm but has no effect.

Content negotiation

Content negotiation is the mechanism by which one URL can serve different representations of the same resource, chosen from preferences the client sends in request headers:

• Accept: which media types, for example application/json or text/html.
• Accept-Language: which languages, for example nl, en;q=0.8.
• Accept-Encoding: which compression, for example gzip, br. The server marks its choice with Content-Encoding.

Each value can carry a quality weight q between 0 and 1 (default 1); q=0 means "not this". If the server has nothing acceptable, it may answer 406 Not Acceptable, though many servers simply send their default. The response should include a Vary header naming the request headers that influenced the choice, so caches do not hand a JSON response to a client that asked for text (see HTTP Caching).

The script below runs a small outage API that returns JSON or plain text depending on Accept. Its preferred() function is deliberately simple: it handles q-values and */*, but not wildcards such as text/*.

import json
import threading
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.error import HTTPError
from urllib.request import Request, urlopen

OUTAGE = {"id": 4711, "area": "Den Haag Centrum", "status": "investigating"}


def preferred(accept_header, available):
    """Return the first available type the client accepts, best q-value first."""
    choices = []
    for position, item in enumerate(accept_header.split(",")):
        media_type, *params = [p.strip() for p in item.split(";")]
        q = 1.0
        for param in params:
            if param.startswith("q="):
                q = float(param[2:])
        if q > 0:                       # q=0 means "not this one"
            choices.append((-q, position, media_type))
    for _, _, media_type in sorted(choices):
        if media_type == "*/*":
            return available[0]
        if media_type in available:
            return media_type
    return None


class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        accept = self.headers.get("Accept", "*/*")
        chosen = preferred(accept, ["application/json", "text/plain"])
        if chosen == "application/json":
            body = json.dumps(OUTAGE).encode("utf-8")
        elif chosen == "text/plain":
            body = "Outage {id} in {area}: {status}\n".format(**OUTAGE).encode("utf-8")
        else:
            self.send_response(406)  # Not Acceptable
            self.send_header("Content-Length", "0")
            self.send_header("Vary", "Accept")
            self.end_headers()
            return
        self.send_response(200)
        if chosen == "text/plain":
            chosen += "; charset=utf-8"     # JSON is always UTF-8: no charset needed
        self.send_header("Content-Type", chosen)
        self.send_header("Content-Length", str(len(body)))
        self.send_header("Vary", "Accept")  # tell caches the answer depends on Accept
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, format, *args):
        pass


server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
threading.Thread(target=server.serve_forever, daemon=True).start()
url = f"http://127.0.0.1:{server.server_address[1]}/outages/4711"

for accept in ["application/json", "text/plain",
               "text/html, text/plain;q=0.5, application/json;q=0.9",
               "application/xml"]:
    try:
        with urlopen(Request(url, headers={"Accept": accept})) as response:
            print(f"Accept: {accept}")
            print(f"  {response.status} {response.headers['Content-Type']}")
            print(f"  {response.read().decode('utf-8').strip()}")
    except HTTPError as error:
        print(f"Accept: {accept}")
        print(f"  {error.code} {error.reason}")

server.shutdown()
server.server_close()

Accept: application/json
  200 application/json
  {"id": 4711, "area": "Den Haag Centrum", "status": "investigating"}
Accept: text/plain
  200 text/plain; charset=utf-8
  Outage 4711 in Den Haag Centrum: investigating
Accept: text/html, text/plain;q=0.5, application/json;q=0.9
  200 application/json
  {"id": 4711, "area": "Den Haag Centrum", "status": "investigating"}
Accept: application/xml
  406 Not Acceptable

The third request is what a browser-like client might send: it prefers HTML, which this API does not offer, and among the rest it weighs JSON (0.9) above plain text (0.5). urlopen raises HTTPError for the 406, which is why the last case is caught separately. (http.server is fine for experiments like this; its documentation warns against using it in production.)

Common mistakes

Most encoding bugs come down to text and bytes being converted twice, or once with the wrong rules:

from urllib.parse import quote

once = quote("2511 CV")
print(once)
print(quote(once))        # encoded twice: % itself becomes %25

text = "Café Noord"
data = text.encode("utf-8")          # what the server sends
print(data)
print(data.decode("utf-8"))          # read with the right charset
print(data.decode("latin-1"))        # read as Latin-1 (ISO-8859-1)

2511%20CV
2511%2520CV
b'Caf\xc3\xa9 Noord'
Café Noord
Café Noord

• Double encoding. A value that is already percent-encoded is encoded again, so %20 becomes %2520 and the server sees the literal text "2511%20CV". It typically happens when code builds a URL by hand and then passes it to a library that encodes as well. Encode exactly once, at the moment you build the URL.
• Charset mismatch. The UTF-8 bytes for é (C3 A9) read as Latin-1 show as two characters, é. This mojibake means the bytes are fine but the label is wrong or missing: a missing charset, a database connection set to Latin-1, or a CSV export opened with the wrong encoding.
• Wrong Content-Type on requests. Sending a JSON body with Content-Type: application/x-www-form-urlencoded (or none) makes many frameworks ignore it or report "missing field". A 400 or 415 Unsupported Media Type on a request that looks right deserves a look at this header first.
• The lost plus sign. A + in a query that was not encoded as %2B arrives as a space.
• Trusting the extension. A file called report.pdf can contain anything. Servers decide by Content-Type and, for uploads, by checking the content itself.