How the Web Works

The web and the internet

People use "the internet" and "the web" as if they were the same thing. They are not. The internet is the global network of networks that moves IP packets between computers (see What Is a Network?). Email, video calls, online games, SSH sessions and DNS all run on it without being "the web".

The World Wide Web (the web) is one service on top of the internet: a system of documents and applications that are identified by addresses, fetched with the Hypertext Transfer Protocol (HTTP) and linked to each other. Tim Berners-Lee proposed it at CERN, the European particle physics laboratory near Geneva, in March 1989. By the end of 1990 he had written the first web server and browser, and in 1991 the software was made available outside CERN. Three ideas from that proposal still carry the web today:

• Addresses (URLs) that name any document on any server.
• A protocol (HTTP) to ask a server for a document and get it back.
• A document format (HTML, the Hypertext Markup Language) that can contain links to other documents.

APIs (application programming interfaces) for programs use the same three building blocks; they just return data instead of pages. That is why this topic covers both. See What Is an API?

The request-response model

The web is built on the client-server model, and HTTP adds a strict rule to it. In the request-response model, the client sends one request and the server sends back exactly one final response to it (possibly preceded by short informational 1xx responses). The server never speaks first: it cannot push a new page to a browser that did not ask for one. Everything a browser shows was requested by the browser.

That rule has consequences you will meet throughout this topic:

• A page that updates itself (a live outage map, a chat) has to keep asking, or switch to another technique such as WebSockets. See Webhooks and Real-Time APIs.
• Each request stands on its own. The server does not remember the previous one unless something like a cookie reminds it; see Cookies, Sessions and State.
• When something fails, there is always a request to look at and a response (or the lack of one) that tells you where it went wrong.

The journey of a page load

Suppose a customer of Kestrel Energy types https://www.example.com/outages into the address bar to check for a power cut. Before anything appears, the browser works through a fixed sequence of steps. Most take milliseconds, and most of them have nothing to do with HTML.

1. Parse the URL

The browser first decides whether you typed an address or a search term. If it is an address, it splits it into parts: the scheme (https), the host (www.example.com), an optional port and the path (/outages). A URL (Uniform Resource Locator) is the full address of one resource on the web: it says which protocol to use, which server to contact and what to ask that server for. The scheme also sets the default port: 443 for https, 80 for http. The parts of a URL, and how special characters travel inside it, are the subject of URLs, Encoding and Content Types.

2. Look up the name in DNS

Computers connect to IP addresses, not names. The browser asks the operating system, which asks its configured DNS resolver, for the address of www.example.com. Answers are cached at every level for as long as the record's time to live allows, so this step is often instant. How the Domain Name System (DNS) finds the answer is explained in DNS.

3. Open a TCP connection, then TLS

With the IP address in hand, the browser opens a Transmission Control Protocol (TCP) connection to port 443 using the three-way handshake. On top of that it runs the Transport Layer Security (TLS) handshake: the server presents its certificate, the browser checks that it is valid for www.example.com and signed by a trusted certificate authority, and both sides agree on encryption keys. Only then is the connection ready for HTTP. TLS itself is explained in PKI and Certificates; what it looks like from the browser's side is in HTTPS and TLS in Practice. (HTTP/3 replaces TCP with QUIC, which merges these two handshakes; see HTTP in Depth.)

4. Send the HTTP request

Now the browser sends a short text message: a method and path, then headers with extra information. Simplified, it looks like this:

GET /outages HTTP/1.1
Host: www.example.com
User-Agent: Mozilla/5.0 (X11; Linux x86_64) …
Accept: text/html,application/xhtml+xml
Accept-Language: en,nl;q=0.8
Cookie: theme=dark

Modern browsers usually speak HTTP/2 or HTTP/3, which send the same information in a compact binary form, but the meaning is identical.

5. The server does its work

For a static file the server just reads it from disk. For the outage page, the application behind the web server might check the session cookie, query a database for current outages, and fill an HTML template with the results. On a large site the request first passes a content delivery network, a load balancer or a reverse proxy (see Web Servers and Reverse Proxies). This is usually the slowest step, and it is invisible to the browser: all it sees is the time until the first byte arrives.

6. Receive the response

The server answers with a status code, headers and a body:

HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Content-Length: 5120
Cache-Control: no-cache

<!doctype html>
<html lang="en">
<head>
  <title>Current outages</title>
  <link rel="stylesheet" href="/css/site.css">
  <script src="/js/map.js" defer></script>
…

The status code (200, "OK") says whether the request worked; the Content-Type header tells the browser this is HTML in UTF-8. Both are covered in detail in HTTP in Depth.

7. Render, and fetch the rest

The browser parses the HTML into a tree of elements, applies the CSS (Cascading Style Sheets) rules, works out where every box goes (layout) and draws the pixels (paint). While parsing, it finds references to other resources: the stylesheet, the script, images, fonts. Each one is a new HTTP request, usually over the same connection. A typical news or shop page needs dozens of requests; you can see them all in the Network panel of the browser developer tools (see Browsers and the Same-Origin Policy).

Web pages and hyperlinks

A web page is a document written in HTML, delivered over HTTP and identified by its URL. What you see on screen is usually more than that one document: the HTML pulls in stylesheets for the look, scripts for behavior, and images, fonts and video. A "web application" such as a customer portal is a page whose scripts keep fetching data and changing what is shown, without loading a new page.

A hyperlink (link) is a reference in a document that the reader can follow to another resource. In HTML it is the a element with an href attribute:

<a href="https://www.example.com/outages">Current outages</a>
<a href="/contact">Contact us</a>
<a href="#map">Jump to the map</a>

The first link is absolute; the second is relative to the current server; the third points to a place inside the same page. Links are what make the web a web: anyone can link to any page without asking permission, and following a link is just another GET request. That openness is also why a broken link (a 404) is so common, and why links in email deserve a second look before you click.

What the browser does

A web browser is the program that fetches web resources on the user's behalf and turns them into an interactive page. Beyond the steps above, it carries a lot of responsibility:

• Networking: DNS lookups, connection reuse, HTTP/2 and HTTP/3, its own cache (see HTTP Caching).
• Rendering: parsing HTML and CSS, layout, painting, running JavaScript.
• State: cookies, local storage, saved passwords and the history.
• Security: checking certificates, keeping sites apart with the same-origin policy, isolating tabs in separate processes, warning about dangerous downloads.

Programs that call APIs (scripts, mobile apps, curl) are HTTP clients too, but they do only the networking part. They do not render anything, and they do not enforce browser security rules such as the same-origin policy. That difference explains many "works in Postman, fails in the browser" puzzles.

Example: one page, many requests

The script below plays both roles on your own machine. It starts a tiny web server on 127.0.0.1 (port 0 asks the operating system for any free port) that serves an HTML page and a stylesheet. Then it behaves like a very small browser: it fetches the page, looks through the HTML for <link rel="stylesheet"> with Python's html.parser, and fetches what it finds. The server writes down every path it was asked for.

import threading
from html.parser import HTMLParser
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.parse import urljoin
from urllib.request import urlopen

FILES = {
    "/": ("text/html; charset=utf-8",
          '<!doctype html><html><head><title>Outage map</title>'
          '<link rel="stylesheet" href="/style.css"></head>'
          '<body><h1>Current outages</h1><p>No known outages.</p></body></html>'),
    "/style.css": ("text/css; charset=utf-8",
                   "body { font-family: sans-serif; }\nh1 { font-size: 2rem; }\n"),
}
seen = []  # paths the server was asked for, in order


class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        seen.append(self.path)
        if self.path not in FILES:
            self.send_error(404)
            return
        content_type, text = FILES[self.path]
        body = text.encode("utf-8")
        self.send_response(200)
        self.send_header("Content-Type", content_type)
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, format, *args):
        pass  # keep the output clean


class StylesheetFinder(HTMLParser):
    """Collects the href of every <link rel="stylesheet">."""
    def __init__(self):
        super().__init__()
        self.stylesheets = []

    def handle_starttag(self, tag, attrs):
        attrs = dict(attrs)
        if tag == "link" and attrs.get("rel") == "stylesheet":
            self.stylesheets.append(attrs["href"])


def fetch(url):
    with urlopen(url) as response:
        body = response.read()
        print(f"GET {url.removeprefix(base)} -> {response.status}",
              response.headers["Content-Type"], f"({len(body)} bytes)")
        return body.decode("utf-8")


server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)  # port 0: any free port
threading.Thread(target=server.serve_forever, daemon=True).start()
base = f"http://127.0.0.1:{server.server_address[1]}"

page = fetch(base + "/")                 # 1. the HTML document
finder = StylesheetFinder()
finder.feed(page)                        # 2. "render": look for more resources
for href in finder.stylesheets:
    fetch(urljoin(base + "/", href))     # 3. one extra request per resource

print("Requests seen by the server:", seen)
server.shutdown()
server.server_close()

GET / -> 200 text/html; charset=utf-8 (168 bytes)
GET /style.css -> 200 text/css; charset=utf-8 (58 bytes)
Requests seen by the server: ['/', '/style.css']

Two requests for one page, and a real page would add scripts, images and fonts the same way. urljoin resolves the href against the page address exactly as a browser does, so a relative link such as css/site.css would work too.

What can go wrong at each step

Because a page load is a chain, "the website doesn't load" can break at any link. The error the user sees usually tells you which one. Work through the chain in order: there is no point debugging HTTP when DNS does not resolve.