Deep Learning

What "deep" means

Deep learning is machine learning with neural networks that have many layers between input and output — from a handful to hundreds. The building blocks are exactly those of Neural Networks: weighted sums, activation functions, a loss function, gradient descent and backpropagation. "Deep" refers only to the number of layers.

Why does depth help? Each layer works on the output of the one before, so the network can build up a hierarchy of features. In an image model, the first layers react to edges and color changes, middle layers combine edges into corners, textures and shapes, and later layers combine those into "wheel", "face" or "keyboard". In classic machine learning a person usually designs the features (the length of an email, the number of links); a deep network learns its own features straight from raw pixels, sound samples or text. That is its main strength, and also why it needs much more data and computation.

Deep learning is a subset of machine learning, which in turn is a subset of AI; see What Is AI? for the bigger picture.

Why it took off after 2012

Neural networks are old — the perceptron dates from 1958 and backpropagation was popularized in 1986 — and Yann LeCun and colleagues described a successful convolutional network for reading handwritten digits (LeNet-5) in 1998. Yet for years other methods usually won. Three things changed by around 2012:

• Data. The internet produced huge amounts of images and text, and projects such as ImageNet labeled more than a million images in a thousand categories for its yearly recognition competition (ILSVRC, the ImageNet Large Scale Visual Recognition Challenge).
• Compute. Graphics cards, built to do millions of small multiplications in parallel for games, turned out to be exactly what neural network training needs. A GPU made training many times faster than a CPU.
• Training tricks. ReLU activations, better weight initialization and dropout (randomly switching off neurons during training against overfitting) made deep networks train reliably.

In 2012 AlexNet — by Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton, trained on two GPUs — won ILSVRC by a large margin. It is widely seen as the start of the deep-learning boom: within a few years deep networks led in image recognition, speech recognition and translation, and in 2016 DeepMind's AlphaGo, which combined deep networks with search and reinforcement learning, beat Lee Sedol at Go.

Convolutional neural networks

An ordinary fully connected layer would give every pixel of a photo its own weight for every neuron — millions of weights, and a cat in the top-left corner would be a completely different pattern from the same cat in the bottom-right. A convolutional neural network (CNN) instead slides small learned filters over the image, so the same few weights detect the same pattern wherever it appears.

A filter (or kernel) is a tiny grid of weights, typically 3 × 3. At each position it is laid on a patch of the image; the pixel values under it are multiplied by its weights and added up, giving one number. Sliding it over the whole image gives a new grid, a feature map, that is high wherever the pattern occurs. The example below applies a hand-made vertical-edge filter to a 6 × 6 image of brightness values:

# A 6x6 grayscale "image": dark on the left (0), bright on the right (9)
image = [
    [0, 0, 0, 9, 9, 9],
    [0, 0, 0, 9, 9, 9],
    [0, 0, 0, 9, 9, 9],
    [0, 0, 9, 9, 9, 9],
    [0, 0, 9, 9, 9, 9],
    [0, 0, 9, 9, 9, 9],
]

# 3x3 filter that responds to "dark on the left, bright on the right"
kernel = [
    [-1, 0, 1],
    [-1, 0, 1],
    [-1, 0, 1],
]


def convolve(image, kernel):
    size = len(kernel)
    rows = len(image) - size + 1
    cols = len(image[0]) - size + 1
    result = []
    for r in range(rows):
        row = []
        for c in range(cols):
            # Lay the filter on the image at (r, c): multiply and add up
            total = 0
            for i in range(size):
                for j in range(size):
                    total += image[r + i][c + j] * kernel[i][j]
            row.append(total)
        result.append(row)
    return result


for row in convolve(image, kernel):
    print(" ".join(f"{value:3}" for value in row))

  0  27  27   0
  9  27  18   0
 18  27   9   0
 27  27   0   0

The 6 × 6 image becomes a 4 × 4 feature map (a 3 × 3 filter fits in four positions per row). Flat areas, all dark or all bright, give 0; the large values trace exactly where the edge runs, including the step where it shifts one column to the left halfway down. In a real CNN nobody writes the filter values: each layer has dozens or hundreds of filters whose weights are learned by backpropagation, and the network discovers for itself that edge detectors are useful. (Strictly, what CNNs compute is called cross-correlation; the name "convolution" stuck anyway.)

Between convolution layers, a pooling layer usually shrinks the feature maps — for example by keeping only the largest value in each 2 × 2 block — so later layers see a coarser, bigger-picture view and the network becomes less sensitive to small shifts. At the end, one or more fully connected layers turn the final features into class probabilities.

CNNs power image classification, object detection, medical image analysis and the reading of scanned forms; see Vision, Speech and Multimodal AI for those applications. Transformers have since become strong competitors in vision too, but CNNs remain widely used because they are efficient and well understood.

Recurrent networks and LSTM

Text, speech, log streams and sensor readings are sequences: the order matters, and the length varies. A recurrent neural network (RNN) processes a sequence one element at a time and carries a hidden state — a list of numbers acting as a running memory — from each step to the next, so earlier elements can influence how later ones are handled. Jeffrey Elman's 1990 paper "Finding structure in time" is a classic early description.

Plain RNNs struggle with long sequences because of vanishing gradients. Backpropagation through many time steps multiplies many small slopes together (remember the flat ends of the sigmoid), so the learning signal from step 100 back to step 1 shrinks to almost nothing. The network effectively forgets anything more than a few steps back.

Long short-term memory (LSTM), introduced by Sepp Hochreiter and Jürgen Schmidhuber in 1997, is a kind of recurrent network whose units have a separate memory cell and learned gates that control what is stored, kept and output (the forget gate was added in 2000). Because information can pass along the memory cell largely unchanged, the gradient survives over many more steps. In the decade before transformers, LSTMs were the workhorse for speech recognition, translation and text prediction.

For most language tasks, recurrent networks have since been superseded by the transformer (2017). An RNN must read word 1 before word 2 before word 3, which is slow to train on GPUs; a transformer processes all positions at once and lets every word look directly at every other word through attention. See Transformers and Large Language Models. RNNs and LSTMs are still used for some time-series and on-device tasks where their small size helps.

Training at scale

Deep networks are trained exactly like the single neuron on Neural Networks, only with far more of everything:

• Hardware. Training is dominated by large matrix multiplications, which GPUs and other AI accelerators do massively in parallel. Large models are spread over many accelerators connected by very fast networks. See AI Infrastructure and Cloud Services.
• Batches. The data is fed in batches of examples; each batch gives one gradient descent step. Larger batches use the hardware better but need more memory.
• Time. A small image model can be trained in hours on one GPU; large language models train for weeks to months on thousands of accelerators. Training is restarted from saved checkpoints when hardware fails, which at that scale happens regularly.
• Cost and energy. Accelerator time is expensive, whether bought or rented in the cloud (see Cloud Costs and FinOps), and it uses a lot of electricity. The International Energy Agency estimated that data centers used about 415 TWh in 2024, around 1.5% of the world's electricity, with AI the most important driver of growth.

Training happens once (or occasionally); inference — using the trained model — happens every time someone asks a question, so for a popular model the running cost over its lifetime can rival or exceed the training cost.

Transfer learning and fine-tuning

Few organizations can afford to train a large network from scratch, and few have millions of labeled examples. Fortunately they rarely need to. Transfer learning means reusing a model trained on one task as the starting point for a different but related task, so that what it already learned — edge and shape detectors, or knowledge of language — carries over.

Suppose you want to sort photos of server-room equipment into "rack", "patch panel", "UPS" and "cable mess", and you have 400 labeled photos. Training a CNN from scratch on 400 photos would overfit badly. Instead you take a CNN pretrained on ImageNet, replace only its final layer with a new one that has four outputs, and train on your photos. The early layers already know edges and shapes; only the last part has to learn your categories.

Fine-tuning is continuing the training of a pretrained model on a smaller, task-specific dataset, so that its weights adapt to the new task. Options range from training only the new final layer (the rest "frozen") to updating all weights with a low learning rate, so the useful knowledge is adjusted rather than overwritten. Parameter-efficient methods such as LoRA (low-rank adaptation) train only a small set of extra weights, which makes fine-tuning even large language models affordable.

For large language models, fine-tuning is how a general model that predicts text becomes a helpful assistant, or learns a company's ticket categories and writing style — see fine-tuning LLMs. Fine-tuning is not always the right tool: to give a model access to current facts, such as your knowledge base, retrieval is usually cheaper and easier to keep up to date — see Embeddings and Retrieval.