Neural Networks

What a neural network is

A neural network is a model built from many small, identical calculating units — artificial neurons — connected in layers, where the output of one layer is the input of the next. The name comes from a loose analogy with nerve cells in the brain, and it should stay loose: a neural network is arithmetic on lists of numbers, not a simulated brain.

Everything goes in and comes out as numbers. A server's CPU and memory use, the pixels of an image, the words of a ticket (after they have been turned into numbers) go in; a probability, a category or the next word comes out. What the network "knows" is stored entirely in its weights and biases — the model parameters it learns during training.

The artificial neuron

An artificial neuron computes a weighted sum of its inputs, adds a bias, and passes the result through an activation function to produce one output number. Take a neuron that judges whether a server needs attention from two inputs, CPU use and memory use (as fractions from 0 to 1):

A weight is a learned number that sets how strongly one input influences a neuron's output: large and positive means "this input pushes the output up a lot", negative means it pushes the output down, near zero means it hardly matters. The bias is an extra learned number added to the sum; it shifts the point at which the neuron "switches on", regardless of the inputs. Here: 0.90 × 2.0 + 0.70 × 1.5 − 2.5 = 0.35.

The perceptron

The idea is old. In 1958 the psychologist Frank Rosenblatt published the perceptron: a single artificial neuron with a step activation (output 1 if the weighted sum is at least zero, otherwise 0) together with a simple rule for learning its weights from labeled examples. It could learn any problem where a straight line (or flat plane) separates the two classes — and nothing else, a limitation analyzed in Marvin Minsky and Seymour Papert's 1969 book Perceptrons. That limitation, and how layers overcome it, is the subject of the section on layers below.

Activation functions

An activation function is the function a neuron applies to its weighted sum to produce its output. Three classic ones:

• Step: 1 if the sum is at least 0, else 0. A hard yes/no, used by the original perceptron. Its slope is zero almost everywhere, which makes it useless for gradient descent (see below).
• Sigmoid: 1 / (1 + e^−z). A smooth S-curve from 0 to 1, so the output reads as a probability. Used in output neurons for yes/no questions; it is the same function as in logistic regression.
• ReLU (rectified linear unit): the sum itself if positive, otherwise 0. Very cheap to compute and it trains well in deep networks, so it (and close variants) is the default in hidden layers today.

import math


def step(z):
    return 1 if z >= 0 else 0


def sigmoid(z):
    return 1 / (1 + math.exp(-z))


def relu(z):
    return max(0.0, z)


# One neuron: is this server in trouble? Inputs are CPU and memory use (0-1).
inputs = [0.90, 0.70]
weights = [2.0, 1.5]
bias = -2.5

z = sum(x * w for x, w in zip(inputs, weights)) + bias
print(f"weighted sum z = {z:.2f}")
print(f"step: {step(z)}  sigmoid: {sigmoid(z):.3f}  ReLU: {relu(z):.2f}")

print()
print("    z  step  sigmoid  ReLU")
for z in (-4, -1, 0, 1, 4):
    print(f"{z:5}  {step(z):4}  {sigmoid(z):7.3f}  {relu(z):4.1f}")

weighted sum z = 0.35
step: 1  sigmoid: 0.587  ReLU: 0.35

    z  step  sigmoid  ReLU
   -4     0    0.018   0.0
   -1     0    0.269   0.0
    0     1    0.500   0.0
    1     1    0.731   1.0
    4     1    0.982   4.0

The table shows the character of each: step jumps, sigmoid bends smoothly and flattens out near 0 and 1, ReLU is zero and then a straight line. The flattening of the sigmoid matters later: where a curve is nearly flat, its slope is nearly zero, and learning slows to a crawl (see Deep Learning on vanishing gradients).

Layers, and why non-linearity matters

Neurons are arranged in layers. The input layer is just the input numbers; one or more hidden layers each take all outputs of the previous layer as their inputs; the output layer produces the answer. When every neuron is connected to every neuron in the next layer, the layer is called fully connected (or dense).

Why more than one neuron? Consider XOR ("exclusive or"): output 1 when exactly one of two inputs is 1. As an IT example: alert when exactly one of two redundant power supplies reports a fault — both fine is normal, both faulty is already a different, louder alarm. Plot the four cases on a square and the two "1" cases sit on opposite corners. No single straight line separates them, so no single neuron can learn XOR. Two hidden neurons can: one learns "at least one fault" (OR), the other "both faulty" (AND), and the output neuron combines them as "OR but not AND".

This only works because of the activation function. Without it, each layer is a weighted sum, and a weighted sum of weighted sums is still just one weighted sum — a hundred layers would collapse into a single straight-line model. The non-linear bend in each neuron (ReLU's corner, the sigmoid's S) is what lets stacked layers build curved, complicated decision boundaries out of simple pieces.

The loss function

To improve, a network needs a single number that says how wrong it currently is. A loss function measures the difference between the model's predictions and the correct answers as one number to be made as small as possible.

• For predicting a quantity (energy use in kWh), the usual loss is mean squared error: the average of (prediction − actual)². Squaring makes big misses count heavily.
• For yes/no and category questions, the usual loss is log loss (cross-entropy): −ln(probability the model gave to the correct answer). Saying 0.9 for a server that really needs attention costs −ln(0.9) ≈ 0.11; saying 0.1 costs −ln(0.1) ≈ 2.30. Confident and wrong is punished hardest.

The choice of loss function is a design decision: it is the definition of "good" the network will optimize, nothing more and nothing less. Language models, for example, are trained with log loss on "what is the next token?" — see Transformers and Large Language Models.

Gradient descent

Picture the loss as a landscape: every possible combination of weights is a location, and the height is the loss there. Training means finding a low point. You cannot see the whole landscape — a real network has millions of weights — but at your current position you can measure the slope. Gradient descent is the method of repeatedly nudging every weight a small step in the direction that makes the loss decrease fastest. The gradient is that slope, one number per weight: how much the loss changes when that weight changes slightly.

The learning rate is the setting that decides how big each step is: new weight = old weight − learning rate × gradient. It is chosen by people, not learned (such settings are called hyperparameters).

Too big a learning rate overshoots the valley and can make the loss explode; too small a learning rate creeps down so slowly that training takes forever or stalls on a flat stretch. In practice people try a few values (0.1, 0.01, 0.001 …) and often lower the rate as training proceeds. Real loss landscapes have many valleys and plateaus, not one neat bowl; gradient descent finds a good low point, not a guaranteed best one — and for large networks that turns out to be good enough.

One full pass through all the training data is an epoch. Small datasets often need many epochs (hundreds or more); large language models often see much of their training text only once or a few times. With big datasets the data is fed in batches (say 32 or 1,024 examples at a time), with one weight update per batch — this is called stochastic or mini-batch gradient descent.

Training a neuron in Python

The example below trains the server neuron from scratch on ten labeled servers. It starts with random weights (seeded, so the output is always the same), and in every epoch it computes the predictions, the log loss and the gradient, then takes one gradient descent step. For this neuron, the gradient of the log loss has a remarkably simple form: for each server, (prediction − label) × input — the error times how much that input contributed.

import math
import random

# (CPU %, memory %) -> needs attention? (1 = yes, 0 = no)
servers = [
    (35, 40, 0), (50, 45, 0), (20, 60, 0), (45, 70, 0), (60, 30, 0),
    (85, 80, 1), (95, 60, 1), (70, 92, 1), (90, 88, 1), (65, 85, 1),
]
data = [(cpu / 100, mem / 100, label) for cpu, mem, label in servers]


def predict(w1, w2, b, cpu, mem):
    z = w1 * cpu + w2 * mem + b          # weighted sum plus bias
    return 1 / (1 + math.exp(-z))         # sigmoid: a probability 0..1


random.seed(42)
w1, w2, b = (random.uniform(-0.5, 0.5) for _ in range(3))
learning_rate = 1.0

for epoch in range(1, 3001):
    g1 = g2 = gb = loss = 0.0
    for cpu, mem, label in data:
        p = predict(w1, w2, b, cpu, mem)
        loss -= label * math.log(p) + (1 - label) * math.log(1 - p)
        error = p - label                 # how far off, and in which direction
        g1 += error * cpu
        g2 += error * mem
        gb += error
    n = len(data)
    # Gradient descent: step each parameter against its gradient
    w1 -= learning_rate * g1 / n
    w2 -= learning_rate * g2 / n
    b -= learning_rate * gb / n
    if epoch in (1, 10, 100, 1000, 3000):
        print(f"epoch {epoch:4}: loss {loss / n:.3f}")

print(f"weights {w1:.2f} {w2:.2f}, bias {b:.2f}")
for cpu, mem in [(30, 35), (60, 65), (75, 75), (92, 95)]:
    p = predict(w1, w2, b, cpu / 100, mem / 100)
    print(f"CPU {cpu}%, memory {mem}% -> {p:.2f}")

epoch    1: loss 0.743
epoch   10: loss 0.611
epoch  100: loss 0.286
epoch 1000: loss 0.054
epoch 3000: loss 0.022
weights 16.01 11.39, bias -17.48
CPU 30%, memory 35% -> 0.00
CPU 60%, memory 65% -> 0.38
CPU 75%, memory 75% -> 0.96
CPU 92%, memory 95% -> 1.00

The loss falls quickly at first and then ever more slowly — the ball reaching the flat bottom of the valley. The learned weights say both inputs matter, CPU a bit more than memory, and the bias sets the tipping point. A server at 60% CPU and 65% memory gets 0.38: uncertain, as it should be, since it lies between the two groups. With a learning rate of 0.01 instead of 1.0, the same code ends at a loss of 0.488 after 3,000 epochs and gives the healthy 30%/35% server 0.35: "too small" in action.

This neuron is mathematically the same as logistic regression. A network is many of these, in layers, trained the same way. Real projects do not write this loop by hand: libraries such as PyTorch and TensorFlow (with its Keras interface) define layers in a few lines, compute all gradients automatically, and run the arithmetic on a GPU. The idea is exactly what you see here.

Backpropagation

For one neuron, the gradient is easy to write down. In a network with hidden layers, a weight deep inside affects the loss only indirectly: it changes a hidden neuron's output, which changes the next layer, which changes the prediction, which changes the loss. Backpropagation is the algorithm that computes the gradient of the loss for every weight in the network by working backward from the output, layer by layer, reusing the results of the layer after it.

The mathematics behind it is the chain rule, which in words says: if A affects B and B affects C, then how much A affects C is (how much A affects B) × (how much B affects C). So the network first runs forward to make a prediction and measure the loss, then runs backward: the output layer works out how much it contributed to the error, passes that blame back to the layer before, which uses it to work out the gradient of each of its own weights and passes a blame signal for its inputs further back. One backward pass gives every weight its gradient, at a cost of the same order as the forward pass (typically about twice) — which is what makes training networks with billions of weights feasible at all.

Backpropagation was popularized by David Rumelhart, Geoffrey Hinton and Ronald Williams in their 1986 Nature paper "Learning representations by back-propagating errors"; earlier forms of the method go back to Seppo Linnainmaa (1970) and Paul Werbos (1974). Hinton shared the 2024 Nobel Prize in Physics with John Hopfield for foundational work on machine learning with artificial neural networks.

What "billions of parameters" means

Every connection has a weight, and every neuron has a bias, so the number of parameters grows quickly with the size of the layers. A fully connected layer from a neurons to b neurons has a × b weights plus b biases:

def count_parameters(layer_sizes):
    total = 0
    for n_in, n_out in zip(layer_sizes, layer_sizes[1:]):
        weights = n_in * n_out           # every input connects to every neuron
        biases = n_out                   # one bias per neuron
        total += weights + biases
    return total


networks = {
    "single neuron (above)": [2, 1],
    "server health, one hidden layer": [2, 4, 1],
    "handwritten digits, 28x28 pixels": [784, 128, 64, 10],
}
for name, sizes in networks.items():
    print(f"{name}: {sizes} -> {count_parameters(sizes):,} parameters")

single neuron (above): [2, 1] -> 3 parameters
server health, one hidden layer: [2, 4, 1] -> 17 parameters
handwritten digits, 28x28 pixels: [784, 128, 64, 10] -> 109,386 parameters

A small digit recognizer already has over a hundred thousand parameters. GPT-3, described in 2020, had 175 billion. Each parameter is one number, stored in 2 bytes at 16-bit precision, so 175 billion parameters need about 350 GB just to hold the weights — more than most single GPUs hold. That is why large models are split across many accelerators, and why techniques such as quantization (storing weights with fewer bits) matter; see AI Infrastructure and Cloud Services. More parameters let a model store more patterns, but they also need more data, compute and energy to train, and size alone does not guarantee a better model.