Derivatives and Rates of Change

Rates of change

A smart meter at a Kestrel Energy customer counts the total energy used, in kilowatt-hours (kWh). The register only ever goes up. Four readings, one per hour:

The register itself is not very interesting; how fast it climbs is. Between hour 0 and hour 1 it rose 1.2 kWh in 1 hour: 1.2 kWh per hour. Energy per unit of time is power, and kWh per hour is simply kilowatts (kW), so the household used 1.2 kW on average in that hour.

A rate of change is the change in one quantity divided by the change in another quantity it depends on, usually time. It answers "how much more per unit?". Speed is the classic example: the change in distance divided by the change in time. In IT you meet rates everywhere: bytes per second from a byte counter, requests per second from a request counter, gigabytes per month of storage growth.

# Meter readings (hour, kWh). Average rate between readings = power in kW.
readings = [(0, 1000.0), (1, 1001.2), (2, 1003.0), (3, 1003.5)]
for (t0, e0), (t1, e1) in zip(readings, readings[1:]):
    print(f"{t0}-{t1} h: {(e1 - e0) / (t1 - t0):.1f} kW")
print(f"0-3 h: {(1003.5 - 1000.0) / 3:.2f} kW on average")

0-1 h: 1.2 kW
1-2 h: 1.8 kW
2-3 h: 0.5 kW
0-3 h: 1.17 kW on average

Over the whole three hours the average was about 1.17 kW, but that hides that the second hour (1.8 kW, perhaps the oven was on) was more than three times as busy as the third. The shorter the interval, the more detail you see. Taking that idea to its limit is exactly what a derivative does.

The difference quotient

Mathematics describes the meter as a function: E(t) is the register value at time t. The average rate of change between two moments a and b is the difference quotient:

Read it as "change in output divided by change in input". Often the second point is written as a small step h after the first, b = a + h, which gives the same quotient in another form:

On a graph, the difference quotient is the slope of the straight line through the two points (a, f(a)) and (b, f(b)), a line called a secant: how many units it rises for each unit it moves to the right.

A worked example with a simple function that curves, f(x) = x^2, between x = 3 and x = 4:

1. f(4) = 16 and f(3) = 9.
2. Change in output: 16 - 9 = 7. Change in input: 4 - 3 = 1.
3. Difference quotient: 7 / 1 = 7. On average, between 3 and 4 the function rises 7 units per unit of x.

Between 3 and 5 the answer is different: \frac{25 - 9}{5 - 3} = \frac{16}{2} = 8. For a curve, the average rate depends on the interval you choose, because the curve gets steeper as you go right.

The derivative: the rate at one moment

What is the rate of change of x^2 exactly at x = 3, not over an interval? Make the step h smaller and smaller and watch the difference quotient:

def f(x):
    return x ** 2

for h in (1, 0.1, 0.01, 0.001):
    print(h, (f(3 + h) - f(3)) / h)

1 7.0
0.1 6.100000000000012
0.01 6.009999999999849
0.001 6.000999999999479

The quotients 7, 6.1, 6.01, 6.001 head for 6. (The long tails of digits are floating-point rounding, not mathematics: see Floating-Point Numbers. Making h much smaller makes that noise worse, not better.) Algebra shows why the answer is exactly 6. For any h \ne 0:

1. Expand: f(3 + h) = (3 + h)^2 = 9 + 6h + h^2.
2. Subtract f(3) = 9: f(3 + h) - f(3) = 6h + h^2.
3. Divide by h: \frac{6h + h^2}{h} = 6 + h.
4. As h gets closer to 0, 6 + h gets closer to 6.

You may not simply put h = 0 into the original quotient (that would divide by zero), but the simplified form 6 + h shows where it is heading. Mathematics writes "the value it heads for as h goes to 0" as a limit, \lim_{h \to 0}.

The derivative of a function f at a point a is the limit of the difference quotient as the step goes to zero, the instantaneous rate of change:

It is written f'(a) and read "f prime of a". So for f(x) = x^2, f'(3) = 6. Speed shown on a car's speedometer is a derivative in this sense: the rate of change of distance at this moment, not an average over the trip. For the meter, the derivative of the energy register is the power being drawn at that instant.

The tangent line

As h shrinks, the secant through (3, 9) and (3 + h, f(3 + h)) turns until it settles into a line that just touches the curve at (3, 9), running in the same direction as the curve there. That line is the tangent line, and its slope is the derivative.

So the derivative has two faces that are the same thing: a rate (how fast f changes right here) and a slope (how steep the graph is right here). Where the derivative is positive the function rises, where it is negative the function falls, and where it is 0 the tangent is horizontal: the graph is momentarily flat, as at the bottom of a valley.

The derivative as a function, and its notation

Nothing in the algebra above depended on the 3. Do it for a general x:

1. (x + h)^2 - x^2 = x^2 + 2xh + h^2 - x^2 = 2xh + h^2.
2. Divide by h: 2x + h.
3. Let h go to 0: 2x.

So the derivative of x^2 is the new function f'(x) = 2x, which gives the slope at every point at once: f'(3) = 6, f'(0) = 0, f'(-1) = -2. Finding the derivative is called differentiating.

Besides f'(x) (Lagrange's notation) you will see Leibniz's notation, which keeps the idea of "a tiny change in y divided by a tiny change in x" visible:

\frac{dy}{dx} is read "dy by dx" and is one symbol, not a real fraction, although it often behaves like one. In physics and engineering the variable is often time: \frac{dE}{dt} is the rate of change of energy over time, that is, power.

Standard derivatives

Working out a limit every time would be slow. Instead, the derivatives of the basic functions have been worked out once, and you look them up (or learn the common ones):

The power rule is the one you will use most. Worked example: the derivative of x^3.

1. Here n = 3. The exponent 3 moves to the front as a factor.
2. The new exponent is one lower: 3 - 1 = 2. Together: n \cdot x^{n-1} = 3 \cdot x^{2}.
3. Result: \frac{d}{dx} x^3 = 3x^2. At x = 2 the slope is 3 \cdot 4 = 12.

The rule also works for negative and fractional powers, because \frac{1}{x} = x^{-1} (derivative -x^{-2} = -\frac{1}{x^2}) and \sqrt{x} = x^{1/2} (derivative \frac{1}{2} x^{-1/2}, the table entry). The trigonometric rules only hold when angles are in radians; see Geometry and Trigonometry.

You can check any entry numerically: compute a difference quotient with a small step and compare. The check below uses a central difference, one small step to each side, which is more accurate than a one-sided step:

import math

def slope(f, x, h=1e-6):
    return (f(x + h) - f(x - h)) / (2 * h)     # central difference

table = [
    ("x^3     -> 3x^2",         lambda x: x ** 3,   lambda x: 3 * x ** 2),
    ("sqrt(x) -> 1/(2 sqrt x)", math.sqrt,          lambda x: 1 / (2 * math.sqrt(x))),
    ("e^x     -> e^x",          math.exp,           math.exp),
    ("2^x     -> 2^x ln 2",     lambda x: 2 ** x,   lambda x: 2 ** x * math.log(2)),
    ("ln x    -> 1/x",          math.log,           lambda x: 1 / x),
    ("log2 x  -> 1/(x ln 2)",   math.log2,          lambda x: 1 / (x * math.log(2))),
    ("sin x   -> cos x",        math.sin,           math.cos),
    ("cos x   -> -sin x",       math.cos,           lambda x: -math.sin(x)),
]
x = 1.5
for name, f, derivative in table:
    print(f"{name:<26}{math.isclose(slope(f, x), derivative(x), rel_tol=1e-6)}")

x^3     -> 3x^2           True
sqrt(x) -> 1/(2 sqrt x)   True
e^x     -> e^x            True
2^x     -> 2^x ln 2       True
ln x    -> 1/x            True
log2 x  -> 1/(x ln 2)     True
sin x   -> cos x          True
cos x   -> -sin x         True

Rules for combined functions

Real formulas combine basic functions. Four rules take them apart. In each, f and g are functions whose derivatives you already know.

Sums and constant factors

The derivative of a sum is the sum of the derivatives, and a constant factor stays where it is:

Worked example: f(x) = 3x^2 + 5x - 7.

1. Term by term: 3x^2 \to 3 \cdot 2x = 6x; 5x \to 5 \cdot 1 = 5; -7 \to 0.
2. Add them: f'(x) = 6x + 5.

Products and quotients

A product is not differentiated factor by factor. The product rule says: derivative of the first times the second, plus the first times the derivative of the second:

Worked example: x^2 \sin x. With f = x^2 (f' = 2x) and g = \sin x (g' = \cos x), the derivative is 2x \sin x + x^2 \cos x.

For a quotient, with g(x) \ne 0:

Worked example: \frac{x}{x + 1}. Here f = x, f' = 1, g = x + 1, g' = 1, so the derivative is \frac{1 \cdot (x + 1) - x \cdot 1}{(x + 1)^2} = \frac{1}{(x + 1)^2}.

The chain rule

Many functions are a function inside a function, a composition: in (3x + 1)^2 the inner function 3x + 1 is fed into the outer function "square it". The chain rule says: differentiate the outer function (leaving the inside alone), then multiply by the derivative of the inner function:

Worked example: (3x + 1)^2.

1. Outer function: square, derivative "2 times the inside": 2 \cdot (3x + 1).
2. Inner function: 3x + 1, derivative 3.
3. Multiply: 2 \cdot (3x + 1) \cdot 3 = 6(3x + 1) = 18x + 6.
4. Check by expanding first: (3x + 1)^2 = 9x^2 + 6x + 1, derivative 18x + 6. The same.

Two more that appear below and in machine learning: (x - 3)^2 has derivative 2(x - 3) \cdot 1 = 2(x - 3), and e^{-0.5x} has derivative e^{-0.5x} \cdot (-0.5).

In Leibniz notation the chain rule looks like cancelling fractions, which is a good way to remember it: if y depends on u and u depends on x, then \frac{dy}{dx} = \frac{dy}{du} \cdot \frac{du}{dx}. Rates multiply along a chain: if a cache's size grows 2 GB per thousand users and users grow 3 thousand per month, the cache grows 2 · 3 = 6 GB per month.

import math

def slope(f, x, h=1e-6):
    return (f(x + h) - f(x - h)) / (2 * h)

x = 1.5
print(round(slope(lambda x: 3 * x ** 2 + 5 * x - 7, x), 4), 6 * x + 5)
print(round(slope(lambda x: x ** 2 * math.sin(x), x), 4),
      round(2 * x * math.sin(x) + x ** 2 * math.cos(x), 4))
print(round(slope(lambda x: x / (x + 1), x), 4), round(1 / (x + 1) ** 2, 4))
print(round(slope(lambda x: (3 * x + 1) ** 2, x), 4), 18 * x + 6)

14.0 14.0
3.1516 3.1516
0.16 0.16
33.0 33.0

Finding a minimum: gradient descent

Training a machine-learning model means adjusting its numbers so that a measure of its error, the loss function, becomes as small as possible. The derivative tells you how to do that: at a minimum the graph is flat, so the derivative is 0; and anywhere else, the sign of the derivative tells you which way is downhill.

Take the small loss function f(x) = (x - 3)^2, whose minimum is obviously at x = 3, and pretend you cannot see that. Its derivative (chain rule, above) is f'(x) = 2(x - 3).

• At x = 0: f'(0) = -6, negative, so the function falls to the right. Move right.
• At x = 5: f'(5) = 4, positive, so the function rises to the right. Move left.

In both cases the right move is against the sign of the derivative, and a steeper slope means you are further away and may take a bigger step. That is gradient descent: start somewhere and repeat

where \eta (the Greek letter eta) is the learning rate, a small number that sets the step size. Worked example with \eta = 0.1, starting at x = 0:

1. Step 1: f'(0) = -6, so x = 0 - 0.1 \cdot (-6) = 0.6.
2. Step 2: f'(0.6) = 2 \cdot (0.6 - 3) = -4.8, so x = 0.6 + 0.48 = 1.08.
3. Step 3: f'(1.08) = -3.84, so x = 1.08 + 0.384 = 1.464. The steps get smaller as the slope flattens.

# Gradient descent on f(x) = (x - 3)^2, derivative f'(x) = 2(x - 3)
x, rate = 0.0, 0.1
for step in range(1, 31):
    x -= rate * 2 * (x - 3)
    if step in (1, 2, 5, 10, 30):
        print(step, round(x, 4))

1 0.6
2 1.08
5 2.017
10 2.6779
30 2.9963

The learning rate matters a great deal. Here are the first five steps for four rates:

for rate in (0.1, 0.5, 0.9, 1.1):
    x, path = 0.0, []
    for step in range(5):
        x -= rate * 2 * (x - 3)
        path.append(round(x, 3))
    print(f"rate {rate}: {path}")

rate 0.1: [0.6, 1.08, 1.464, 1.771, 2.017]
rate 0.5: [3.0, 3.0, 3.0, 3.0, 3.0]
rate 0.9: [5.4, 1.08, 4.536, 1.771, 3.983]
rate 1.1: [6.6, -1.32, 8.184, -3.221, 10.465]

With 0.1 the steps creep toward 3. With 0.5 it lands on 3 in one step (a lucky fit for this particular function). With 0.9 it overshoots and zigzags around 3, slowly settling. With 1.1 every step overshoots further than the last and x runs away: the training diverges. A real model has millions or billions of numbers instead of one x; the derivative with respect to each of them together is called the gradient, and backpropagation computes it by applying the chain rule layer by layer. The idea is the same as in these few lines. See also a tiny training loop.

Rates in IT monitoring

Most monitoring data starts as counters that only go up, like the meter register: total bytes received by a network interface, total requests served, total errors. Dashboards show rates, and those are difference quotients: take two samples of the counter, subtract, divide by the time between them. A counter that went from 1,250,000,000 bytes to 1,265,000,000 bytes in 10 seconds shows a rate of 1,500,000 bytes per second, or 12 Mbit/s. Prometheus's rate() function, for example, is in essence this calculation over a time window, with extra handling for the problem below.

A shorter window gives a closer view of the instantaneous rate (the derivative) but more noise; a longer window gives a smoother line that reacts late. That trade-off is the difference quotient's step h again.

Rates of change also steer physical processes. The D in a PID controller stands for derivative: the controller reacts not only to how far a temperature or pressure is from its setpoint, but also to how fast that gap is changing, so it can brake before it overshoots. Its I stands for integral, adding up the error over time, which is the subject of Integrals and Accumulation.