Vectors and Matrices

What is a vector?

A vector is an ordered list of numbers, called its entries or components. The number of entries is its dimension. Geometrically, a vector with two entries is an arrow in the plane: the vector (3, 4) points 3 steps right and 4 steps up. With more entries it is an arrow you cannot draw, but every calculation works the same way.

In IT, most vectors are not arrows but records of measurements. The electricity use of a Kestrel Energy customer in January, February and March, 310, 280 and 295 kWh, is a vector with three entries. An AI embedding that represents the meaning of a sentence is a vector with hundreds or thousands of entries. Vectors are written in bold, and their entries with a subscript that says which position they hold:

A single ordinary number, as opposed to a vector, is called a scalar: a quantity with a size but no direction, such as a price of 0.27 euros per kWh or a factor of 1.1. The name comes from its job: multiplying a vector by a scalar scales it. In Python a vector is simply a list or tuple of numbers (numerical libraries such as NumPy add fast vector types, but the math is the same).

Adding and scaling vectors

Both basic operations work entry by entry. To add two vectors of the same dimension, add the matching entries; to multiply a vector by a scalar, multiply every entry:

Worked example: a household has a main meter and a separate meter for its heat pump.

1. Main meter, January to March: (310, 280, 295) kWh. Heat pump: (240, 210, 150) kWh.
2. Total use is the sum, entry by entry: (310 + 240, 280 + 210, 295 + 150) = (550, 490, 445) kWh.
3. Next year's forecast assumes 10% less use: multiply by the scalar 0.9, giving (495, 441, 400.5) kWh.

main = [310, 280, 295]
heat_pump = [240, 210, 150]

total = [a + b for a, b in zip(main, heat_pump)]
forecast = [round(0.9 * x, 1) for x in total]
print(total)
print(forecast)
print(main + heat_pump)        # careful: + on lists joins them

[550, 490, 445]
[495.0, 441.0, 400.5]
[310, 280, 295, 240, 210, 150]

Length and distance

The norm of a vector is its length: the square root of the sum of the squares of its entries. It is written with double bars:

For two entries this is the Pythagorean theorem from Geometry and Trigonometry: the arrow (3, 4) is the long side of a right triangle with sides 3 and 4. Worked out:

1. Square each entry: 3² = 9 and 4² = 16.
2. Add: 9 + 16 = 25.
3. Take the square root: √25 = 5.

The formula simply keeps going for more entries; with three it is the length of an arrow in space, with 1,536 it is the length of an embedding. The distance between two vectors is the norm of their difference, \lVert \mathbf{a} - \mathbf{b} \rVert: subtract entry by entry, then take the length. That is how the k-nearest-neighbors method decides which known examples are "closest" to a new one (see k-nearest neighbors).

Dividing a vector by its own norm gives a unit vector: same direction, length 1. This is called normalizing, and many embedding models return vectors that are already normalized.

import math

def norm(v):
    return math.sqrt(sum(x * x for x in v))

print(norm([3, 4]))
print(norm([550, 490, 445]))                      # this year's usage vector
print(norm([a - b for a, b in zip([3, 4], [6, 8])]))   # distance between two points

unit = [x / norm([3, 4]) for x in [3, 4]]
print(unit, norm(unit))

5.0
860.5957239029253
5.0
[0.6, 0.8] 1.0

The second result is an irrational number; Python shows it rounded to about 16 significant digits, as every float is.

The dot product

The dot product of two vectors of the same dimension multiplies the matching entries and adds up the results. Its outcome is a single number (a scalar), not a vector:

You have used it without the name. A bill is a dot product: the kWh used in each month times the tariff in that month, added up. For use (210, 190) kWh and tariffs (0.27, 0.29) euros per kWh:

1. Multiply the first entries: 210 × 0.27 = 56.70.
2. Multiply the second entries: 190 × 0.29 = 55.10.
3. Add: 56.70 + 55.10 = 111.80 euros.

An artificial neuron in a neural network does exactly the same: it takes the dot product of its inputs with its weights, adds a bias and passes the result on (see the artificial neuron). Training a network is, at heart, adjusting the numbers in billions of dot products.

The dot product and the angle

The dot product also has a geometric meaning. If \theta (theta) is the angle between two arrows, then:

Read: "a dot b equals the length of a times the length of b times the cosine of the angle between them". The cosine is 1 when the arrows point the same way, 0 when they are at right angles and −1 when they point in opposite directions (see Geometry and Trigonometry for the cosine). So the sign of the dot product already says something: positive means "roughly the same direction", zero means perpendicular, negative means "roughly opposite".

Cosine similarity

Rearranging the formula gives the cosine of the angle on its own:

This is cosine similarity: a score from −1 to 1 that measures how much two vectors point in the same direction, regardless of how long they are. Semantic search compares the embedding of a question with the embeddings of documents this way and returns the documents with the highest score.

Worked out for the two vectors in the figure:

1. Dot product: 3 × 4 + 4 × 3 = 12 + 12 = 24.
2. Lengths: both are √(9 + 16) = 5.
3. Cosine similarity: 24 ÷ (5 × 5) = 24 ÷ 25 = 0.96.

import math

a = [3, 4]
b = [4, 3]
dot = sum(x * y for x, y in zip(a, b))
length = lambda v: math.sqrt(sum(x * x for x in v))
print(dot, length(a), length(b))
print(round(dot / (length(a) * length(b)), 4))
print(round(math.degrees(math.acos(dot / (length(a) * length(b)))), 1), "degrees")

24 5.0 5.0
0.96
16.3 degrees

import math

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    return dot / (math.dist(a, [0] * len(a)) * math.dist(b, [0] * len(b)))

print(round(cosine([1, 1], [10, 10]), 4), round(math.dist([1, 1], [10, 10]), 4))
print(round(cosine([1, 0], [0, 1]), 4))       # perpendicular
print(round(cosine([2, 1], [-2, -1]), 4))     # opposite

1.0 12.7279
0.0
-1.0

Matrices

A matrix is a rectangular table of numbers arranged in rows and columns. A matrix with m rows and n columns is called an "m \times n matrix" (say "m by n"; rows always come first). The entry in row i and column j of matrix \mathbf{A} is written a_{ij}.

Kestrel's usage of three customers over two months is a 3 × 2 matrix: one row per customer, one column per month.

Here u_{21} = 340: the second customer's use in the first month. Each row is a vector (one customer over time) and each column is a vector (all customers in one month).

The transpose \mathbf{U}^\mathsf{T} swaps rows and columns: the 3 × 2 matrix becomes a 2 × 3 matrix with one row per month. A graph's adjacency matrix is a square matrix, and for an undirected graph it equals its own transpose.

Matrix times vector

Multiplying a matrix by a vector means: take the dot product of each row with the vector. The result is a vector with one entry per row. The number of columns of the matrix must equal the number of entries in the vector, or the dot products do not line up. Applied to the usage matrix and the tariff vector (0.27, 0.29), this gives all three bills at once:

The first row reproduces the bill from the dot-product section; the other rows are the same recipe for the other customers.

# Monthly usage (kWh) of 3 customers in 2 months, times tariff vector
usage = [[210, 190],
         [340, 360],
         [120, 115]]
tariff = [0.27, 0.29]                 # euros per kWh in month 1 and month 2
bills = [round(sum(u * t for u, t in zip(row, tariff)), 2) for row in usage]
print(bills)

[111.8, 196.2, 65.75]

A layer of a neural network is exactly this operation: the inputs form a vector, each neuron's weights form one row of a matrix, and matrix times vector computes every neuron's weighted sum in one go (see layers). That is why AI hardware is built to multiply matrices fast.

Matrix multiplication

Matrix multiplication combines two matrices by taking the dot product of every row of the first with every column of the second. If \mathbf{A} is m \times n and \mathbf{B} is n \times p, the product \mathbf{AB} is m \times p, and its entry in row i, column j is:

Read: "row i of A dot column j of B". The two inner numbers of the sizes must match (the n in m \times n and n \times p); the outer two give the size of the result. Matrix times vector is the special case p = 1.

Worked example. Kestrel offers two plans: a fixed tariff of 0.28 euros per kWh in both months, and a variable tariff of 0.27 and 0.29. Put the plans side by side as the columns of a 2 × 2 tariff matrix, and one multiplication shows every customer's bill under every plan:

1. Check the sizes: usage is 3 × 2, tariffs are 2 × 2. Inner numbers 2 and 2 match; the result will be 3 × 2 (customers × plans).
2. Row 1, column 1 (customer 1, fixed): 210 × 0.28 + 190 × 0.28 = 58.80 + 53.20 = 112.00.
3. Row 1, column 2 (customer 1, variable): 210 × 0.27 + 190 × 0.29 = 111.80.
4. Repeat for rows 2 and 3.

def matmul(A, B):
    assert len(A[0]) == len(B), "inner dimensions must match"
    return [[round(sum(A[i][k] * B[k][j] for k in range(len(B))), 2)
             for j in range(len(B[0]))]
            for i in range(len(A))]

usage = [[210, 190], [340, 360], [120, 115]]      # 3 x 2: customers x months
plans = [[0.28, 0.27],                            # 2 x 2: months x plans
         [0.28, 0.29]]                            # column 0 fixed, column 1 variable
for customer, (fixed, variable) in enumerate(matmul(usage, plans), start=1):
    print(f"customer {customer}: fixed {fixed:6.2f}  variable {variable:6.2f}")

customer 1: fixed 112.00  variable 111.80
customer 2: fixed 196.00  variable 196.20
customer 3: fixed  65.80  variable  65.75

Order matters

With ordinary numbers, 3 × 5 = 5 × 3. With matrices that is usually false: \mathbf{AB} \ne \mathbf{BA}. Often only one of the two products even exists (a 3 × 2 times a 2 × 2 is fine; a 2 × 2 times a 3 × 2 is not). Even for square matrices, the results differ:

Multiplying by the second matrix on the right swaps the columns of the first; on the left it swaps the rows.

The identity matrix

The identity matrix \mathbf{I} is the square matrix with 1s on the diagonal from top left to bottom right and 0s everywhere else. It is the matrix version of the number 1: multiplying by it changes nothing, \mathbf{AI} = \mathbf{IA} = \mathbf{A}.

def matmul(A, B):
    return [[sum(A[i][k] * B[k][j] for k in range(len(B))) for j in range(len(B[0]))]
            for i in range(len(A))]

A = [[1, 2], [3, 4]]
B = [[0, 1], [1, 0]]
I = [[1, 0], [0, 1]]
print(matmul(A, B))
print(matmul(B, A))
print(matmul(A, I), matmul(I, A))

[[2, 1], [4, 3]]
[[3, 4], [1, 2]]
[[1, 2], [3, 4]] [[1, 2], [3, 4]]

Matrices as transformations

A 2 × 2 matrix times a 2-entry vector gives another 2-entry vector, so a matrix can be seen as a machine that moves points in the plane. Scaling, mirroring and rotating are all such machines. Rotation by an angle \theta counterclockwise around the origin is:

Worked example: rotate the point (1, 0) by 90°.

1. cos 90° = 0 and sin 90° = 1, so the matrix is \begin{pmatrix} 0 & -1 \\ 1 & 0 \end{pmatrix}.
2. First row dot (1, 0): 0 × 1 + (−1) × 0 = 0.
3. Second row dot (1, 0): 1 × 1 + 0 × 0 = 1.
4. The result is (0, 1): the point on the x-axis has moved a quarter turn to the y-axis, as expected.

import math

def rotate(point, degrees):
    t = math.radians(degrees)            # Python's trig functions use radians
    x, y = point
    return (round(math.cos(t) * x - math.sin(t) * y, 10),
            round(math.sin(t) * x + math.cos(t) * y, 10))

print(rotate((1, 0), 90))
print(rotate((1, 1), 180))
print(rotate(rotate((1, 0), 90), 90))     # two quarter turns = a half turn

(0.0, 1.0)
(-1.0, -1.0)
(-1.0, 0.0)

The rounding to 10 decimals hides tiny float errors: the computer's cos 90° is not exactly 0 but about 6 × 10⁻¹⁷ (see Floating-Point Numbers).

Doing one transformation after another corresponds to multiplying their matrices, so a whole chain of moves — scale, then rotate, then mirror — can be combined into a single matrix in advance. Because matrix multiplication is not commutative, the order of the steps matters, just as "rotate, then stretch horizontally" gives a different picture from "stretch horizontally, then rotate". Graphics cards apply such matrices to millions of points per frame; image editors use them to rotate and resize pictures; and screen rotation on a phone is a rotation matrix applied to the whole display.