Robot Vision

How robots use cameras: the pinhole camera model, intrinsics and calibration, fiducial markers such as AprilTag and ArUco, stereo depth, and detecting objects to pick.

What a camera gives a robot

A camera is the richest sensor a robot can carry: one image holds hundreds of thousands of measurements of color and brightness. It is also cheap. But an image is flat. A camera tells the robot in which direction something is, not how far away it is, and a robot that wants to grasp a part needs a position in meters, in its own coordinate frame.

Robot vision is about closing that gap. This page covers the geometry: how a point in the world becomes a pixel, how to measure the camera so the mapping can be reversed, and three ways to get depth back: a marker of known size, a second camera, or a depth sensor. Recognizing what is in the image is computer vision, covered in the AI topic; depth cameras and lidar are on Sensors for Robots.

Typical jobs for a camera on a robot:

  • Picking: find a part on a conveyor or in a bin and compute where the gripper must go.
  • Inspection: check a weld, a label or a gauge reading.
  • Localization: a mobile robot recognizes markers or features to know where it is (see Localization and Mapping).
  • Safety and people: some systems watch the shared space around a robot; the safety rules for that are on Robot Safety and Standards.

The pinhole camera model

The pinhole camera model describes a camera as a single point, the camera center, through which every light ray passes on its way to a flat image plane. A point in front of the camera lands on the image where its ray crosses that plane. Real lenses gather far more light than a pinhole, but after calibration they behave almost exactly like one, which is why nearly all robot vision starts here.

The model works in the camera's own coordinate frame. By the usual convention for cameras, z points forward along the optical axis, x to the right in the image and y down, because image rows are counted from the top. A point at (X, Y, Z) meters lands on pixel (u, v):

u = f_x \frac{X}{Z} + c_x \qquad v = f_y \frac{Y}{Z} + c_y

Here f_x and f_y are the focal length expressed in pixels, and (c_x, c_y) is the principal point, the pixel where the optical axis hits the image (close to the center). The heart of the formula is the division by Z: an object twice as far away looks half as big. The picture below shows it with similar triangles.

Side view of the pinhole camera model. A camera center sits on a horizontal optical axis. An image plane stands a distance f to the right of it. A point P at height H and distance Z sends a ray through the camera center; the ray crosses the image plane at height h, which equals f times H divided by Z. optical axis (z) camera center image plane P H h f Z h = f · H / Z
The small triangle (f, h) and the large triangle (Z, H) have the same shape, so h / f = H / Z. A real camera's image plane sits behind the center and the image is upside down; drawing it in front gives the same numbers without the flip.

In Python, with a 640 × 480 camera whose focal length is 600 pixels: 

fx, fy = 600.0, 600.0   # focal length in pixels
cx, cy = 320.0, 240.0   # principal point: the center of a 640 x 480 image


def project(X, Y, Z):
    """A point in the camera frame (meters) -> pixel coordinates (u, v)."""
    u = fx * X / Z + cx
    v = fy * Y / Z + cy
    return round(u, 1), round(v, 1)


print(project(0.10, 0.05, 0.80))   # a bolt 0.8 m in front of the camera
print(project(0.20, 0.10, 1.60))   # twice as far and twice as far aside
print(project(0.10, 0.05, 1.60))   # same offset, twice as far away
print(project(0.00, 0.00, 2.00))   # on the optical axis
(395.0, 277.5)
(395.0, 277.5)
(357.5, 258.8)
(320.0, 240.0)

The first two points land on the same pixel. That is the central limitation of a single camera: everything on one ray through the camera center produces the same image point, so depth is lost.

From a pixel back to a ray

Running the formula backward turns a pixel into a direction. With a known depth Z — from a depth camera, a table at a known height, or a marker — the point is fixed:

X = \frac{(u - c_x)\,Z}{f_x} \qquad Y = \frac{(v - c_y)\,Z}{f_y} 
fx, fy = 600.0, 600.0
cx, cy = 320.0, 240.0


def back_project(u, v, Z):
    """Pixel (u, v) plus a known depth Z -> point in the camera frame."""
    X = (u - cx) * Z / fx
    Y = (v - cy) * Z / fy
    return round(X, 3), round(Y, 3), Z


# The same pixel at three possible depths: one ray, three points.
for Z in (0.4, 0.8, 1.6):
    print(back_project(395.0, 277.5, Z))
(0.05, 0.025, 0.4)
(0.1, 0.05, 0.8)
(0.2, 0.1, 1.6)

Camera intrinsics

The four numbers f_x, f_y, c_x, c_y are the camera intrinsics: the parameters that describe the camera itself, independent of where it is mounted. They are usually written as a 3 × 3 matrix K, so the projection becomes one matrix-vector product followed by a division by the last entry:

Z \begin{bmatrix} u \\ v \\ 1 \end{bmatrix} = K \begin{bmatrix} X \\ Y \\ Z \end{bmatrix}, \qquad K = \begin{bmatrix} f_x & 0 & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{bmatrix} 
K = [[600.0, 0.0, 320.0],
     [0.0, 600.0, 240.0],
     [0.0, 0.0, 1.0]]


def mat_vec(M, p):
    return [sum(M[i][j] * p[j] for j in range(3)) for i in range(3)]


point = [0.10, 0.05, 0.80]        # X, Y, Z in meters
h = mat_vec(K, point)             # Z * (u, v, 1)
print([round(x, 2) for x in h])
u, v = h[0] / h[2], h[1] / h[2]   # divide by the last entry
print(round(u, 1), round(v, 1))
[316.0, 222.0, 0.8]
395.0 277.5

The camera's position and orientation in the world — its pose — are called the extrinsics. Intrinsics belong to the camera and lens; extrinsics change whenever someone moves the bracket. Both matter, and they are calibrated separately.

Lens distortion

Real lenses bend straight lines, most visibly near the edges of a wide-angle image: a barrel-shaped bulge or a pincushion squeeze. The common radial model scales each normalized image point (x = X/Z, y = Y/Z) by a polynomial in its squared distance r^2 = x^2 + y^2 from the center:

x_d = x\,(1 + k_1 r^2 + k_2 r^4) \qquad y_d = y\,(1 + k_1 r^2 + k_2 r^4) 
fx, fy, cx, cy = 600.0, 600.0, 320.0, 240.0
k1, k2 = -0.28, 0.07   # radial distortion: barrel-shaped, as in many wide lenses


def project_distorted(X, Y, Z):
    x, y = X / Z, Y / Z                    # normalized image coordinates
    r2 = x * x + y * y
    factor = 1 + k1 * r2 + k2 * r2 * r2
    return round(fx * x * factor + cx, 1), round(fy * y * factor + cy, 1)


for X in (0.05, 0.20, 0.40):
    ideal = round(fx * X / 1.0 + cx, 1)
    u, v = project_distorted(X, 0.0, 1.0)
    print(f"X = {X:.2f} m: ideal u = {ideal}, with distortion u = {u}")
X = 0.05 m: ideal u = 350.0, with distortion u = 350.0
X = 0.20 m: ideal u = 440.0, with distortion u = 438.7
X = 0.40 m: ideal u = 560.0, with distortion u = 549.7

Near the center the error is nothing; near the edge it is ten pixels — at one meter that is more than a centimeter of position error. Vision libraries "undistort" images or points with the calibrated coefficients before any geometry is done.

Camera calibration

Camera calibration is the procedure that measures a camera's intrinsics and lens distortion (and, in a second step, its pose on the robot). The standard method comes from Zhengyou Zhang's paper "A flexible new technique for camera calibration" (IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000): photograph a flat pattern of known geometry, usually a printed checkerboard, from a range of angles and distances. Every corner's true position on the board is known, and its pixel position is detected in each photo; the calibration searches for the intrinsics, distortion coefficients and board poses that explain all those corners at once.

The quality number to look at is the reprojection error: project the known corners through the calibrated model and measure how far they land from where the detector found them. An average well below one pixel is typical for a good calibration. 

import math

# Checkerboard corners: where the calibrated model predicts them (u, v)
# and where the corner detector actually found them in the image.
predicted = [(412.3, 288.1), (452.9, 287.6), (493.2, 287.4), (412.8, 328.7)]
detected = [(412.6, 287.8), (452.5, 287.9), (493.4, 287.0), (412.9, 329.2)]

squared = []
for (pu, pv), (du, dv) in zip(predicted, detected):
    squared.append((pu - du) ** 2 + (pv - dv) ** 2)

rms = math.sqrt(sum(squared) / len(squared))
print(f"RMS reprojection error: {rms:.2f} pixels")
RMS reprojection error: 0.47 pixels

OpenCV, the most widely used open-source vision library, does the whole job in a few calls. It is not installed on the 64-bit Linux machine (ARM64) that checks this site's examples, so this block is not run: 

import glob
import cv2
import numpy as np

pattern = (9, 6)                       # inner corners per row and column
square = 0.025                         # 25 mm squares
board = np.zeros((9 * 6, 3), np.float32)
board[:, :2] = np.mgrid[0:9, 0:6].T.reshape(-1, 2) * square

object_points, image_points = [], []
for name in sorted(glob.glob("calib/*.png")):
    gray = cv2.cvtColor(cv2.imread(name), cv2.COLOR_BGR2GRAY)
    found, corners = cv2.findChessboardCorners(gray, pattern, None)
    if found:
        object_points.append(board)
        image_points.append(corners)

rms, K, dist, rvecs, tvecs = cv2.calibrateCamera(
    object_points, image_points, gray.shape[::-1], None, None)
print("RMS reprojection error:", rms)
print("K =", K)

For a camera on a robot there is one more step, hand-eye calibration: measuring where the camera sits relative to the robot's base (a fixed camera) or to its flange (a camera on the arm). The robot moves a pattern or the camera through several poses, and the fixed transform between camera and robot is solved from those pairs.

Fiducial markers

A fiducial marker is a printed pattern of known size and design that is placed in a scene on purpose, so software can find it reliably and measure its pose. Instead of recognizing an arbitrary object — hard — the robot looks for a black-and-white square it was designed to see — easy and fast. Markers mark docking stations, calibration targets, pallets, shelf positions and test rigs.

AprilTag and ArUco

Two marker families dominate robotics. An AprilTag is a square black-and-white fiducial marker that encodes an ID in a grid of cells, designed at the University of Michigan for robust detection and accurate pose estimation. It was introduced by Edwin Olson at the 2011 IEEE International Conference on Robotics and Automation (ICRA), improved as AprilTag 2 (Wang and Olson, 2016), and the current "AprilTag 3" library is described in Krogius, Haggenmiller and Olson's "Flexible Layouts for Fiducial Tags" (2019). ArUco markers (Garrido-Jurado and colleagues, Pattern Recognition, 2014) work on the same principle and ship with OpenCV. Both come in dictionaries: sets of patterns chosen to be as different from each other as possible, so a partly hidden or blurred marker is not mistaken for another.

Because the marker's real size and corner layout are known, one calibrated camera can compute its full pose — position and orientation — from the four corner pixels. This is a perspective-n-point problem: find the pose that projects known 3D points onto the observed pixels. The simplest part of it, distance, follows directly from the pinhole model: a marker of size S that appears s pixels wide is at distance

Z = f\,\frac{S}{s}
fx = 600.0         # focal length in pixels (from calibration)
tag_size = 0.10    # printed tag: 10 cm black square, edge to edge

for pixels in (120.0, 80.0, 40.0, 20.0):
    distance = fx * tag_size / pixels
    print(f"tag {pixels:5.1f} px wide -> {distance:.2f} m away")
tag 120.0 px wide -> 0.50 m away
tag  80.0 px wide -> 0.75 m away
tag  40.0 px wide -> 1.50 m away
tag  20.0 px wide -> 3.00 m away

The far tag spans only 20 pixels: one pixel of detection error is then five percent of the distance. Marker size is a design choice — make tags big enough for the farthest distance at which the robot must still read them accurately.

Stereo vision

Stereo vision recovers depth from two cameras a known distance apart, the way two eyes do. Each camera sees the same point at a slightly different horizontal position; that shift, the disparity d in pixels, is larger for close objects and smaller for far ones. With focal length f (in pixels) and baseline B (the distance between the cameras):

Z = \frac{f\,B}{d}
Two cameras side by side, a baseline B apart, both looking up at a point P at depth Z. A short image plane sits a distance f in front of each camera. The ray from P to the left camera crosses its image plane right of center, the ray to the right camera crosses left of center. The sum of the two offsets is the disparity d. P left camera right camera B f Z d = 15 + 15 = 30
In the drawing f = 40, B = 120 and Z = 160 units: the point lands 15 units right of center in the left image and 15 units left of center in the right one, so d = 30 = f · B / Z.

The formula has a catch that every stereo user meets. Disparity is measured in whole or fractional pixels, and a fixed error in d causes a depth error that grows with the square of the distance:

\Delta Z \approx \frac{Z^2}{f\,B}\,\Delta d 
f = 700.0    # focal length in pixels (both cameras the same, rectified)
B = 0.06     # baseline: 6 cm between the two cameras

print(" Z (m)  disparity (px)  error per 1 px (m)")
for Z in (0.5, 1.0, 2.0, 4.0):
    d = f * B / Z                 # disparity this depth produces
    error = Z * Z / (f * B)       # how far Z moves if d is off by 1 px
    print(f"{Z:5.1f}  {d:14.1f}  {error:18.3f}")
 Z (m)  disparity (px)  error per 1 px (m)
  0.5            84.0               0.006
  1.0            42.0               0.024
  2.0            21.0               0.095
  4.0            10.5               0.381

At half a meter a one-pixel mistake costs 6 mm; at four meters it costs almost 40 cm. A wider baseline or a longer focal length helps, at the price of a bigger device and a narrower view. Stereo also needs texture: on a blank white wall there is nothing to match between the two images. Active stereo cameras solve that by projecting a dot pattern; time-of-flight and structured-light cameras measure depth in other ways (see Sensors for Robots).

From pixels to the robot

A point in the camera frame is not yet useful to the arm. The robot plans in its own base frame, so the last step is a homogeneous transform from camera to base, measured by hand-eye calibration:

\mathbf{p}_{\text{base}} = R\,\mathbf{p}_{\text{cam}} + \mathbf{t}

Here a camera hangs 0.80 m above the table, 0.50 m in front of the robot, looking straight down. Its z axis therefore points down (opposite to the base's), its x axis matches the base's and its y axis is flipped. A detected part gives a pixel and a depth reading: 

# Camera 0.80 m above the robot base and 0.50 m in front of it, looking
# straight down: camera x = base x, camera y = -base y, camera z = -base z.
T_base_cam = [
    [1.0, 0.0, 0.0, 0.50],
    [0.0, -1.0, 0.0, 0.00],
    [0.0, 0.0, -1.0, 0.80],
    [0.0, 0.0, 0.0, 1.0],
]


def transform(T, p):
    x, y, z = p
    q = [T[i][0] * x + T[i][1] * y + T[i][2] * z + T[i][3] for i in range(3)]
    return [round(c, 3) + 0.0 for c in q]


fx, fy, cx, cy = 600.0, 600.0, 320.0, 240.0
u, v, Z = 350.0, 255.0, 0.78                 # detected part: pixel and depth
p_cam = ((u - cx) * Z / fx, (v - cy) * Z / fy, Z)
print("camera frame:", [round(c, 3) for c in p_cam])
print("base frame:  ", transform(T_base_cam, p_cam))
camera frame: [0.039, 0.02, 0.78]
base frame:   [0.539, -0.02, 0.02]

The part's top is 2 cm above the table, 54 cm in front of the robot and 2 cm to its right. From here inverse kinematics computes the joint angles that bring the gripper there. In a ROS 2 system the transform tree (tf2) does this bookkeeping; see ROS 2.

Detecting objects to pick

Geometry says where a pixel is; something must first say which pixels belong to the part. There are two broad approaches:

  • Classic image processing for controlled scenes: thresholding a dark part on a light conveyor, finding edges, matching a template. Fast, predictable and easy to explain, but sensitive to lighting and to parts that look different.
  • Learned models for varied scenes: an object detection model, usually a convolutional neural network or a transformer, draws a box around each part; a segmentation model marks its exact pixels. Training needs labeled images of the real parts, and the accuracy must be measured on images the model has not seen.

A typical pick pipeline then runs:

  1. Capture a color image and a depth image at the same moment.
  2. Detect the part and choose a grasp point (often the center of its top face, or a point a learned model suggests).
  3. Read the depth at that pixel (a median over a small patch is more robust than one pixel).
  4. Back-project to the camera frame, transform to the base frame.
  5. Plan an approach from above, move, grip, and check the gripper sensor that the part is really there.

Learned grasping and robot foundation models that go from image to motion directly are on AI in Robotics.

Practical problems

Most vision failures on a shop floor are not algorithm failures but physics:

  • Light. Sunlight through a skylight, a reflection on a shiny part, a lamp replaced by one of a different color. Industrial installations control the light with their own lighting and shields; a detection that only works at 10 a.m. is not finished.
  • Motion blur. A moving conveyor or a moving camera smears the image; short exposure times need more light.
  • A bumped camera. If the bracket moves by one degree, the extrinsics are wrong and every pick is off. A fixed marker in view lets the system detect that it has moved.
  • Timing. The image shows where the part was. On a moving conveyor the robot must add the distance traveled since the image was taken.
  • Dirty lenses. Dust and oil mist are normal in a factory; cleaning is part of maintenance.