Vision, Speech and Multimodal AI

Computer vision

Computer vision is the field of AI that extracts information from images and video: what is in them, where it is, and what is happening. To a computer an image is a grid of numbers, three per pixel for red, green and blue. Turning those numbers into "a cracked insulator on the left pole" is what vision models learn from labeled examples. For years the workhorse was the convolutional neural network; newer models also use the transformer architecture known from language models. The tasks, from coarse to fine:

• Image classification: one label (or a few) for the whole image. "This photo shows a gas meter." "This cabinet is open or closed."
• Object detection: find each object and draw a box around it, with a label and a confidence score.
• Segmentation: label every pixel, giving the exact outline of each object rather than a box. Used where area or shape matters, such as measuring corrosion on a pipe.

At an energy company the use cases are concrete: reading the digits from meter photos that customers upload, spotting damaged cables or vegetation too close to power lines in drone footage, checking that installers wear helmets in site photos, and counting solar panels on aerial images. Most of these are object detection: the task of finding each instance of the relevant object classes in an image and returning its class, a bounding box around it and a confidence score.

Measuring a detection: intersection over union

How do you decide whether a predicted box is "right"? It will never match a person's box to the pixel. The standard measure is intersection over union (IoU): the area where the two boxes overlap, divided by the area they cover together. Identical boxes score 1, boxes that do not touch score 0, and a detection typically counts as correct at an IoU of 0.5 or more.

The example computes IoU for three predicted boxes against the box an inspector drew around a damaged cable in a photo. Detection libraries compute exactly this, millions of times, when they score a model.

# Boxes are (x1, y1, x2, y2) in pixels: top-left and bottom-right corners.
def iou(a, b):
    """Intersection over union of two boxes, from 0 (no overlap) to 1."""
    width = min(a[2], b[2]) - max(a[0], b[0])
    height = min(a[3], b[3]) - max(a[1], b[1])
    if width <= 0 or height <= 0:
        return 0.0
    overlap = width * height
    area_a = (a[2] - a[0]) * (a[3] - a[1])
    area_b = (b[2] - b[0]) * (b[3] - b[1])
    return overlap / (area_a + area_b - overlap)


# Where an inspector marked the damaged cable in a photo ...
truth = (120, 80, 320, 200)
# ... and three boxes a detection model predicted.
predictions = {
    "model A": (130, 90, 330, 210),
    "model B": (200, 60, 420, 180),
    "model C": (400, 250, 480, 300),
}

for name, box in predictions.items():
    score = iou(truth, box)
    verdict = "hit" if score >= 0.5 else "miss"
    print(f"{name}: IoU {score:.2f} -> {verdict}")

model A: IoU 0.77 -> hit
model B: IoU 0.31 -> miss
model C: IoU 0.00 -> miss

Model B found the right cable but placed the box too far to the right, so it counts as a miss (and, in most scoring schemes, as a false positive). Counting hits and misses this way over a test set gives the precision and recall of a detector, the same measures explained in Evaluating Models.

Optical character recognition and documents

Optical character recognition (OCR) converts images of printed or handwritten text, such as scans, photos and PDF files without a text layer, into machine-readable text. Printed text in good scans is largely a solved problem; handwriting, stamps, crumpled paper and phone photos taken at an angle are still hard.

Plain text is rarely the goal. An accounts payable team wants the supplier, invoice number, date, VAT and total from each invoice, in fields, not a page of text. Document AI (also called document intelligence or information extraction) combines OCR with layout analysis and models that understand structure: key-value pairs ("Invoice total: € 1,240.00"), tables and checkboxes. Typical uses are invoices, delivery notes, scanned application forms, ID documents and meter cards.

Speech recognition and text-to-speech

Speech recognition (speech-to-text, also called automatic speech recognition or ASR) converts spoken audio into text. Text-to-speech (TTS, speech synthesis) does the reverse: it generates spoken audio from text. Both improved enormously with deep learning; modern synthetic voices are hard to tell apart from real ones.

Where they are used in IT and customer service:

• Call centers: transcribing calls in real time, suggesting knowledge articles to the agent while the customer talks, summarizing the call into the ticket afterward, and spotting topics across thousands of calls. Splitting a recording by speaker ("who said what") is called diarization.
• Voice bots: speech recognition, a language model and text-to-speech chained together, so a customer can report a power outage by phone at night (see virtual agents).
• Accessibility: live captions in meetings for people who are deaf or hard of hearing, dictation for people who cannot type easily, and screen readers that read text aloud for people who are blind or have low vision.

Recording and transcribing calls processes personal data, and sometimes sensitive data, so callers must be informed and retention limited. Cloning a specific person's voice needs that person's explicit consent; reputable services require it.

Multimodal models

Vision, speech and language used to be separate models chained together. A multimodal model is a single model that accepts, and sometimes produces, more than one kind of data, such as text, images, audio and video. Most frontier models of the GPT, Claude and Gemini families accept images and documents next to text as of 2026, and several handle audio directly.

The practical change is that you can simply show the problem. A service desk analyst pastes a screenshot of an error dialog and asks what it means; an engineer uploads a photo of a network rack and asks which cable goes where; a model reads a chart in a PDF report. One model handling both sides also keeps context that a chain of separate models would lose, such as the tone of voice in a spoken complaint.

Multimodal input is still tokens, and images in particular can use many of them, so they add to cost (see Tokens and cost). Models can misread small text, count objects badly and describe details that are not there with full confidence. And text inside an image counts as input: a screenshot can carry an instruction aimed at the model, a variant of prompt injection (see AI Security).

Generating images, audio and video

A diffusion model is a generative model that creates an image (or audio, or video) by starting from random noise and removing that noise in many small steps, each step guided by the prompt, until a clean result remains. It learns this by doing the opposite during training: it takes real images, adds increasing amounts of noise, and learns to predict and remove the noise at every level.

The idea appeared in a 2015 paper by Sohl-Dickstein et al.; it became practical with denoising diffusion probabilistic models (Ho et al., 2020). Latent diffusion (Rombach et al., 2022), the basis of Stable Diffusion, made it affordable by denoising a compressed representation of the image instead of every pixel. The prompt steers each step through a text encoder, which is why the quality of the prompt matters here too. The same principle is used for video (denoising many frames consistently) and for audio and music. Not every generator is a diffusion model, though: some image and speech generators predict their output token by token like language models, and many current systems combine both approaches.

Generated media raise their own questions: who owns the output and whether training on copyrighted work was lawful is still being decided in courts in several countries, and a generated product photo or "screenshot" can mislead even when nobody intends harm. Label generated content clearly in anything customer-facing.

Deepfakes and provenance

A deepfake is synthetic or manipulated audio, video or images that realistically show a real person saying or doing something they never said or did. The technology that dubs a training video into five languages also produces a fake voice message from the CFO asking for an urgent payment. For IT and security teams the most immediate risk is social engineering: cloned voices on the phone and faked video calls (see Social Engineering and Phishing). The defense is procedural rather than technical: verify unusual requests through a second, known channel, such as calling back on a number from the directory, and never let a voice or face alone authorize a payment or a password reset.

Detecting fakes after the fact is an arms race that detectors tend to lose, so the industry also works from the other end: proving where genuine content comes from. Content credentials, based on the C2PA standard (Coalition for Content Provenance and Authenticity), attach signed metadata to a file recording which camera or tool created it and how it was edited. Support is growing in cameras, editing software and generators, but it is not universal, and metadata can be stripped when a file is re-saved or screenshotted; missing credentials prove nothing. Some generators also add invisible watermarks to their output.

The law is catching up as well, with labeling duties for generated content and criminal offenses for the most harmful fakes.

Accuracy issues and bias

Vision and speech models are trained on data, and they work best on input that resembles it. Typical causes of errors:

• Images: poor lighting, glare on a meter's glass, motion blur, unusual angles, dirty lenses, and objects that look different from the training photos (a new meter model, snow on a cable).
• Speech: background noise, poor phone lines, people talking over each other, and accents, dialects and speech patterns that were rare in the training data. Domain vocabulary, such as product names and Dutch street names in an English call, trips models up too.
• Documents: low-resolution scans, handwriting, unusual layouts and mixed languages.

Errors that fall unevenly on groups of people are bias. Research such as the Gender Shades study (Buolamwini and Gebru, 2018) found commercial gender classification systems far less accurate for women with darker skin than for men with lighter skin, and speech recognition has been shown to make more errors for some accents and dialects. When the error lands on a customer, for instance a voice bot that keeps misunderstanding one group of callers, it becomes a service and fairness problem, discussed in Responsible AI. The remedy is the same as everywhere in AI: test on data from your real users and conditions, report accuracy per group, set confidence thresholds, and route uncertain cases to a person.