AI Infrastructure and Cloud Services

Training versus inference

Every AI system has two phases (see how machines learn). Training adjusts the model's parameters on a large dataset; it runs once (or periodically) on thousands of accelerators for weeks. Inference uses the finished model to answer a request; each request is small, but a popular service handles millions of them a day. The two have very different cost profiles:

Rough rules of thumb from the research literature make the difference concrete. Training a transformer costs about 6 arithmetic operations per parameter per training token (Kaplan et al., 2020); generating one token costs about 2 operations per parameter. The "Chinchilla" study (Hoffmann et al., 2022) found that compute-optimal training uses about 20 training tokens per parameter. For a 7-billion-parameter model on a hypothetical accelerator that sustains 10¹⁵ operations per second:

parameters = 7e9                   # a 7-billion-parameter model
training_tokens = 20 * parameters  # "Chinchilla" rule: about 20 tokens per parameter

# Common approximations for transformer models:
# training costs about 6 operations per parameter per token,
# generating one token costs about 2 operations per parameter.
training_flops = 6 * parameters * training_tokens
flops_per_token = 2 * parameters

ACCELERATOR = 1e15  # a hypothetical accelerator sustaining 10^15 operations/s

print(f"training tokens:        {training_tokens:.2e}")
print(f"training compute:       {training_flops:.2e} operations")
print(f"  on one accelerator:   {training_flops / ACCELERATOR / 86400:.0f} days")
print(f"compute per token:      {flops_per_token:.2e} operations")
answer_tokens = 500
print(f"one 500-token answer:   {answer_tokens * flops_per_token / ACCELERATOR * 1000:.0f} ms of compute")
answers = training_flops / (answer_tokens * flops_per_token)
print(f"answers equal to training: {answers:.2e}")

training tokens:        1.40e+11
training compute:       5.88e+21 operations
  on one accelerator:   68 days
compute per token:      1.40e+10 operations
one 500-token answer:   7 ms of compute
answers equal to training: 8.40e+08

One answer is a tiny fraction of the training cost, but after about 840 million answers inference has used as much compute as training. A widely used model reaches that quickly, which is why providers put so much effort into cheaper inference. Real frontier models are far larger than 7B and are trained on many more tokens than the Chinchilla ratio, so treat these numbers as an illustration of proportions, not as real costs.

For users, inference cost shows up as a price per million input and output tokens (output tokens are usually more expensive), and speed shows up as time to first token and tokens per second. Reasoning models generate many internal tokens before answering, so they cost more and take longer per question; see working with LLMs.

AI accelerators

An AI accelerator is a processor designed to run the core operation of neural networks — multiplying large matrices of numbers — far faster and more efficiently than a general-purpose CPU, by doing thousands of operations in parallel, often at reduced precision. Three families matter:

GPUs

The GPU (graphics processing unit) was built to calculate millions of pixels in parallel, which turned out to be exactly the arithmetic neural networks need. Since AlexNet was trained on two gaming GPUs in 2012 (see deep learning), GPUs have been the workhorse of AI. NVIDIA dominates the data center market with successive generations such as Hopper, Blackwell and Rubin, and its CUDA software ecosystem is a large part of that lead. Alternatives include AMD Instinct GPUs and the cloud providers' own chips: AWS Trainium and Inferentia, Microsoft Maia and Google's TPUs.

TPUs

A TPU (Tensor Processing Unit) is Google's custom AI accelerator, an application-specific chip built only for neural-network math. Most customers rent TPUs on Google Cloud; since 2026 Google also sells TPU systems to a few very large customers for their own data centers. Google trains and serves its own Gemini models on them.

NPUs

An NPU (neural processing unit) is a small, power-efficient AI accelerator built into a laptop, phone or tablet processor, for running models locally: background blur in video calls, live captions, photo search, small language models. NPU performance is quoted in TOPS (trillions of operations per second). Microsoft requires an NPU that can do more than 40 TOPS for a Copilot+ PC; example chips are Snapdragon X Elite, AMD Ryzen AI 300 and Intel Core Ultra 200V. NPUs are fast only for the operations and number formats they were designed for, which is why models often have to be quantized (next section) to run on them.

Memory as the bottleneck

For language models, the question is usually not "is the chip fast enough?" but "does the model fit in memory, and how fast can the memory deliver it?" Two simple facts explain most of it:

1. Size: the weights must fit in the accelerator's memory (GPU VRAM or, for an NPU or CPU, system RAM). Memory needed ≈ number of parameters × bytes per parameter, plus extra for the KV cache (the model's working memory for the conversation so far, which grows with the context length) and the runtime.
2. Speed: to generate each token, the model has to read essentially all its weights from memory once. So the upper limit on tokens per second for one conversation is about memory bandwidth ÷ model size in bytes. The multiplications are fast; the waiting for data is slow.

Quantization

Quantization means storing a model's weights with fewer bits — for example 8 or 4 bits instead of 16 or 32 — so the model needs less memory, usually generates tokens faster (it moves less data per token) and uses less energy, at the cost of some accuracy. Rounding a weight from a 16-bit number to one of 16 possible 4-bit values loses detail, but because a model has billions of weights the errors largely average out; well-quantized 8-bit models are usually close to the original, and 4-bit is a common compromise for running models locally. The program computes memory and the bandwidth limit for two model sizes:

BYTES_PER_PARAMETER = {"16-bit": 2, "8-bit": 1, "4-bit": 0.5}
models = {"7B": 7e9, "70B": 70e9}

GB = 1e9         # decimal gigabytes, as on spec sheets
LAPTOP = 100     # memory bandwidth in GB/s (an illustrative modern laptop)
SERVER = 1000    # memory bandwidth in GB/s (an illustrative data center accelerator)

print("model precision  weights  needs ~   tokens/s at 100 / 1000 GB/s")
for name, parameters in models.items():
    for precision, size in BYTES_PER_PARAMETER.items():
        weights_gb = parameters * size / GB
        needed_gb = weights_gb * 1.2  # rough 20% extra for KV cache and runtime
        # every generated token reads all weights once: bandwidth / size
        laptop_tps = LAPTOP / weights_gb
        server_tps = SERVER / weights_gb
        print(f"{name:<6}{precision:<9}{weights_gb:>6.1f} GB{needed_gb:>6.1f} GB"
              f"{laptop_tps:>11.1f} /{server_tps:>6.1f}")

model precision  weights  needs ~   tokens/s at 100 / 1000 GB/s
7B    16-bit     14.0 GB  16.8 GB        7.1 /  71.4
7B    8-bit       7.0 GB   8.4 GB       14.3 / 142.9
7B    4-bit       3.5 GB   4.2 GB       28.6 / 285.7
70B   16-bit    140.0 GB 168.0 GB        0.7 /   7.1
70B   8-bit      70.0 GB  84.0 GB        1.4 /  14.3
70B   4-bit      35.0 GB  42.0 GB        2.9 /  28.6

Reading the table:

• A 7B model at 4-bit fits easily on a laptop with 16 GB of RAM and is usable there; at 16-bit it does not fit comfortably (about 17 GB needed) and is slow.
• A 70B model at 16-bit needs 140 GB for the weights alone — more than any single consumer GPU has — so it needs one data center accelerator with very large memory or is split across several. Even at 4-bit it needs a high-end workstation.
• Halving the bits doubles the maximum speed, because there is half as much data to read per token. That is why quantization matters for speed, not only for fitting.

The 20% margin is a rough rule; long contexts can need much more KV cache. Real servers also process many conversations in one pass (batching), so their total throughput is far higher than the per-conversation limit shown here.

Cloud AI services as of 2026

Most organizations do not run large models themselves; they use a cloud platform that hosts many models behind one API, plus tools around them. A model hosting service is such a managed service: the provider runs the hardware and the model, and you send requests and pay per use (per token) or for reserved capacity. Product names in this area change often; the names below are as of October 2026.

Across providers, the platforms offer the same building blocks:

• Model catalog: a choice of models from the cloud provider itself and from other companies (closed and open-weight), with documentation of what each is good at and what it costs.
• Hosting: serverless pay-per-token endpoints, or dedicated capacity for predictable load, in a region you choose.
• Fine-tuning: adapting a model with your own examples (see deep learning).
• Safety filters: configurable filters on input and output for harmful content, prompt injection attempts and sensitive data (for example Amazon Bedrock Guardrails and the content safety features in Microsoft Foundry).
• Evaluation: tools to test a model or an application against a set of questions and score the answers before and after changes.
• Agents and retrieval: managed building blocks for RAG and agents.

Shared responsibility for AI services

The shared responsibility model applies to AI services too, and it maps well onto the familiar service models: a hosted model behind an API is close to SaaS or PaaS, a model you deploy on rented GPUs is closer to IaaS.

Read the data-processing terms before sending personal data: whether prompts are stored, for how long, whether they are used for training (enterprise offerings usually say no), and where they are processed. Under the GDPR this is a processor relationship that needs an agreement; see privacy and data protection and data residency.

Open-weight models and local inference

An open-weight model is a model whose trained weights are published for download, so anyone can run it on their own hardware. That is not the same as open source: the license may restrict use (Meta's Llama Community License is not an OSI-approved open-source license), and the training data and code are usually not published. Well-known open-weight families as of 2026 include Llama, Mistral, DeepSeek, Qwen, Google's Gemma and OpenAI's gpt-oss models; some are released under permissive licenses such as Apache 2.0 or MIT.

Local inference means running a model on hardware you control — a laptop, a workstation or your own servers — instead of calling a cloud API. Common tools, named here as examples:

• llama.cpp: an efficient inference engine written in C/C++, which also defined the GGUF file format for quantized models.
• Ollama: a user-friendly runner built on llama.cpp's GGML library; one command downloads and starts a model and exposes a local API.
• LM Studio (desktop app) and vLLM (a high-throughput server for GPUs).

ollama pull llama3.2      # download a small model (example name; names change)
ollama run llama3.2 "Summarize this error: ..."

The trade-offs:

• Privacy and control: prompts and documents never leave your machine or network, which can settle questions about personal data, confidential code or data residency. The model version only changes when you change it.
• Hardware: see the memory table above. Small models (a few billion parameters, 4-bit) run on a modern laptop, ideally with a GPU, an NPU or fast unified memory; larger models need serious GPUs.
• Quality: models that fit on a laptop are noticeably less capable than frontier cloud models, especially for complex reasoning and long contexts.
• Operations: you patch, monitor, secure and update it. A downloaded model file is software from the internet: take it from a trusted source and check it like any other dependency (see supply chain attacks).

Energy use

AI runs in data centers, and data centers use electricity and water for cooling. The International Energy Agency (IEA) gives the most-cited figures: in its April 2025 report Energy and AI, it estimated that data centers used about 415 TWh in 2024, roughly 1.5% of the world's electricity. Its 2026 update put 2025 at about 485 TWh (+17%, with AI-focused data centers growing about 50%) and projects about 950 TWh by 2030 in its base case, roughly 3% of global electricity demand, with AI as the most important driver.

These are projections, and they depend on assumptions about chip efficiency, demand and how fast new data centers and grid connections can be built; other forecasts differ widely. Two trends pull in opposite directions: the energy needed per AI task is falling fast thanks to better chips and smaller models, while reasoning models and video generation use far more energy per request than a simple text answer. Locally the effect can be large even when the global share is small: data centers cluster in a few regions, where they compete for grid capacity. The ethical side of this is discussed under responsible AI.

Choosing: API or self-hosting

Many organizations combine both: a cloud API for general assistants and complex tasks, and a small local or private model for sensitive data or high-volume simple tasks. Whatever you choose, budget for it the way you would for any cloud workload (see cloud costs and FinOps): set spending limits on API keys, because a loop in an agent can spend a month's budget in an afternoon.