Compute in the Cloud

Virtual machines as a service

The most basic cloud service is a server you did not have to buy. A cloud instance is a virtual machine (VM) that runs on the provider's hardware and that you create, start, stop and delete through a portal or an application programming interface (API). Underneath it is ordinary virtualization: a hypervisor shares one physical host between many VMs (see Virtualization and Hypervisors).

This is infrastructure as a service (IaaS). The provider is responsible for the hardware, the data center and the hypervisor; you are responsible for everything from the operating system upward: patches, firewall rules on the instance, the software you install and the data on it (see Service Models).

Creating an instance means making a few choices:

• a region and availability zone (where it runs);
• an instance type (how big it is);
• a machine image (what it starts from);
• a virtual network and subnet, and the firewall rules that apply (see Cloud Networking);
• how you log in: a Secure Shell (SSH) key pair for Linux, or an administrator password or directory account for Windows;
• disks, and optionally a boot script.

$ aws ec2 run-instances \
    --image-id ami-0123456789abcdef0 \
    --instance-type t3.micro \
    --count 1 \
    --metadata-options "HttpTokens=required,HttpEndpoint=enabled" \
    --user-data file://cloud-init.yaml \
    --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=web-01}]' \
    --query "Instances[0].InstanceId" --output text

i-0a1b2c3d4e5f67890

Without a subnet or security group in the command, AWS uses the account's default network. The metadata option requires the newer, token-based version of the instance metadata service (IMDSv2), which is a recommended hardening step. The same in the Azure command-line interface (CLI):

$ az vm create \
    --resource-group rg-web \
    --name web-01 \
    --image Ubuntu2404 \
    --size Standard_B2s \
    --admin-username azureuser \
    --generate-ssh-keys \
    --public-ip-address "" \
    --nsg-rule NONE \
    --custom-data cloud-init.yaml

By default az vm create adds a public Internet Protocol (IP) address and opens SSH to the whole internet; --public-ip-address "" and --nsg-rule NONE prevent that, and you reach the VM through Azure Bastion or a virtual private network (VPN) instead. See the first common mistake in Cloud Networking.

Instance types

An instance type (Azure: VM size; Google Cloud: machine type) is a fixed combination of virtual central processing units (vCPUs), memory and often network and disk performance, with its own price per hour. You cannot pick "3 vCPUs and 5 GB"; you pick the type that fits best. Most providers group types into families by their balance of resources:

Names encode the details. AWS m7i.large is family M (general purpose), generation 7, i for an Intel processor, size large (2 vCPUs, 8 GiB). Azure Standard_D4s_v5 is the D family with 4 vCPUs, s for premium-storage support, version 5. Google Cloud n2-standard-4 is the N2 series with 4 vCPUs and the standard memory ratio. Newer generations usually give more performance per unit of price, so check that you are not running on an old generation out of habit.

Burstable types (AWS T, Azure B, Google Cloud's shared-core E2 types) are cheap because they assume the CPU is mostly idle. They earn credits while idle and spend them in short bursts; a burstable instance that is busy all day runs out of credits and is either throttled or charged extra.

What a vCPU is

A vCPU is usually not a whole processor core. On most Intel and AMD instance types, one vCPU is one hardware thread, so two vCPUs share one physical core through simultaneous multithreading (see CPUs). On many Arm-based types, such as AWS Graviton, one vCPU is a full core. That is why "4 vCPUs" on two different types can perform quite differently: compare benchmarks or test your own workload rather than counting vCPUs.

Images and boot configuration

Machine images

A machine image is a template for an instance's boot disk: an operating system, often with preinstalled software and settings, plus metadata such as the processor architecture. AWS calls it an Amazon Machine Image (AMI); Azure and Google Cloud simply call it an image. Every new instance starts as a copy of one.

• Public images come from the provider or the operating system vendor: Ubuntu, Debian, Red Hat Enterprise Linux, Windows Server. They are updated regularly.
• Marketplace images contain commercial software, sometimes with its license cost added to the hourly price.
• Custom images are made by you from an instance you configured. Azure keeps them in an Azure Compute Gallery; Google Cloud groups versions into image families.

A golden image is a custom image that an organization builds and tests as its standard: patched, hardened, with the monitoring agent installed. It is rebuilt automatically on a schedule, often with a tool such as HashiCorp Packer, so that new instances never start from an outdated base.

User data and cloud-init

An image is the same for every instance. The last steps that differ per instance or change often are done at first boot by a script passed as user data (Azure: custom data). On Linux, the standard tool that reads it is cloud-init, preinstalled in most cloud images. It accepts a shell script or a YAML (YAML Ain't Markup Language) file starting with #cloud-config:

#cloud-config
package_update: true
packages:
  - nginx
write_files:
  - path: /var/www/html/index.html
    content: |
      <h1>web-01 is running</h1>
runcmd:
  - systemctl enable --now nginx

This updates the package lists, installs nginx, writes a start page and starts the service. On Windows, the equivalent is a PowerShell script run by the provider's agent (EC2Launch on AWS).

The trade-off between the two: everything in the image makes boot fast and predictable; everything in user data keeps the image generic but makes boot slower and dependent on package mirrors being reachable. A common balance is a golden image with the software installed and user data only for per-environment settings.

Instance storage and attached disks

An instance can have two very different kinds of disk:

• Attached network disks (AWS Elastic Block Store, EBS; Azure managed disks; Google Cloud Persistent Disk and Hyperdisk) live on separate storage systems and are reached over the network. They survive a stop and start, can be detached and attached to another instance, and can be backed up with snapshots. The boot disk is normally one of these.
• Local instance storage (AWS instance store, the Azure temporary disk, Google Cloud Local SSD) is physically inside the host. It is very fast, often included in the price — and ephemeral.

Ephemeral means the data disappears when the instance stops, is deallocated, moves to another host or the host fails. A normal reboot usually keeps it. Use it only for data you can lose: caches, scratch space, temporary files, or copies of data that also live elsewhere. Details per storage type are on Cloud Storage.

Scaling

Vertical scaling

Vertical scaling (scaling up or down) means giving one instance more or fewer resources by changing its instance type, for example from 2 to 8 vCPUs. It needs no change to the application, which makes it the natural choice for software that cannot run as several copies, such as many traditional databases. Its limits: the instance usually has to stop and start to change type, and there is always a largest type.

Horizontal scaling

Horizontal scaling (scaling out or in) means adding or removing instances that do the same work, with a load balancer spreading requests over them. It has no hard ceiling, adds redundancy (one instance can fail without an outage), and can happen without downtime. It requires an application designed for it: instances must be stateless, keeping sessions, uploads and data in a shared database, cache or storage service instead of on their own disk.

Autoscaling

Autoscaling is horizontal scaling done automatically: a group of identical instances, built from one launch template (image, instance type, user data), grows and shrinks according to rules. The group has three numbers:

• minimum: never fewer instances than this;
• maximum: never more, which also caps the cost;
• desired: how many there should be right now. The policies change this number; the group then launches or terminates instances to match it.

Common policies:

• Target tracking: keep a metric near a value, such as average CPU at 50%. The group calculates how many instances that needs.
• Scheduled: change the minimum or desired count at set times, for example more instances before the working day starts.
• Step or simple scaling: add or remove a fixed number of instances when an alarm fires.
• Predictive scaling, offered by some providers: forecast the load from past days and scale in advance.

The group also replaces instances that fail their health checks, which makes autoscaling useful even for a group with a fixed size. Groups scale in more cautiously than they scale out, because removing capacity too early hurts users and adding it too late does as well.

Simulating a day of autoscaling

This simulation applies a target-tracking policy (50% CPU) with a minimum of 2, a maximum of 10, and a scheduled action that keeps at least 6 instances during office hours. Each instance has 2 vCPUs; the load is the CPU demand in busy vCPUs per hour. To keep it short, it scales in by at most one instance per hour, a simple stand-in for the cool-down behavior of real groups.

import math

# Average CPU demand of a web application per hour, in "busy vCPUs".
load = [1.2, 0.9, 0.8, 0.8, 0.9, 1.4, 2.6, 4.8, 7.5, 8.8, 9.4, 9.9,
        10.6, 10.2, 9.1, 8.7, 8.9, 9.6, 11.8, 13.1, 12.0, 7.9, 4.2, 2.1]

VCPUS_PER_INSTANCE = 2
TARGET_CPU = 0.50            # target tracking: keep average CPU near 50%
MIN_SIZE, MAX_SIZE = 2, 10
BUSINESS_HOURS = range(7, 18)
SCHEDULED_MIN = 6            # scheduled action: at least 6 from 07:00 to 18:00

def desired_capacity(busy_vcpus, hour):
    wanted = math.ceil(busy_vcpus / (VCPUS_PER_INSTANCE * TARGET_CPU))
    floor = SCHEDULED_MIN if hour in BUSINESS_HOURS else MIN_SIZE
    return min(MAX_SIZE, max(floor, wanted))

instances = MIN_SIZE
instance_hours = 0
print(f"{'hour':5} {'load':>5} {'instances':>9} {'avg CPU':>8}")
for hour, busy in enumerate(load):
    target = desired_capacity(busy, hour)
    # Scale out at once, but scale in by at most one instance per hour.
    instances = target if target > instances else max(target, instances - 1)
    instance_hours += instances
    cpu = busy / (instances * VCPUS_PER_INSTANCE)
    print(f"{hour:02d}:00 {busy:5.1f} {instances:9} {cpu:8.0%}")

print(f"Instance-hours with autoscaling: {instance_hours}")
print(f"Instance-hours at a fixed {MAX_SIZE}: {MAX_SIZE * len(load)}")

hour   load instances  avg CPU
00:00   1.2         2      30%
01:00   0.9         2      22%
02:00   0.8         2      20%
03:00   0.8         2      20%
04:00   0.9         2      22%
05:00   1.4         2      35%
06:00   2.6         3      43%
07:00   4.8         6      40%
08:00   7.5         8      47%
09:00   8.8         9      49%
10:00   9.4        10      47%
11:00   9.9        10      50%
12:00  10.6        10      53%
13:00  10.2        10      51%
14:00   9.1        10      45%
15:00   8.7         9      48%
16:00   8.9         9      49%
17:00   9.6        10      48%
18:00  11.8        10      59%
19:00  13.1        10      66%
20:00  12.0        10      60%
21:00   7.9         9      44%
22:00   4.2         8      26%
23:00   2.1         7      15%
Instance-hours with autoscaling: 170
Instance-hours at a fixed 10: 240

Three things to read from the output:

• At 07:00 the schedule, not the load, sets the count: six instances are ready before the morning peak.
• From 18:00 to 20:00 the group hits its maximum and CPU rises above the target. The maximum protects the budget, at the price of slower responses; raise it if that is not acceptable.
• The cautious scale-in leaves spare instances late in the evening. Overall the group uses 170 instance-hours instead of 240 for a fixed fleet sized for the peak — about 30% less.

Pricing options

The same instance can be paid for in several ways. The structures below are common to the large providers at the time of writing; exact discounts and terms change, so check the provider's pricing pages. Costs as a whole are covered in Cloud Costs and FinOps.

Spot capacity

A spot instance runs on the provider's spare capacity at a steep discount, on the condition that the provider can stop or delete it when it needs the capacity back. At the time of writing, AWS gives a two-minute warning before such an interruption, and Azure Spot VMs and Google Cloud Spot VMs about 30 seconds. Google Cloud's older preemptible VMs, which also ran for at most 24 hours, have been succeeded by Spot VMs.

Spot capacity suits work that can stop and resume: a rendering job that saves its progress, or one of twenty identical web workers whose loss the others absorb. It does not suit a single database server. Many autoscaling groups mix the two: a base of on-demand or reserved instances, with spot instances on top for the peaks.

Running instances day to day

Right-sizing

Instances are often created too large "to be safe" and never revisited. Right-sizing means comparing an instance's actual use — CPU, memory, network over several weeks — with its type, and moving to a smaller or different family when it is mostly idle. The providers' advisor tools (AWS Compute Optimizer, Azure Advisor, Google Cloud's recommender) suggest candidates. Measure memory too: it is not reported by default on every platform and needs an agent inside the instance.

Stopping versus terminating

• Stop: the instance shuts down and you stop paying for compute time. Its attached disks remain and are still billed, and so may reserved public IP addresses. Local instance storage is lost. When started again, it may run on a different host.
• Terminate (AWS) or delete (Azure, Google Cloud): the instance is gone for good. Depending on its settings, its boot disk is deleted with it, while extra data disks may be left behind — still costing money until you delete them.

For instances that must never be deleted by accident, turn on termination or delete protection. For everything else, the healthier habit is to treat instances as replaceable: anything that matters lives in images, code and attached storage, so a deleted instance can be recreated in minutes.

Names across providers