DevOps and CI/CD

This page gives the cloud view of DevOps and CI/CD. The Git & CI/CD topic covers pipelines in depth: GitHub Actions, testing, deployment pipelines, GitOps and other CI/CD platforms.

What DevOps is

Traditionally, developers wrote software and "threw it over the wall" to an operations team that had to run it. Developers were rewarded for change, operations for stability, so releases became rare, large and frightening. DevOps is a culture and set of practices in which the people who build software and the people who run it work as one team with shared goals, so that changes can be delivered quickly and reliably. It is often summarized as four ideas:

• Collaboration: shared responsibility for the software in production — "you build it, you run it".
• Automation: builds, tests, infrastructure and deployments done by machines, the same way every time.
• Measurement: decisions based on data about delivery speed, failures and user experience.
• Sharing: knowledge, tools and blameless reviews of incidents, so the whole organization learns.

DevOps is not a product you buy, and a "DevOps engineer" who maintains pipelines alone does not make an organization DevOps if development and operations still work against each other. The tools on this page are enablers; the culture is the point.

The software delivery lifecycle

In DevOps the lifecycle is a loop rather than a line: plan, code, build, test, release, deploy, operate, monitor — and what is learned from monitoring feeds the next plan. The middle part, from a commit to running in production, is automated as a pipeline. The smaller each trip around the loop, the less can go wrong in it and the faster a problem is traced to its cause.

Continuous integration

Continuous integration (CI) is the practice of merging every developer's changes into the shared main branch frequently — at least daily — where each merge is automatically built and tested. Merging small changes often keeps conflicts small; testing each one means a broken build is noticed within minutes and traced to one change, instead of surfacing weeks later in a painful "integration phase".

A CI run typically:

1. Checks out the exact commit (see version control with Git).
2. Installs dependencies at pinned versions.
3. Runs linters and formatters.
4. Runs the automated tests.
5. Reports the result on the pull request; a red result blocks the merge.

CI only works if the tests are trusted: a suite that fails at random ("flaky" tests) teaches people to ignore red builds.

Continuous delivery versus continuous deployment

Both are abbreviated CD, and they differ in one step. Continuous delivery means every change that passes the pipeline is automatically built, tested and deployed to a production-like environment, so it is always ready to release; the release to production itself is a decision, usually a manual approval. With continuous deployment there is no manual step: every change that passes all automated checks goes to production on its own.

Pipelines

A pipeline is an automated, ordered series of stages that takes a code change from commit to deployment, where each stage must succeed before the next one starts. A typical one:

• Build: compile or assemble the code.
• Test: unit tests, then integration tests.
• Scan: look for known-vulnerable dependencies, committed secrets and insecure code patterns.
• Package: produce one versioned artifact, such as a container image.
• Deploy to staging: install that exact artifact in a production-like environment and run smoke tests.
• Approve: a person signs off (continuous delivery only).
• Deploy to production: the same artifact again, using one of the strategies below.

Common pipeline platforms are GitHub Actions, GitLab CI/CD, Jenkins, and the providers' own services: Azure Pipelines (part of Azure DevOps), AWS (Amazon Web Services) CodePipeline with CodeBuild, and Google Cloud Build.

An example: GitHub Actions

A GitHub Actions workflow is a YAML file in .github/workflows/. This one tests every push and pull request; on the main branch it also builds a container image and deploys it to staging and then production.

name: ci-cd

on:
  push:
    branches: [main]
  pull_request:

permissions:
  contents: read

jobs:
  test:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: python -m pytest

  build:
    needs: test
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-24.04
    permissions:
      contents: read
      packages: write
    steps:
      - uses: actions/checkout@v7
      - uses: docker/login-action@v4
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - uses: docker/build-push-action@v7
        with:
          push: true
          tags: ghcr.io/example-corp/shop-web:${{ github.sha }}

  deploy-staging:
    needs: build
    runs-on: ubuntu-24.04
    environment: staging
    steps:
      - uses: actions/checkout@v7
        with:
          persist-credentials: false  # deploy step needs no Git token
      - run: ./scripts/deploy.sh staging "ghcr.io/example-corp/shop-web:${{ github.sha }}"
        env:
          DEPLOY_TOKEN: ${{ secrets.DEPLOY_TOKEN }}

  deploy-production:
    needs: deploy-staging
    runs-on: ubuntu-24.04
    environment: production
    steps:
      - uses: actions/checkout@v7
        with:
          persist-credentials: false  # deploy step needs no Git token
      - run: ./scripts/deploy.sh production "ghcr.io/example-corp/shop-web:${{ github.sha }}"
        env:
          DEPLOY_TOKEN: ${{ secrets.DEPLOY_TOKEN }}

• on sets the triggers: pushes to main and every pull request.
• The top-level permissions gives the automatic GITHUB_TOKEN read-only access to the repository. Only the build job raises this, and only to what it needs: packages: write to push the image. Without this block the token may get broader default rights.
• Each job runs on a fresh virtual machine (runs-on). Pinning a specific Ubuntu version instead of ubuntu-latest makes builds reproducible; many examples use ubuntu-latest for brevity. needs orders the jobs into stages; if skips the build for pull requests.
• uses runs a published action, pinned to a major version (the current ones at the time of writing). Actions run with your job's permissions and secrets, so pinning matters: the strictest practice is to pin each action to a full commit SHA (Secure Hash Algorithm) digest, which cannot be moved to different code later.
• The image tag is the commit SHA (github.sha), so every image is traceable to exactly one commit and the same image moves through staging and production.
• environment: staging and environment: production link the jobs to GitHub environments. Required reviewers configured on the production environment (in the repository settings, not in this file) turn that job into the manual approval step.
• ${{ secrets.DEPLOY_TOKEN }} reads a secret stored in GitHub's encrypted secret store, scoped per environment. The value never appears in the file and is masked in logs. Better still, many clouds let a workflow exchange a short-lived OIDC (OpenID Connect) token for temporary cloud credentials (the job then needs permissions: id-token: write), so no long-lived secret is stored at all.

Artifacts and registries

An artifact is the versioned output of a build: a container image, a Python wheel, a zip file for a function. The rule is build once, deploy many: the artifact tested in staging is the one that goes to production, never a rebuild that could pick up a different dependency. Artifacts are stored in a registry or repository — a container registry such as Amazon ECR (Elastic Container Registry), Azure Container Registry, Google Artifact Registry or GitHub Container Registry, or a package repository. Keep old versions: they are what a rollback deploys.

Deployment strategies

Even with good tests, some problems only show up under real traffic. A deployment strategy decides how much of that traffic meets a new version, and how quickly you can go back.

Rolling deployment

In a rolling deployment, the instances running the old version are replaced by the new version a few at a time, until all run the new one. It needs no extra capacity beyond a small surplus, and it is the default in Kubernetes deployments and many autoscaling groups (see Kubernetes basics). For a while both versions serve users, so they must be compatible, and a rollback is another rolling update in the other direction. How a Kubernetes Deployment does it, with maxSurge and maxUnavailable: rolling updates.

Blue-green deployment

A blue-green deployment runs two identical production environments: "blue" serves the current version, while the new version is installed and tested on "green". Then the router or load balancer switches all traffic to green at once. Rollback is switching back, which takes seconds, as long as blue is kept running for a while. The price is double capacity during the switch, and care with databases, which both versions usually share.

Canary release

A canary release sends a small share of real traffic — say 5% — to the new version, compares its error rate and latency with the old version, and only increases the share step by step while the numbers stay healthy. If they do not, all traffic goes back to the old version. The name comes from the canaries miners took underground as an early warning: a bad release hurts a few users, not all of them.

Canary decisions are usually automated: the pipeline reads the metrics after each step and promotes or rolls back. This simulation compares the canary's error rate with the stable version's (the baseline) at each step, and rolls back if the canary is more than half a percentage point worse:

STEPS = [5, 25, 50]  # percent of traffic sent to the new version
MAX_EXTRA_ERRORS = 0.5    # canary may be at most 0.5 points worse than baseline


def canary_release(version, measurements):
    """measurements: per step, (baseline errors, canary errors, requests each)."""
    print(f"Releasing {version}")
    for percent, (base_err, new_err, requests) in zip(STEPS, measurements):
        base_rate = 100 * base_err / requests
        new_rate = 100 * new_err / requests
        verdict = "ok" if new_rate - base_rate <= MAX_EXTRA_ERRORS else "too many errors"
        print(f"  {percent:>3}% traffic: baseline {base_rate:.2f}%, canary {new_rate:.2f}% -> {verdict}")
        if verdict != "ok":
            print("  ROLL BACK: all traffic back to the old version")
            return False
    print(f"  PROMOTED: {version} now serves all traffic")
    return True


canary_release("v2.4.0", [(12, 14, 2000), (11, 13, 2000), (13, 12, 2000)])
canary_release("v2.5.0", [(12, 15, 2000), (11, 38, 2000)])

Releasing v2.4.0
    5% traffic: baseline 0.60%, canary 0.70% -> ok
   25% traffic: baseline 0.55%, canary 0.65% -> ok
   50% traffic: baseline 0.65%, canary 0.60% -> ok
  PROMOTED: v2.4.0 now serves all traffic
Releasing v2.5.0
    5% traffic: baseline 0.60%, canary 0.75% -> ok
   25% traffic: baseline 0.55%, canary 1.90% -> too many errors
  ROLL BACK: all traffic back to the old version

Version 2.5.0 looked fine at 5% but failed at 25%, so only a quarter of users met the problem, for a short time. Real canary analysis uses more requests per step, several metrics (errors, latency, saturation) and statistical tests, because with few requests a small difference can be pure chance.

Feature flags

A feature flag (or feature toggle) is a setting, checked by the code at run time, that turns a feature on or off without deploying new code. It separates deploying code from releasing a feature: unfinished work can be merged and deployed switched off (which keeps continuous integration possible), a new feature can be enabled for internal users or 10% of customers first, and a misbehaving feature can be switched off in seconds. Flags are managed in a configuration service or a dedicated product, such as Azure App Configuration or AWS AppConfig. Remove flags once a feature is fully released, because every flag doubles the paths through the code that need testing.

Infrastructure and configuration in the pipeline

The same pipeline principles apply to the platform the application runs on. With infrastructure as code, a change to a network or a database setting is a pull request: the pipeline runs terraform plan (or a CloudFormation change set, or a Bicep what-if) and posts the result for reviewers, and after the merge it applies exactly that plan. The details — saved plans, credentials, approvals and drift jobs — are on IaC in Pipelines. Kubernetes manifests and application configuration follow the same path. Some teams go one step further with GitOps: an agent in the environment continuously makes it match what is in a Git repository, so Git is the single source of truth and every change is a commit.

Measuring delivery: the DORA metrics

The DevOps Research and Assessment (DORA) program, now part of Google Cloud, studies thousands of teams each year. Its research found that a few measures separate high-performing teams from the rest, and that speed and stability go together rather than against each other:

• Deployment frequency: how often changes reach production.
• Change lead time (formerly lead time for changes): how long from commit to running in production.
• Change fail rate (formerly change failure rate): the share of deployments that cause a failure needing a fix or rollback.
• Failed deployment recovery time (formerly time to restore service): how long it takes to recover from such a failure.

DORA added a fifth, deployment rework rate, in 2024; the first four remain the usual core. Use them to see whether a team is improving, not to compare or rank teams: any number that becomes a target is easily gamed.