Helm, Kustomize and GitOps

One application, many environments

The Kestrel meter-api needs a handful of Kubernetes objects: a Deployment, a Service, a ConfigMap, perhaps an HPA and network policies. It also runs in a test and a production environment that differ in a few places: the number of replicas, the image tag, the log level, the namespace. Copying all YAML files per environment works for a week; then the copies drift apart, and nobody knows which differences are intentional.

Two tools solve this in opposite ways, and a third practice decides how the result reaches the cluster:

• Helm templates: the YAML contains placeholders, filled in from values per environment.
• Kustomize patches: the base YAML stays plain and valid; each environment lists its changes on top.
• GitOps delivers: the desired state lives in Git and an agent keeps the cluster in line with it.

All three apply the ideas of infrastructure as code and declarative configuration to Kubernetes. Helm output on this page is illustrative (Helm was not installed on the test machine); the Kustomize output is real.

Helm

Helm is the package manager for Kubernetes: it installs, upgrades and removes applications made of many Kubernetes objects as one unit, comparable to what a Linux package manager does for software on a server. It is a CNCF (Cloud Native Computing Foundation) project and the usual way vendors ship software for Kubernetes: databases, monitoring stacks, ingress and gateway controllers.

Charts

A Helm chart is the package: a directory (or an archive, often stored in an OCI registry next to container images) with metadata, default values and templates.

portal/
├── Chart.yaml            name, chart version, app version
├── values.yaml           default values
└── templates/
    ├── deployment.yaml   templates that produce Kubernetes objects
    ├── service.yaml
    └── _helpers.tpl      shared snippets (names, labels)

Chart.yaml:

apiVersion: v2              # chart format; still valid in Helm 4
name: portal
description: Kestrel Energy customer portal
type: application
version: 0.3.0              # version of the chart itself (semantic versioning)
appVersion: "1.4.2"         # version of the application it deploys

The chart version and the application version are separate: a fix in a template gets a new chart version for the same application, following semantic versioning. A template mixes YAML with Go template syntax, so it is not valid YAML until Helm has rendered it:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "portal.fullname" . }}
  labels:
    {{- include "portal.labels" . | nindent 4 }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      {{- include "portal.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      labels:
        {{- include "portal.selectorLabels" . | nindent 8 }}
    spec:
      containers:
        - name: portal
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          {{- with .Values.extraEnv }}
          env:
            {{- toYaml . | nindent 12 }}
          {{- end }}
          resources:
            {{- toYaml .Values.resources | nindent 12 }}

Templates are powerful — loops, conditions, functions — and that is also their risk: a chart where every line is a condition is hard to read and harder to review. helm template renders a chart locally to plain YAML, which is the quickest way to see what it will really create; helm lint checks a chart for common errors.

Values

A values file is a YAML file with the settings a chart's templates read through .Values. The chart's own values.yaml holds the defaults; you override them per environment with -f values-production.yaml and, for single values, --set:

# values.yaml (inside the chart)
replicaCount: 1
image:
  repository: registry.example.com/kestrel/portal
  tag: "1.4.2"
resources:
  requests:
    cpu: 100m
    memory: 128Mi
extraEnv:
  - name: LOG_LEVEL
    value: debug
  - name: FEATURE_NEW_BILLS
    value: "on"

# values-production.yaml (in Kestrel's deployment repository)
replicaCount: 3
resources:
  requests:
    cpu: 250m
extraEnv:
  - name: LOG_LEVEL
    value: warning

Later sources win: chart defaults, then each -f file in order, then --set. The merge is deep for maps but not for lists, which is a classic surprise. This sketch merges the same values the way Helm does:

import json


def merge(base, override):
    """Combine values the way Helm does: maps merge key by key,
    everything else (strings, numbers, lists) is replaced."""
    result = dict(base)
    for key, value in override.items():
        if isinstance(value, dict) and isinstance(result.get(key), dict):
            result[key] = merge(result[key], value)
        else:
            result[key] = value
    return result


chart_defaults = {  # values.yaml inside the chart
    "replicaCount": 1,
    "image": {"repository": "registry.example.com/kestrel/portal", "tag": "1.4.2"},
    "resources": {"requests": {"cpu": "100m", "memory": "128Mi"}},
    "extraEnv": [{"name": "LOG_LEVEL", "value": "debug"},
                 {"name": "FEATURE_NEW_BILLS", "value": "on"}],
}
production_file = {  # -f values-production.yaml
    "replicaCount": 3,
    "resources": {"requests": {"cpu": "250m"}},
    "extraEnv": [{"name": "LOG_LEVEL", "value": "warning"}],
}
set_option = {"image": {"tag": "1.4.3"}}  # --set image.tag=1.4.3

values = merge(merge(chart_defaults, production_file), set_option)
print(json.dumps(values, indent=2))

{
  "replicaCount": 3,
  "image": {
    "repository": "registry.example.com/kestrel/portal",
    "tag": "1.4.3"
  },
  "resources": {
    "requests": {
      "cpu": "250m",
      "memory": "128Mi"
    }
  },
  "extraEnv": [
    {
      "name": "LOG_LEVEL",
      "value": "warning"
    }
  ]
}

memory: 128Mi survived, because resources is a map and was merged. But FEATURE_NEW_BILLS is gone: extraEnv is a list, and the production list replaced the default one completely. A production override of one list item silently drops the others. (In Helm, setting a key to null in an override removes it.)

Releases

A Helm release is one installation of a chart in a cluster, with a name, a namespace and a history of numbered revisions. Helm stores each revision in the cluster (as a Secret in the release's namespace), which is what makes helm rollback possible. One chart can be installed many times as different releases.

$ helm upgrade --install portal oci://registry.example.com/charts/portal \
    --version 0.3.1 -n kestrel-portal -f values-production.yaml \
    --rollback-on-failure
$ helm history portal -n kestrel-portal

Release "portal" has been upgraded. Happy Helming!
NAME: portal
LAST DEPLOYED: Tue Oct  6 10:31:07 2026
NAMESPACE: kestrel-portal
STATUS: deployed
REVISION: 2
TEST SUITE: None
REVISION  UPDATED                   STATUS      CHART         APP VERSION  DESCRIPTION
1         Tue Oct  6 10:12:31 2026  superseded  portal-0.3.0  1.4.2        Install complete
2         Tue Oct  6 10:31:07 2026  deployed    portal-0.3.1  1.4.3        Upgrade complete

upgrade --install installs the release if it does not exist and upgrades it otherwise, which makes it safe to run from a pipeline. --version pins the chart version; without it, Helm takes the newest one, and an upgrade of a vendor chart can arrive unannounced. --rollback-on-failure rolls the release back if the upgrade does not become healthy. helm rollback portal 1 returns to revision 1 by creating revision 3 with the old content.

Helm 4

Helm 4.0.0 was released on 12 November 2025, the first major version in six years; as of October 2026 the current release is 4.3.0. Existing charts (apiVersion: v2) keep working. The changes most people notice:

• Server-side apply is the default for new installs: the API server, not Helm, merges the changes and tracks which tool owns which field. Upgrades of existing releases keep the method they were installed with.
• Renamed flags: --atomic became --rollback-on-failure and --force became --force-replace. The old names still work with a warning; update scripts now.
• Charts can be installed by OCI digest, so a release references exactly one immutable chart; helm registry login takes only a domain name; post-renderers are now plugins, and plugins can run as WebAssembly.

Kustomize

Kustomize customizes plain Kubernetes YAML without templates: a kustomization.yaml file lists the resource files to use and the changes to make — namespace, labels, image tags, replica counts, patches, generated ConfigMaps. It is built into kubectl (since version 1.14): kubectl kustomize renders the result, and kubectl apply -k applies it. A standalone kustomize binary also exists (v5.8.2 as of October 2026). The version inside kubectl is shown by kubectl version:

$ kubectl version --client

Client Version: v1.37.1
Kustomize Version: v5.8.1

Everything in this section was recorded for real (only resource names were shortened): Kustomize renders locally and needs no cluster. (The image name kestrel-meter-api is a local build of the meter-api.)

A base

The base holds the parts every environment shares, as ordinary, valid Kubernetes YAML:

$ find . -type f | sort

./base/deployment.yaml
./base/kustomization.yaml
./base/service.yaml
./overlays/production/kustomization.yaml

base/deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: meter-api
spec:
  replicas: 1
  selector:
    matchLabels:
      app: meter-api
  template:
    metadata:
      labels:
        app: meter-api
    spec:
      containers:
        - name: meter-api
          image: kestrel-meter-api:1.1
          ports:
            - containerPort: 8000
          envFrom:
            - configMapRef:
                name: meter-api-config
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
          resources:
            requests:
              cpu: 50m
              memory: 64Mi
            limits:
              memory: 128Mi

base/service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: meter-api
spec:
  selector:
    app: meter-api
  ports:
    - port: 80
      targetPort: 8000

base/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml
labels:
  - pairs:
      app.kubernetes.io/part-of: kestrel-energy
configMapGenerator:
  - name: meter-api-config
    literals:
      - LOG_LEVEL=debug

labels adds a label to every object; configMapGenerator creates the ConfigMap the Deployment refers to with envFrom.

Overlays

A Kustomize overlay is a directory with its own kustomization.yaml that uses a base as a resource and lists only what is different for one environment:

overlays/production/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: kestrel-prod
resources:
  - ../../base
replicas:
  - name: meter-api
    count: 3
images:
  - name: kestrel-meter-api
    newTag: "1.2"
configMapGenerator:
  - name: meter-api-config
    behavior: merge
    literals:
      - LOG_LEVEL=warning

Production gets its own namespace, three replicas, image tag 1.2 and a quieter log level. Render it:

$ kubectl kustomize overlays/production

apiVersion: v1
data:
  LOG_LEVEL: warning
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/part-of: kestrel-energy
  name: meter-api-config-b6dh7b267t
  namespace: kestrel-prod
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/part-of: kestrel-energy
  name: meter-api
  namespace: kestrel-prod
spec:
  ports:
  - port: 80
    targetPort: 8000
  selector:
    app: meter-api
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/part-of: kestrel-energy
  name: meter-api
  namespace: kestrel-prod
spec:
  replicas: 3
  selector:
    matchLabels:
      app: meter-api
  template:
    metadata:
      labels:
        app: meter-api
    spec:
      containers:
      - envFrom:
        - configMapRef:
            name: meter-api-config-b6dh7b267t
        image: kestrel-meter-api:1.2
        name: meter-api
        ports:
        - containerPort: 8000
        readinessProbe:
          httpGet:
            path: /health
            port: 8000
        resources:
          limits:
            memory: 128Mi
          requests:
            cpu: 50m
            memory: 64Mi

Comparing the rendered base with the rendered overlay shows exactly what the overlay changes (diff exits with status 1 because there are differences):

$ diff <(kubectl kustomize base) <(kubectl kustomize overlays/production)

3c3
<   LOG_LEVEL: debug
---
>   LOG_LEVEL: warning
8c8,9
<   name: meter-api-config-47668c6k28
---
>   name: meter-api-config-b6dh7b267t
>   namespace: kestrel-prod
15a17
>   namespace: kestrel-prod
28a31
>   namespace: kestrel-prod
30c33
<   replicas: 1
---
>   replicas: 3
42,43c45,46
<             name: meter-api-config-47668c6k28
<         image: kestrel-meter-api:1.1
---
>             name: meter-api-config-b6dh7b267t
>         image: kestrel-meter-api:1.2

Three things to notice in the real output:

• The ConfigMap is called meter-api-config-b6dh7b267t, not meter-api-config. The generator appends a hash of the content, and rewrites the reference in the Deployment to match. Change a value and the name changes, so the Deployment's pod template changes and Kubernetes rolls out new pods that read the new configuration — the hash is how Kustomize avoids the "edited the ConfigMap, nothing happened" problem described on ConfigMaps, Secrets and Configuration.
• The overlay never repeated the Deployment; it only named changes. The base stays valid YAML that you can read without knowing Kustomize.
• The output is sorted by kind (ConfigMap, Service, Deployment) and fields alphabetically, so diffs of rendered output stay small and readable.

Helm or Kustomize?

Many teams use both: vendor software through Helm charts, their own services with Kustomize, and occasionally Kustomize patches on top of a rendered chart. Whatever the mix, make one tool the owner of each object; two tools changing the same Deployment end in a tug of war.

GitOps with Argo CD and Flux

Rendering is half the job; the result also has to reach the cluster, every time, the same way. GitOps does that with a pull-based agent inside the cluster that watches a Git repository and runs a reconciliation loop: render the charts and overlays from Git, compare with what is running, and apply the difference. The principles, push versus pull, promotion between environments and drift are explained on Deployment Pipelines and GitOps; this section shows what it looks like in Kubernetes terms.

In Argo CD (v3.5.3 as of October 2026), an Application object points at a path in a repository and a destination in the cluster. Argo CD recognizes a Kustomize directory or a Helm chart by itself:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: meter-api-production
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://git.example.com/kestrel/deploy.git
    targetRevision: main
    path: meter-api/overlays/production
  destination:
    server: https://kubernetes.default.svc   # the cluster Argo CD runs in
    namespace: kestrel-prod
  syncPolicy:
    automated:
      prune: true      # delete objects that were removed from Git
      selfHeal: true   # undo manual changes in the cluster

Flux (v2.9.6 as of October 2026) splits the same job in two objects: a source that fetches the repository, and a Flux Kustomization that applies a path from it:

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: kestrel-deploy
  namespace: flux-system
spec:
  interval: 1m
  url: https://git.example.com/kestrel/deploy.git
  ref:
    branch: main
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: meter-api-production
  namespace: flux-system
spec:
  interval: 10m
  sourceRef:
    kind: GitRepository
    name: kestrel-deploy
  path: ./meter-api/overlays/production
  prune: true
  targetNamespace: kestrel-prod

What changes in daily work once GitOps is in place:

• Deploying is merging. Yara Haddad's release of meter-api 1.2 to production is a pull request that changes newTag in the production overlay; review and approval happen there, and the merge is the deployment.
• Rolling back is reverting. git revert of that commit brings the old tag back. A helm rollback or kubectl rollout undo in the cluster is undone by the agent at its next sync, because Git still says otherwise.
• Manual changes do not stick. With self-heal on, a kubectl edit is reverted within minutes; with it off, it shows up as drift. Either way, an emergency change must also go into Git.

Secrets and the supply chain

• No real secrets in values files or overlays. They end up in Git, in Helm's release history in the cluster and in rendered output in CI logs. Use an external secret store, or encrypted secrets designed for Git — see secrets in GitOps and ConfigMaps, Secrets and Configuration.
• Pin what you install. A chart version (or, with Helm 4, a chart digest) and image tags or digests in Git mean that the same commit always deploys the same thing.
• Treat charts as code you run. A third-party chart can create ClusterRoles, privileged pods and webhooks; render it with helm template and read what it creates before installing it, and take charts only from sources you trust.
• The deployment tools are part of the supply chain. In March 2026 attackers published malicious versions of the widely used Trivy scanner and repointed tags of its GitHub Action — see Container Security. Install Helm, kubectl and agents from official releases with checksums, and pin CI actions by commit.