GitOps with Helm

Lesson 19 — Using Git as the single source of truth for Helm-based Kubernetes deployments

1. GitOps Principles

GitOps is an operational model where the desired state of your infrastructure and applications is declared in a Git repository. An in-cluster agent continuously compares the live state against the declared state and reconciles any differences. Four principles define the pattern:

1. Declarative — the system's desired state is expressed declaratively (Kubernetes manifests, Helm charts, Kustomize overlays).
2. Versioned and immutable — the desired state is stored in Git, giving you a full audit trail, blame history, and the ability to reproduce any past state.
3. Pulled automatically — agents inside the cluster pull the desired state from Git, rather than having CI pipelines push to the cluster. This eliminates the need to expose cluster credentials to external systems.
4. Continuously reconciled — agents monitor for drift between live and desired state and automatically correct it.

Helm fits naturally into this model. Charts provide the packaging and templating layer, while GitOps tools provide the delivery and reconciliation layer. Values files per environment (staging, production) live in Git alongside chart references, making environment promotion a pull request.

2. ArgoCD with Helm

Application CRD

ArgoCD models each deployment as an Application custom resource. When the source is a Helm chart, you configure it under spec.source.helm:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: oci://registry.example.com/charts
    chart: my-app
    targetRevision: 1.4.0
    helm:
      releaseName: my-app
      valueFiles:
        - values-production.yaml
      parameters:
        - name: image.tag
          value: "v2.3.1"
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Key fields under source.helm:

• valueFiles — a list of values files relative to the chart or Git repo root. ArgoCD merges them in order.
• parameters — individual value overrides, equivalent to --set on the CLI.
• releaseName — the Helm release name. If omitted, ArgoCD uses the Application name.

The syncPolicy.automated block enables continuous reconciliation. With selfHeal: true, any manual kubectl change is reverted. With prune: true, resources removed from Git are deleted from the cluster.

ApplicationSet

When you manage dozens of applications or multiple clusters, defining individual Application resources becomes tedious. ApplicationSet solves this with generators:

• Git directory generator — creates one Application per subdirectory in a Git repo. If your repo has apps/redis/, apps/nginx/, and apps/api/, three Applications are generated automatically.
• Cluster generator — creates Applications for every cluster registered in ArgoCD, optionally filtered by label selectors.
• Matrix generator — combines two generators (e.g., Git directories x clusters) to produce the cross-product.

3. FluxCD with Helm

HelmRepository Source

Flux separates sources (where charts come from) from releases (how they are deployed). A HelmRepository defines a chart source:

apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: bitnami
  namespace: flux-system
spec:
  type: oci
  url: oci://registry-1.docker.io/bitnamicharts
  interval: 10m

Flux supports both traditional Helm repos (index.yaml-based) and OCI registries. The interval controls how often Flux checks for new chart versions.

HelmRelease

A HelmRelease resource declares the desired Helm release:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: redis
  namespace: caching
spec:
  interval: 30m
  chart:
    spec:
      chart: redis
      version: "18.x"
      sourceRef:
        kind: HelmRepository
        name: bitnami
        namespace: flux-system
  valuesFrom:
    - kind: ConfigMap
      name: redis-values
    - kind: Secret
      name: redis-secret-values
  values:
    architecture: replication
    replica:
      replicaCount: 3

The valuesFrom field is particularly powerful. It lets you pull values from ConfigMaps and Secrets, keeping sensitive configuration (database passwords, API keys) separate from the HelmRelease manifest. Inline values are merged last, providing defaults and non-sensitive overrides.

Image Update Automation

Flux can automatically update image tags in Git when new container images are pushed to a registry. The pipeline involves three resources:

1. ImageRepository — scans a container registry for new tags.
2. ImagePolicy — selects the latest tag matching a semver range or filter pattern.
3. ImageUpdateAutomation — commits the updated tag back to Git, which then triggers a HelmRelease reconciliation.

You mark the line in your values file with a comment marker so Flux knows which field to update:

image:
  repository: ghcr.io/org/my-app
  tag: 1.2.3  # {"$imagepolicy": "flux-system:my-app"}

4. Drift Detection and Self-Healing

Both ArgoCD and FluxCD continuously compare the live cluster state against the desired state in Git. When someone runs kubectl edit or kubectl scale directly, the tool detects the drift:

• ArgoCD marks the Application as OutOfSync. With selfHeal: true, it automatically re-syncs to the Git-declared state.
• FluxCD re-applies the HelmRelease on each interval. Manual changes are overwritten at the next reconciliation cycle.

This self-healing property is a core benefit of GitOps: the cluster always converges to the declared state, eliminating configuration drift that accumulates over time.

5. Rollback Strategies

In a GitOps workflow, rollbacks follow Git semantics rather than Helm's built-in helm rollback:

6. Helm Template in CI vs Native Helm Support

There are two schools of thought for using Helm in a GitOps pipeline:

Rendered Manifests (helm template in CI)

• CI runs helm template and commits the rendered YAML to a "deploy" branch or directory.
• The GitOps agent applies plain manifests — no Helm awareness needed.
• PR diffs show the exact Kubernetes resources that will change.
• Downside: you lose Helm release metadata, hooks, and the ability to use helm list.

Native Helm Support

• ArgoCD and Flux render the chart at sync time using their built-in Helm integration.
• Charts remain first-class objects with proper release tracking.
• Helm hooks (pre-install, post-upgrade) are executed by the agent.
• Downside: PR diffs show values file changes, not the resulting manifest changes (though both tools offer diff previews).

7. Secrets in GitOps

Storing secrets in Git requires encryption. Two dominant approaches:

SOPS with FluxCD

Mozilla SOPS encrypts YAML/JSON values while leaving keys readable. Flux has native SOPS support — it decrypts at reconciliation time using age keys or cloud KMS:

# Encrypt a values file
sops --encrypt --age $(cat age-key.pub) values-secret.yaml > values-secret.enc.yaml

# Reference in HelmRelease
# valuesFrom:
#   - kind: Secret
#     name: helm-secrets
#     valuesKey: values-secret.yaml

ArgoCD Vault Plugin (AVP)

AVP replaces placeholder strings in manifests with values fetched from HashiCorp Vault, AWS Secrets Manager, or other backends at sync time:

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  annotations:
    avp.kubernetes.io/path: "secret/data/production/db"
stringData:
  password: 

8. Multi-Cluster GitOps

Scaling GitOps across clusters typically follows one of two patterns:

• Hub-and-spoke — a central management cluster runs ArgoCD/Flux and deploys to remote clusters. ArgoCD's ApplicationSet with a cluster generator is ideal here.
• Per-cluster agents — each cluster runs its own Flux instance pointing to the same Git repo but different paths (e.g., clusters/us-east-1/, clusters/eu-west-1/).

In both models, the Git repo structure is critical. A common layout:

gitops-repo/
  base/                  # shared HelmRelease / Application definitions
  clusters/
    staging/
      kustomization.yaml # patches for staging
    production-us/
      kustomization.yaml
    production-eu/
      kustomization.yaml

Environment-specific values are layered via Kustomize patches or separate values files, while base chart references remain DRY.