Security & Best Practices

Lesson 20 — Hardening Helm charts for production with defence-in-depth

1. Pod Security Standards

Kubernetes defines three Pod Security Standards (PSS) that set progressively stricter policies for what pods are allowed to do. These standards replaced the deprecated PodSecurityPolicy and are enforced via the built-in Pod Security Admission controller.

Level	Description	Use Case
Privileged	Unrestricted; no security restrictions applied	System-level daemons (CNI, storage drivers, log collectors)
Baseline	Blocks known privilege escalation vectors while remaining broadly compatible	Legacy workloads being migrated to stricter standards
Restricted	Heavily restricted; follows current pod-hardening best practices	All application workloads in production

You apply these standards by labelling a namespace:

apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

The enforce mode rejects non-compliant pods. The audit mode logs violations without blocking, and warn shows warnings to the user. A common migration strategy is to start with warn, fix all violations, then switch to enforce.

2. securityContext Templating

Kubernetes provides two scopes for security context: pod-level and container-level. Pod-level settings apply to all containers in the pod, while container-level settings override or extend them for individual containers.

Pod-Level securityContext

Pod-level settings govern shared concerns: the user/group identity, filesystem group ownership, and the seccomp profile.

spec:
  template:
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        runAsGroup: 1000
        fsGroup: 1000
        seccompProfile:
          type: RuntimeDefault

runAsNonRoot: true is the single most important setting. It causes the kubelet to reject any container whose image runs as UID 0. The fsGroup ensures mounted volumes are group-owned by the specified GID, which matters when the container writes to persistent volumes.

Container-Level securityContext

Container-level settings control the individual container's kernel capabilities and filesystem access:

containers:
  - name: app
    securityContext:
      readOnlyRootFilesystem: true
      allowPrivilegeEscalation: false
      capabilities:
        drop: ["ALL"]

readOnlyRootFilesystem: true prevents the container from writing to its root filesystem, forcing it to use explicitly mounted volumes for any writes. This limits the damage an attacker can do if they gain code execution. allowPrivilegeEscalation: false prevents a process from gaining more privileges than its parent, blocking exploits that use setuid binaries. Dropping all capabilities removes every Linux kernel capability (mount, network raw, etc.) unless you explicitly add specific ones back.

Helm Pattern

Make securityContext values configurable but default to the most restrictive settings. Consumers who need relaxed settings (e.g., an init container that needs NET_BIND_SERVICE) can override in their values file.

Templating Both Levels

# templates/deployment.yaml
spec:
  template:
    spec:
      {{- with .Values.podSecurityContext }}
      securityContext:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      containers:
        - name: {{ .Chart.Name }}
          {{- with .Values.securityContext }}
          securityContext:
            {{- toYaml . | nindent 12 }}
          {{- end }}

# values.yaml
podSecurityContext:
  runAsNonRoot: true
  fsGroup: 1000
  seccompProfile:
    type: RuntimeDefault
securityContext:
  readOnlyRootFilesystem: true
  allowPrivilegeEscalation: false
  capabilities:
    drop: ["ALL"]

3. Image Pull Secrets and Private Registries

Production clusters pull images from private registries. Kubernetes needs credentials to authenticate. There are two common approaches:

Per-pod imagePullSecrets: Each pod spec references one or more Secrets of type kubernetes.io/dockerconfigjson. This is explicit but repetitive if many Deployments use the same registry.

ServiceAccount-level: Attach the pull secret to the ServiceAccount. Every pod using that ServiceAccount inherits the credentials automatically. This is cleaner for charts that create a dedicated ServiceAccount.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: {{ include "myapp.fullname" . }}
imagePullSecrets:
  {{- toYaml .Values.imagePullSecrets | nindent 2 }}
automountServiceAccountToken: false

Note automountServiceAccountToken: false. Unless the application needs to call the Kubernetes API, disable the automatic mounting of the ServiceAccount token. An exposed token gives an attacker API access scoped to whatever RBAC permissions the ServiceAccount holds.

4. Least-Privilege RBAC

Every application should run under its own ServiceAccount with the minimum permissions required. The default ServiceAccount in each namespace is shared by all pods that do not specify one, making it a poor choice for production workloads.

When defining RBAC rules, follow these principles:

Use Role (namespaced) over ClusterRole unless the app genuinely needs cross-namespace access
Enumerate specific resources and verbs — never use wildcards ("*")
Use resourceNames to scope to specific objects when the app only reads its own ConfigMap or Secret
Separate read and write roles if different components need different levels of access

rules:
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get", "watch"]
    resourceNames: ["{{ include "myapp.fullname" . }}-config"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list"]

Warning

A Role with verbs: ["*"] on secrets lets an attacker read every secret in the namespace. Even list on secrets is dangerous because it returns secret data in the response. Grant get with explicit resourceNames instead.

5. Supply Chain Security

Supply chain attacks target the artefacts you deploy rather than your running application. Helm and container images both need verification.

Chart signing and verification: Helm supports GPG-based chart signing. When you run helm package --sign, Helm creates a .prov provenance file alongside the .tgz. Consumers verify integrity with helm install --verify. In CI/CD, always verify charts pulled from external repositories.

Image digest pinning: Container image tags are mutable. The tag :v1.2.3 can be re-pushed to point at a different image. Digests (@sha256:...) are immutable and guarantee you deploy exactly the image you tested. Structure your values to support both tag and digest, preferring digest when set:

image: "{{ .Values.image.repository }}{{- if .Values.image.digest }}@{{ .Values.image.digest }}{{- else }}:{{ .Values.image.tag | default .Chart.AppVersion }}{{- end }}"

The :latest tag is the worst offender: it changes with every push and provides zero reproducibility. Ban it from all environments beyond local development.

6. Resource Limits and LimitRange

Containers without resource limits can consume unbounded CPU and memory, starving other workloads on the same node. Always set both requests (what the scheduler guarantees) and limits (the hard ceiling).

As a safety net, deploy a LimitRange in every namespace. It automatically injects default requests and limits into pods that omit them:

apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
spec:
  limits:
    - default:
        cpu: "500m"
        memory: "256Mi"
      defaultRequest:
        cpu: "100m"
        memory: "128Mi"
      type: Container

A LimitRange does not replace setting resources in your chart — it catches the cases where someone forgets.

7. Network Security

By default, every pod in a Kubernetes cluster can communicate with every other pod. NetworkPolicies restrict this to only the traffic your application actually needs.

Start with a default-deny policy for both ingress and egress, then add specific allow rules:

policyTypes: ["Ingress", "Egress"]
ingress:
  - from:
      - podSelector:
          matchLabels:
            app: frontend
    ports:
      - port: 8080
egress:
  - to:
      - podSelector:
          matchLabels:
            app: database
    ports:
      - port: 5432
  - to:                           # Always allow DNS
      - namespaceSelector: {}
    ports:
      - port: 53
        protocol: UDP

Remember to always allow DNS egress (port 53 UDP) or pods will fail to resolve any service names. This is the most common mistake when implementing egress policies.

8. Secrets Hygiene

Secrets must never appear in values.yaml files committed to Git. Even base64-encoded values in Kubernetes Secret manifests are trivially decoded. Instead:

Pass secrets at install time with --set flags sourced from CI/CD secret stores
Use External Secrets Operator to sync secrets from AWS Secrets Manager, HashiCorp Vault, or Azure Key Vault
Use Sealed Secrets for encrypted secrets that are safe to commit to Git
Mount secrets as files rather than environment variables — environment variables appear in process listings and crash dumps

9. Policy Enforcement Tools

Manual review does not scale. Integrate automated security scanning into your CI/CD pipeline:

Tool	Type	How It Works
kube-score	Static analysis	Scores rendered manifests against best practices; runs in CI with `helm template \| kube-score score -`
Polaris	Audit + admission	CLI audit mode for CI plus a webhook that blocks non-compliant deployments at admission time
OPA/Gatekeeper	Policy engine	Define custom policies as Rego code; enforced as a Kubernetes admission controller via ConstraintTemplates

A good strategy is to run kube-score and Polaris in CI (fast feedback) and deploy Gatekeeper in the cluster as a last line of defence (enforcement at admission).

Summary

Security in Helm is defence in depth. No single control is sufficient. Combine restricted security contexts, least-privilege RBAC, image digest pinning, network policies, external secret management, and automated policy enforcement to build a chart that is hardened by default and auditable by design.