Lesson 05 Security Scanning & Image Signing
Theme:

Security Scanning & Image Signing

Lesson 05 · Cloud Infrastructure & DevOps · ~15 min read

1. The supply-chain threat model in plain terms

A green CI pipeline is a confidence signal about your code. It says nothing about the packages that ended up inside your container image. Supply-chain attacks exploit this gap: instead of compromising your code directly, an attacker compromises something your code depends on.

Four touch points define the threat surface for a containerised application:

The conclusion is not that these attacks are common; it is that a green test suite cannot rule them out. Scanning and signing are the complementary controls that cover what tests cannot.

2. Vulnerability scanning: Trivy

Trivy (by Aqua Security, open-source) is a comprehensive vulnerability scanner that inspects a container image and reports known CVEs across two broad categories:

CVEs are classified by severity: UNKNOWN, LOW, MEDIUM, HIGH, CRITICAL. For CI gates, a common policy is to fail only on HIGH and CRITICAL — this catches severe issues without drowning the team in noise from LOW informational findings.

The ignore-unfixed: true flag is important in practice. Many CVEs — particularly in OS packages — have been published but have no available upstream fix. Failing the build on unfixed CVEs creates a situation where the team cannot merge at all, even after applying every patch that exists. Setting ignore-unfixed: true limits failures to CVEs that can be remediated by upgrading a package. You can still see the unfixed CVEs in the log; they just don't gate the build.

A brief mention of the alternative: Grype (Anchore, also open-source) is a strong competitor. It pairs naturally with Syft (see next section) for SBOM-first workflows. The capability gap between the two is small; the practical advice is to pick one, establish a policy, and stay consistent — mixing scanners produces different severity ratings for the same CVE, which confuses triage.

3. SBOMs and why they matter

A Software Bill of Materials (SBOM) is a machine-readable list of every software component in a deliverable — the full transitive dependency graph, not just the top-level packages in requirements.txt. Two standard formats dominate:

Tools: Syft (Anchore) generates SBOMs from live images, filesystems, or lock files. Trivy can also emit SBOMs using the --format spdx-json flag.

Two reasons SBOMs matter in practice:

Regulatory. US Executive Order 14028 (May 2021) requires federal software suppliers to provide SBOMs as part of software procurement. The NTIA published minimum-elements guidance defining what must be included. Even for non-federal work, this framework is increasingly referenced in enterprise procurement contracts.

Operational. When a high-profile CVE drops, the first question is: "which of our images ships the affected package?" Without SBOMs, answering this requires re-scanning every running image — a slow process. With SBOMs stored as build artifacts or registry attestations, a single query across SBOMs answers the question in seconds. Log4Shell (CVE-2021-44228, December 2021) was the canonical demonstration: teams with SBOMs identified their exposure in minutes; teams without spent days.

SBOMs can be stored in two ways. As a CI build artifact (uploaded via actions/upload-artifact), they are accessible from the Actions run page. As a registry attestation (signed with cosign, stored in the OCI registry alongside the image), they are verifiable and tamper-proof and travel with the image wherever it is copied.

4. Image signing with cosign

cosign (part of the Sigstore project, under Linux Foundation) solves a specific problem: between docker build and docker run, an image can be tampered with or substituted. A registry credential leak, a misconfigured registry policy, or a compromised CD pipeline could all result in a different image being deployed than was built. Signatures provide a cryptographic attestation that a specific image digest was produced by a specific, expected process.

Static key signing. You generate a keypair with cosign generate-key-pair. The private key goes into a CI secret; the public key is distributed to verifiers. This works, but it introduces a long-lived secret: the private key. If the secret is ever leaked or rotated without updating all verifiers, the trust model breaks.

Keyless signing (OIDC). This is the canonical pattern for GitHub Actions in 2024+. The flow:

  1. GitHub Actions mints a short-lived OIDC token for the workflow run. The token encodes the repository, workflow path, and commit SHA.
  2. cosign sends this token to Fulcio, Sigstore's certificate authority. Fulcio verifies the token with GitHub's OIDC endpoint and issues a short-lived code-signing certificate tied to the workflow identity (e.g., https://github.com/org/repo/.github/workflows/security.yml@refs/heads/main).
  3. cosign signs the image digest with the ephemeral private key, then discards it. The signature and the certificate are stored in the OCI registry (as a separate tag on the same image) and are also logged to Rekor, Sigstore's transparency log.
  4. At verify time, a verifier runs cosign verify with --certificate-identity-regexp and --certificate-oidc-issuer. cosign checks: (a) the signature is valid, (b) the cert chain leads to Fulcio's root CA, (c) the cert identity matches the expected workflow. All three must pass.

The critical verification step is pinning the certificate identity. Verifying that the signature is cryptographically valid (step a) is not enough: an attacker who controls any GitHub Actions workflow in any repository could sign an image with their own keyless identity. The --certificate-identity-regexp flag pins the verification to your specific repository and workflow path, closing this gap.

The workflow must have permissions: id-token: write for the OIDC token exchange to work. This permission should be scoped to the job that performs signing, not the entire workflow, to follow the principle of least privilege.

5. Branch protection

GitHub's branch protection rules transform status checks from decorative indicators into merge gates. Without branch protection, a developer can merge a PR regardless of whether the CI checks pass. With branch protection enabled on main, GitHub physically prevents the merge button from being activated until all required checks are green.

The relevant settings are: Require status checks to pass before merging (where you list the specific check names — lint, test (3.11), test (3.12), build (api), build (worker) — adding the Trivy scan jobs promotes them from informational to required), and Require a pull request before merging (at least one approval, preventing self-merge).

Optional but valuable: Require linear history (no merge commits, forcing rebase or squash) and Require signed commits (commits must carry a GPG or SSH signature). Both increase auditability.

The net effect is that a branch protection configuration is a policy document checked into GitHub's infrastructure. It specifies: what proof of correctness is required before code can reach production. Scanning and signing make the "green check" mean more than "tests pass" — it means "tests pass, no unfixed HIGH/CRITICAL CVEs found, SBOM generated."

6. Required vs informational checks

When you first add a new check — a linter, a scanner, a coverage gate — it often produces noisy results. Running it as an informational check (it runs but does not block) lets the team observe the signal without being blocked by it. After a few weeks, once the team has calibrated thresholds and fixed the initial findings, the check can be promoted to required.

The promotion process matters: adding a check as required on day one means one flaky or noisy result blocks every PR. Add required checks incrementally. The same principle applies to Trivy: start with severity: HIGH,CRITICAL and ignore-unfixed: true before considering whether to tighten the policy later.

GitHub shows the distinction clearly: checks listed under "Required" in the branch protection rule block the merge button; all other checks that run appear in the PR status list as informational.

7. Common pitfalls

Common pitfalls

  1. Hard-failing on every CVE. Without ignore-unfixed: true plus a sensible severity floor (HIGH,CRITICAL), the build fails on dozens of informational findings with no available fix. The team learns to ignore the red, which defeats the purpose. Tune severity and fix-availability filtering from the start, not after the team is already numb to the noise.
  2. Storing the cosign private key in a CI secret when you don’t need to. Keyless OIDC signing requires only permissions: id-token: write on the workflow job. There is no secret to rotate, no key to leak. If you find yourself generating a keypair and putting the private key in a secret, stop and use keyless instead.
  3. Forgetting --yes in CI. By default, cosign sign asks for interactive confirmation before writing to the registry. In a non-TTY runner (every GitHub Actions runner), this hangs forever, eventually timing out. Always pass --yes in automated contexts.

References