Lesson 08 Terraform Basics + Remote State
Theme:

Lab 08 — Replace the by-hand VM with Terraform; move state to S3

Lesson 08 · Cloud Infrastructure & DevOps · ~75 min

Goal: by the end of this lab you have destroyed Lesson 07’s by-hand VM and re-created an equivalent one entirely via terraform apply. Terraform state lives in an S3 bucket with a DynamoDB lock. A terraform destroy + terraform apply rhythm replaces the console clicks from L07, and your laptop is no longer the source of truth for the infrastructure.

Prerequisites

Step 1 — Tear down the by-hand VM from L07

In the AWS console, terminate the EC2 instance you launched in Lesson 07. Also release the Elastic IP if you allocated one (EC2 → Elastic IPs → Actions → Release). We are going to re-create equivalent resources via Terraform, so the state file must start from a clean baseline with no orphaned instances or IP addresses.

GCP / Azure / DigitalOcean

If you used a different cloud for L07, terminate or delete the instance and release any static IP you reserved. The rest of this lab is AWS-specific; the principles (init / plan / apply / remote state) are identical across providers.

Step 2 — Read through the Terraform module before running anything

Open cloud-devops/sample-app/terraform/. Four files define the infrastructure:

versions.tf pins Terraform and the AWS provider, and contains the backend "s3" block commented out. The reader uncommtents it in Step 6.

# Main Terraform module for the URL shortener VM.
# After Task 9b lands, the backend "s3" block below is COMMENTED OUT — the
# reader uncomments it during Lab 08 Step 6 after the backend bootstrap apply.
# Reader-facing copy: cloud-devops/lesson08-terraform-basics/lab_guide.html.

terraform {
  required_version = ">= 1.6.0"
  required_providers {
    aws = { source = "hashicorp/aws", version = "~> 5.0" }
  }

  # Uncomment AFTER running the backend bootstrap module:
  # backend "s3" {
  #   bucket         = "url-shortener-tfstate-<your-handle>"
  #   key            = "prod/terraform.tfstate"
  #   region         = "us-east-1"
  #   dynamodb_table = "url-shortener-tflock"
  #   encrypt        = true
  # }
}

provider "aws" {
  region = var.region
}

variables.tf exposes the five input knobs: region, instance_type (defaulted to t3.micro), ssh_key_name, ghcr_owner, domain, and acme_email.

variable "region" {
  type    = string
  default = "us-east-1"
}

variable "instance_type" {
  type    = string
  default = "t3.micro"
}

variable "ssh_key_name" {
  type        = string
  description = "Name of an existing EC2 key pair in this region."
}

variable "ghcr_owner" {
  type        = string
  description = "GitHub username/org owning the GHCR images."
}

variable "domain" {
  type        = string
  description = "Public DNS name to serve. Set to 'localhost' for tls-internal."
}

variable "acme_email" {
  type        = string
  description = "Contact email for Let's Encrypt cert issuance."
}

main.tf declares four resources: an AMI data source, a security group, the EC2 instance, and an Elastic IP.

data "aws_ami" "ubuntu" {
  most_recent = true
  owners      = ["099720109477"] # Canonical
  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd-gp3/ubuntu-noble-24.04-amd64-server-*"]
  }
}

resource "aws_security_group" "web" {
  name        = "url-shortener-web"
  description = "Allow SSH (22), HTTP (80), HTTPS (443) inbound."
  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"] # tighten to your IP in homework Task 2
  }
  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
  ingress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

locals {
  user_data = templatefile("${path.module}/user_data.sh", {
    ghcr_owner = var.ghcr_owner
    domain     = var.domain
    acme_email = var.acme_email
  })
}

resource "aws_instance" "vm" {
  ami                    = data.aws_ami.ubuntu.id
  instance_type          = var.instance_type
  key_name               = var.ssh_key_name
  vpc_security_group_ids = [aws_security_group.web.id]
  user_data              = local.user_data
  tags                   = { Name = "url-shortener" }
}

resource "aws_eip" "vm" {
  instance = aws_instance.vm.id
}

outputs.tf prints the public IP, a ready-to-copy SSH command, and next-steps text after every apply.

output "public_ip" {
  value = aws_eip.vm.public_ip
}

output "ssh_command" {
  value = "ssh ubuntu@${aws_eip.vm.public_ip}"
}

output "next_steps" {
  value = <<-EOT
    1. Point DNS A record for ${var.domain} -> ${aws_eip.vm.public_ip}
    2. SSH in and check `docker compose -f /srv/url-shortener/docker-compose.prod.yml ps`
    3. Smoke test: curl -fsS https://${var.domain}/healthz
  EOT
}

user_data.sh runs on the VM’s first boot. It installs Docker, creates /srv/url-shortener/, writes the compose file and Caddyfile, generates random passwords into .env, then runs docker compose pull && up -d. The heredoc stubs for the compose file and Caddyfile are filled in during Step 4 below.

#!/usr/bin/env bash
# user_data.sh — runs on first boot of the VM via Terraform's `user_data` arg.
# Installs Docker, drops the compose + Caddy files into /srv/url-shortener/,
# writes a starter .env, then brings the stack up.
# Reader-facing copy: cloud-devops/lesson08-terraform-basics/lab_guide.html.
#
# NOTE: The <<'COMPOSE'> and <<'CADDY' heredocs below are STUBS.
# Lab 08 Step 4 walks you through filling them in (Option A: inline the file
# contents; Option B: curl from GitHub raw; Option C: scp post-boot).

set -euxo pipefail
exec > >(tee /var/log/url-shortener-bootstrap.log) 2>&1

apt-get update
apt-get install -y ca-certificates curl
install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg \
  -o /etc/apt/keyrings/docker.asc
chmod a+r /etc/apt/keyrings/docker.asc
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] \
  https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" \
  > /etc/apt/sources.list.d/docker.list
apt-get update
apt-get install -y docker-ce docker-ce-cli containerd.io \
  docker-buildx-plugin docker-compose-plugin

mkdir -p /srv/url-shortener/caddy

# STUB: replace with the full contents of cloud-devops/sample-app/docker-compose.prod.yml
cat > /srv/url-shortener/docker-compose.prod.yml <<'COMPOSE'
# TODO: paste docker-compose.prod.yml contents here (Lab 08 Step 4, Option A)
COMPOSE

# STUB: replace with the full contents of cloud-devops/sample-app/caddy/Caddyfile
cat > /srv/url-shortener/caddy/Caddyfile <<'CADDY'
# TODO: paste Caddyfile contents here (Lab 08 Step 4, Option A)
CADDY

cat > /srv/url-shortener/.env <<EOF
GHCR_OWNER=${ghcr_owner}
DOMAIN=${domain}
ACME_EMAIL=${acme_email}
DB_PASSWORD=$(openssl rand -hex 16)
SECRET_KEY=$(openssl rand -hex 32)
WORKER_INTERVAL=60
EOF
chmod 600 /srv/url-shortener/.env

cd /srv/url-shortener
docker compose -f docker-compose.prod.yml pull
docker compose -f docker-compose.prod.yml up -d

Step 3 — Bootstrap the remote-state backend

The backend S3 bucket and DynamoDB lock table must exist before the main module can use them. The terraform/backend/ directory is a tiny separate Terraform project (local state, no backend block) that creates them.

cd cloud-devops/sample-app/terraform/backend
terraform init
terraform apply -var bucket_name=url-shortener-tfstate-<your-handle>

Replace <your-handle> with a globally-unique suffix (S3 bucket names are unique across all AWS accounts). A common pattern: url-shortener-tfstate-johndoe-2026.

Expected: Terraform creates six resources (S3 bucket, versioning config, encryption config, public-access-block, DynamoDB table) and prints the outputs. Verify:

aws s3 ls | grep tfstate
aws dynamodb list-tables --query 'TableNames'

Expected: the bucket and url-shortener-tflock table appear in the output.

Why a separate module?

The bootstrap module’s own state is local (a terraform.tfstate file in terraform/backend/). It can’t use the S3 backend it’s in the process of creating — that would be circular. The small local state file is intentionally kept on disk as documentation of what was created; the .gitignore in terraform/ only covers the main module’s state files.

Step 4 — Fill in the user_data.sh heredoc stubs

The user_data.sh that ships in sample-app/terraform/ contains stub heredocs for the compose file and Caddyfile so the file stays under 200 lines. Choose one of three options to fill them in before the first apply:

Option A (recommended for this lab) — inline. Copy the full text of cloud-devops/sample-app/docker-compose.prod.yml into the <<'COMPOSE' heredoc, and cloud-devops/sample-app/caddy/Caddyfile into the <<'CADDY' heredoc. The whole user_data.sh ends up around 180 lines; AWS’s user-data limit is 16 KB, so there is room.

Option B — fetch from GitHub raw. Replace the heredocs with curl commands:

curl -fsSL https://raw.githubusercontent.com/<owner>/<repo>/main/cloud-devops/sample-app/docker-compose.prod.yml \
  > /srv/url-shortener/docker-compose.prod.yml
curl -fsSL https://raw.githubusercontent.com/<owner>/<repo>/main/cloud-devops/sample-app/caddy/Caddyfile \
  > /srv/url-shortener/caddy/Caddyfile

Works for public repos; needs a deploy key or personal access token scoped read:repo for private ones.

Option C — scp after boot. Leave user_data.sh minimal (just installs Docker and creates /srv/url-shortener/), then after apply returns the public IP, scp the files from your laptop:

scp cloud-devops/sample-app/docker-compose.prod.yml ubuntu@<public_ip>:/srv/url-shortener/
scp cloud-devops/sample-app/caddy/Caddyfile ubuntu@<public_ip>:/srv/url-shortener/caddy/
ssh ubuntu@<public_ip> "cd /srv/url-shortener && docker compose -f docker-compose.prod.yml pull && docker compose -f docker-compose.prod.yml up -d"

The rest of this lab assumes Option A.

Step 5 — First terraform apply (local state)

Write your variable values to terraform.tfvars (this file is already covered by .gitignore):

cd cloud-devops/sample-app/terraform
cat > terraform.tfvars <<EOF
ssh_key_name = "<your-existing-EC2-key-pair-name>"
ghcr_owner   = "<your-github-username>"
domain       = "urls.example.com"
acme_email   = "you@example.com"
EOF

If you do not have a domain, set domain = "localhost" and Caddy will use its internal CA.

terraform init
terraform plan
terraform apply

Confirm yes when prompted. Expected: roughly 90 seconds. The outputs section prints your new public IP:

Outputs:

public_ip = "203.0.113.42"
ssh_command = "ssh ubuntu@203.0.113.42"
next_steps = <<EOT
  1. Point DNS A record for urls.example.com -> 203.0.113.42
  2. SSH in and check `docker compose -f /srv/url-shortener/docker-compose.prod.yml ps`
  3. Smoke test: curl -fsS https://urls.example.com/healthz
EOT

The VM is up but user_data.sh is still running. Wait 60–90 seconds before testing.

Step 6 — Switch to remote state (terraform init -migrate-state)

Open versions.tf. Uncomment the backend "s3" block and fill in your bucket name:

backend "s3" {
  bucket         = "url-shortener-tfstate-<your-handle>"
  key            = "prod/terraform.tfstate"
  region         = "us-east-1"
  dynamodb_table = "url-shortener-tflock"
  encrypt        = true
}

Migrate the local state to S3:

terraform init -migrate-state

Terraform will ask: “Do you want to copy existing state to the new backend?” Answer yes. The local terraform.tfstate becomes a stub; the real state now lives in S3.

Verify the migration succeeded:

aws s3 ls s3://url-shortener-tfstate-<your-handle>/prod/
# Expected: terraform.tfstate

terraform state list
# Expected: data.aws_ami.ubuntu / aws_security_group.web / aws_instance.vm / aws_eip.vm

Step 7 — Smoke-test

ssh ubuntu@$(terraform output -raw public_ip)

sudo tail -50 /var/log/url-shortener-bootstrap.log
docker compose -f /srv/url-shortener/docker-compose.prod.yml ps

Expected: bootstrap log ends with lines about services starting; ps shows 5 running containers (caddy, api, worker, postgres, redis).

From your laptop:

curl -fsS https://urls.example.com/healthz
# Expected: {"status":"ok"}

curl -fsS -X POST -d '{"url":"https://example.com"}' \
  -H 'Content-Type: application/json' https://urls.example.com/shorten
# Expected: {"short":"https://urls.example.com/<id>"}

If you used domain = "localhost", replace urls.example.com with localhost and accept the locally-signed cert prompt.

Step 8 — Destroy and re-apply (the cattle dance)

terraform destroy

Confirm yes. Expected: roughly 60 seconds. The VM, EIP, and security group are all removed. The S3 state file and DynamoDB table remain (they are managed by the bootstrap module, not the main module).

terraform apply

Confirm yes. Expected: roughly 90 seconds. A brand-new VM appears with a brand-new public IP. Postgres will be empty because the data volume was on the destroyed EBS. This is the point of the cattle model: any single VM can die and be replaced from code in under two minutes. Durable state (the database) belongs outside the VM — Module 4 introduces managed RDS instead of a sidecar Postgres container.

Step 9 — Verify the DynamoDB lock works

Open two terminals. In terminal A, start a slow apply:

# Terminal A
terraform apply

Before confirming, switch to terminal B and try another plan:

# Terminal B
terraform plan

Expected: terminal B blocks immediately with an error similar to:

Error: Error acquiring the state lock
Lock Info:
  ID:        a4f8a2b1-...
  Operation: OperationTypePlan
  ...
  Path:      url-shortener-tfstate-<handle>/prod/terraform.tfstate

The DynamoDB row prevents two simultaneous state mutations. Back in terminal A, confirm no (or yes if you want to proceed). The lock releases and terminal B can proceed normally.

What you should leave with

Troubleshooting

Common issues