Lesson 08 Terraform Basics + Remote State
Theme:

Terraform Basics + Remote State

Lesson 08 · Cloud Infrastructure & DevOps · ~14 min read

1. Why Terraform, why now

Module 1 made the image. Module 2 made the pipeline. Lesson 07 made the deploy command. The remaining manual step is the VM itself. Provisioning hardware by clicking through a web console is the largest unautomated surface left in the workflow, and it has every problem a non-code workflow has: no review, no diff, no version history, no reproducibility across teammates. If a colleague joins the project, you either walk them through the console or write a doc that drifts out of date the moment someone changes a security-group rule.

Terraform converts that surface into code with three properties. It is declarative — you say what you want (a t3.micro running Ubuntu, security group rules for ports 22/80/443, an Elastic IP attached to the instance); Terraform figures out which API calls achieve that. It is idempotent — running apply against an already-correct state is a no-op; it only changes what has drifted from the declaration. And it is provider-agnostic — the same HCL syntax targets AWS, GCP, Azure, Cloudflare, GitHub, even Datadog or PagerDuty. This lesson uses AWS for concreteness; the lab maps each resource to its GCP and Azure equivalent in sidebar notes.

The immediate payoff is narrow but real: instead of 18 console clicks, a teammate types terraform apply and gets the same infrastructure in 90 seconds. The medium-term payoff is that all infrastructure changes go through a pull request, get a terraform plan diff in the PR comment, and are reviewed before apply runs. Module 5 wires that loop into CI automatically.

2. The core loop: init / plan / apply / destroy

Four commands cover roughly 95% of Terraform usage:

terraform init reads the required_providers and backend blocks from your .tf files, downloads provider plugins to .terraform/, and configures the backend connection. It is idempotent: running it twice is safe. Run it once per fresh checkout, and again after editing versions.tf or the backend block.

terraform plan reads the current state, queries the cloud API for actual resource attributes (the “refresh” step), compares those to your .tf declarations, and prints the diff. No changes are applied. The diff is structured: + for resources to create, - for resources to destroy, ~ for in-place attribute changes, and -/+ for resources that must be replaced (destroyed and re-created) because an immutable attribute changed.

terraform apply runs plan again (so the diff you approve is the diff that executes), prompts for confirmation, then applies the changes in dependency order. It writes the updated state to the configured backend when it finishes.

terraform destroy reverses apply. It computes the diff as “everything in state → nothing” and tears down all managed resources. In a lab this is the polite way to avoid cloud charges after the session; in production, destroy with the wrong workspace pointed at the wrong account is the fastest way to have a bad day. The lab introduces workspace discipline in the homework precisely to build the habit of confirming context before running destructive commands.

In CI pipelines you will see apply -auto-approve to skip the interactive prompt. In a hands-on lab, always use the interactive form — the prompt is the last chance to catch a typo in .tf before it touches real infrastructure.

3. Resources, data sources, and variables

Three kinds of blocks account for almost all of the HCL you will write:

resource "TYPE" "NAME" { ... } declares a cloud object for Terraform to manage. TYPE is the provider-specific identifier (aws_instance, aws_security_group, aws_eip). NAME is the local handle you use to reference the resource elsewhere in your configuration (aws_instance.vm, aws_security_group.web). Terraform maintains a one-to-one mapping between each resource block and a real cloud object.

data "TYPE" "NAME" { ... } looks up an existing object without managing it. The URL-shortener module uses data "aws_ami" "ubuntu" to find the latest Ubuntu 24.04 LTS AMI ID by a name pattern — rather than hard-coding the AMI ID (which changes across regions and when Canonical publishes updates). Data sources are read during plan; they do not create, modify, or destroy anything.

variable "NAME" { type = ..., default = ... } declares an input knob. Variables can be set via a terraform.tfvars file, a -var flag on the CLI, or a TF_VAR_name environment variable. The URL-shortener module exposes region, instance_type, ssh_key_name, ghcr_owner, domain, and acme_email as variables so the same .tf files work for any environment without modification.

HCL syntax notes

HashiCorp Configuration Language looks JSON-like but has two gotchas that trip up newcomers. Strings are always double-quoted ("foo", never 'foo'). References between blocks are unquoted: write aws_security_group.web.id, not "aws_security_group.web.id". The quoted form is a string literal; the unquoted form is a resource reference that Terraform resolves at plan time.

4. The VM user_data.sh pattern

The Module 3 sweet spot is that Terraform creates the VM and a bash script ships with the Terraform module to run on first boot. The script installs Docker, pulls the compose file from a known location, writes the .env, and brings the stack up. By the time terraform apply returns the public IP, the URL shortener is already listening.

The user_data argument on aws_instance (or metadata_startup_script on GCP Compute Engine) accepts a string that cloud-init or the metadata daemon executes as root on first boot. Terraform’s templatefile() function threads in HCL variable values before passing the script to the API:

locals {
  user_data = templatefile("${path.module}/user_data.sh", {
    ghcr_owner = var.ghcr_owner
    domain     = var.domain
    acme_email = var.acme_email
  })
}

resource "aws_instance" "vm" {
  user_data = local.user_data
  # ...
}

Inside user_data.sh, ${ghcr_owner} is a Terraform template expression that expands to the variable value. The script is plain bash after that: it runs the same Docker engine install commands from L07 Step 2, creates /srv/url-shortener/, writes the compose and Caddy files, generates random passwords into .env, and runs docker compose pull && up -d.

One practical limit: AWS’s user_data is capped at 16 KB. The full compose file + Caddyfile inlined as heredocs comes to well under 5 KB, so there is room to spare. If the script grows large (e.g., pulling config from Parameter Store), the common pattern is to keep user_data.sh minimal and use a post-provisioner or AWS SSM Session Manager to run the heavyweight setup — but that complexity is deferred to Module 4.

5. State: what it is and why it is load-bearing

When you write resource "aws_instance" "vm" { instance_type = "t3.micro" }, Terraform on first apply will create the instance and record its real cloud ID (i-0abc123) in terraform.tfstate. Future plan operations look in state, find aws_instance.vm = i-0abc123, query the EC2 API for that instance’s current attributes, and compare those to the .tf declaration. Without state, every plan would believe every resource needs to be created from scratch — and every apply would create duplicates.

State is load-bearing: lose it and the real cloud resources become orphans that Terraform cannot see or manage. They will keep running, keep costing money, and Terraform will have no idea they exist unless you manually re-import them with terraform import. The corollary: modifying state by hand is the most dangerous thing you can do in Terraform. The supported escape hatches are terraform state rm (stop managing a resource without destroying it) and terraform state mv (rename a resource in state after a .tf refactor).

Remote state moves terraform.tfstate to S3 (or GCS, Azure Blob, or a Terraform Cloud workspace) so teammates share it. Two reasons this matters in practice:

  1. Reproducibility. Anyone with AWS credentials and the backend config can run apply against the same infrastructure. The state is the ground truth, not a file on your laptop.
  2. Locking. DynamoDB (for the S3 backend) or Cloud Storage’s native locking (for GCS) prevents two engineers from running apply simultaneously. Without locking, concurrent applies write conflicting state and the infrastructure ends up in an unknown state.

The cost is operational: the backend resources (S3 bucket + DynamoDB table) must exist before terraform init on the main module will succeed. The lab solves that with a separate bootstrap module.

6. The bootstrap module pattern

Inside sample-app/terraform/backend/ lives a tiny separate Terraform project: versions.tf declares the provider and the three input variables; main.tf creates the S3 bucket (with versioning, server-side encryption, and public-access-block) and the DynamoDB lock table; there is no backend block, so it uses local state by design. Running terraform apply there first produces the bucket and table. The main sample-app/terraform/ module then declares a backend "s3" block pointing at those resources.

The local state file at terraform/backend/terraform.tfstate records the bucket and table ARNs. Some teams commit this file intentionally — the resources it describes will never change after creation, so the file is effectively documentation of “here is where state lives.” Other teams run the bootstrap manually once and never touch it again. The lab notes both options; the .gitignore under terraform/ excludes *.tfstate* by default but the backend subdirectory has an explanatory comment about the exception.

Common pitfall: committing the main module’s state to git

After terraform init -migrate-state, the local terraform.tfstate in the main module directory becomes a stub or an empty file. Delete it or leave it — the real state is in S3. What you must not do is commit a live terraform.tfstate to a public repo. Terraform state files contain resource IDs, IP addresses, RDS endpoint hostnames, and sometimes sensitive output values. Committing state files to git is one of the top-10 credential leak vectors in this whole course. Always have *.tfstate* in your .gitignore for the main module.

7. Common pitfalls

Four things that go wrong

  1. Lost state. Committed .tfstate to a public repo — or deleted the file while it was the only copy. Resource IDs and sensitive output values leaked (or infrastructure became unmanageable). Mitigation: .gitignore *.tfstate* for the main module; use remote state from day one in any shared work.
  2. Concurrent applies without a lock. Two engineers run apply against the same S3 backend but the DynamoDB lock table is missing or misconfigured. State becomes inconsistent with the real cloud. Always configure dynamodb_table on every S3 backend block; never omit it in the name of simplicity.
  3. destroy against the wrong workspace or AWS profile. terraform destroy reads the current state. If your shell has the wrong AWS_PROFILE set, you might destroy prod while thinking you’re in dev. Use named workspaces, tag every resource with terraform.workspace, and put the active profile in your shell prompt so it is always visible before you type a destructive command.
  4. Forgetting terraform init after a backend change. Editing the backend "s3" block requires terraform init -reconfigure (or -migrate-state) before plan will work. Without re-init, the CLI continues using the cached backend config from the previous init and the new block is silently ignored.

References