Terraform Isn't Infrastructure as Code. It's Infrastructure Documentation

Introduction

The pitch for Terraform is usually automation: write HCL once, apply it anywhere, stop clicking through the AWS console. That's real value. But after six years of inheriting other teams' Terraform, the kind you open at 2am during an incident and try to reason about under pressure, I've come to believe the automation is almost secondary. The more important property of well-written Terraform is that it tells the next engineer exactly what was built, why it was built that way, and what breaks if it changes.

Terraform best practices are not a style guide. They are an operational risk framework disguised as code formatting preferences. The difference between a Terraform module with descriptive variable names, inline comments explaining non-obvious configuration choices, and a README that answers the first five questions a new engineer will have, and a module without those things, is the difference between infrastructure your whole team can own and infrastructure only its original author understands.

This post is about treating Terraform as documentation first and automation second, and what that shift looks like in practice.

The Readability Gap Nobody Budgets For

There is a category of technical debt that doesn't show up on any burndown chart: the cost of infrastructure that works correctly but cannot be safely modified by anyone who didn't write it. It's not broken. It doesn't produce alerts. It just sits in the codebase as a source of ambient risk: something that works until someone has to touch it.

Terraform is particularly susceptible to this because HCL is expressive enough to encode significant complexity without looking complex from the outside. A module with four input variables and twenty outputs can contain conditional logic, data source lookups, dynamic blocks, and lifecycle rules that interact in ways that only become visible when a plan produces an unexpected diff. The surface looks simple. The behavior isn't.

The failure mode I've seen most consistently: a cloud engineer builds a Terraform module under time pressure. The module works. It gets used across twelve environments. Eighteen months later, the engineer has moved to a different company. A new engineer needs to change one parameter. They read the module, make what looks like a scoped change, run a plan, and see seventy-two resource modifications they didn't expect. They can't tell which of the seventy-two are correct consequences of their change and which are something they broke.

They have two options: spend a day reverse-engineering the module's logic to understand the plan, or ask someone who might know, if that person still works there. Neither is acceptable for a change that should have taken an hour.

The readability gap is the delta between "the time it takes to make this change confidently" and "the time it should take." In unreadable Terraform, that delta is measured in days. In well-documented Terraform, it's measured in minutes.

What makes this particularly expensive is that the gap widens over time. Infrastructure doesn't stay static. Every undocumented behavior is a compounding debt. Every engineer who touches the module without understanding it fully risks adding more undocumented behavior. After two years, you have a module that six people have touched, none of whom fully understood it, and whose current behavior is a palimpsest of individual decisions that no one can fully reconstruct.

AWS Deep Dive: Where Readable Terraform Pays Off Most

Variable Declarations as the Contract

The variables.tf file is the public interface of every Terraform module. It's the first thing an engineer reads when trying to understand what a module does and what they need to provide. A variable declaration that reads like a type signature is wasted documentation real estate. A variable declaration that reads like a specification is the difference between confident usage and uncertain guessing.

Compare these two declarations for the same variable:

Opaque:

variable "retention" {
  type    = number
  default = 7
}

Documented:

variable "log_retention_days" {
  description = <<-EOT
    Number of days to retain CloudWatch Log Group entries before expiration.
    Valid values: 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545,
    731, 1096, 1827, 2192, 2557, 2922, 3288, 3653.
    Note: CloudWatch only accepts specific values; any other number will cause
    a perpetual diff on apply. For compliance environments requiring 1-year
    minimum retention, set to 365 or higher.
    Default of 7 days is appropriate for development environments only.
  EOT
  type        = number
  default     = 7

  validation {
    condition     = contains([1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1096, 1827, 2192, 2557, 2922, 3288, 3653], var.log_retention_days)
    error_message = "log_retention_days must be one of the values accepted by CloudWatch Logs. See variable description for valid values."
  }
}

The second version does four things the first doesn't: it uses a name that carries intent (log_retention_days vs. retention), it documents the AWS-specific constraint that CloudWatch only accepts specific values and that any other number produces a perpetual diff, it flags the compliance implication, and it enforces the constraint at plan time with a validation block that produces a meaningful error message.

The CloudWatch retention days gotcha is real and specific: if you set retention_in_days to a value not in the accepted list (say, 45), Terraform will apply without error the first time because the AWS API rounds or rejects silently depending on the SDK version, but subsequent plans will show a perpetual diff as Terraform sees the actual stored value and compares it to your configuration. Engineers who haven't hit this before will spend time chasing a "drift" that is actually a misconfigured variable.

Comments That Carry Institutional Memory

The conventional wisdom in application code, that well-written code doesn't need comments, breaks down in Terraform for the same reason it breaks down in infrastructure code generally: HCL encodes business decisions, security requirements, and AWS behavioral workarounds that have no natural home in the syntax.

The categories of Terraform that need comments are:

Non-default lifecycle rules with a reason:

resource "aws_db_instance" "primary" {
  # ...

  lifecycle {
    # prevent_destroy is set globally across all environments, not just production.
    # Context: In Q3 2023, a `terraform destroy` targeting the dev workspace
    # accidentally ran against staging due to a misconfigured workspace variable.
    # Requiring explicit lifecycle override before any destroy is the defense.
    # To destroy this resource, temporarily set prevent_destroy = false in a
    # dedicated PR, apply, destroy, then revert.
    prevent_destroy = true

    # ignore_changes on username: RDS requires username set at creation;
    # changes post-creation are ignored by AWS and produce a perpetual diff.
    ignore_changes = [username, password]
  }
}

AWS-specific behavior that isn't obvious from the resource configuration:

resource "aws_iam_role_policy_attachment" "ecs_execution" {
  role       = aws_iam_role.ecs_execution.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"

  # This managed policy grants ecr:GetAuthorizationToken, ecr:BatchCheckLayerAvailability,
  # ecr:GetDownloadUrlForLayer, and ecr:BatchGetImage: the minimum permissions
  # needed for ECS to pull images from ECR. It also includes CloudWatch Logs
  # permissions. If this attachment is removed, tasks will fail to start with
  # "CannotPullContainerError". Do not remove without replacing with an inline
  # policy that covers the same ECR and Logs permissions.
}

Configuration that looks wrong but is intentionally correct:

resource "aws_security_group_rule" "allow_all_outbound" {
  type              = "egress"
  from_port         = 0
  to_port           = 0
  protocol          = "-1"
  cidr_blocks       = ["0.0.0.0/0"]
  security_group_id = aws_security_group.app.id

  # Unrestricted outbound is intentional for this security group.
  # The application makes outbound calls to: Stripe API (443),
  # internal services via VPC (various ports), and S3 via VPC Gateway
  # Endpoint (no actual internet traffic for S3). Restricting outbound
  # would require maintaining a port allowlist that changes with dependencies.
  # The inbound rules provide the actual security boundary.
  # Last reviewed: 2024-11-12 by @platform-team
}

That last comment does something subtle but important: it records a security decision as a deliberate choice with reasoning, not an oversight. Without it, the next security review treats it as a finding. With it, the reviewer can evaluate the reasoning rather than just flagging the rule.

Module READMEs as Onboarding Infrastructure

A Terraform module without a README is a module that requires reading every .tf file to understand. That's acceptable for a twenty-line module. It's not acceptable for a module with eight files, forty variables, and cross-module dependencies.

A module README that functions as onboarding infrastructure answers five questions without the reader needing to open any .tf file:

1. What does this module create?
2. What are the required inputs and what do they control?
3. What are the most important optional inputs and when should they be set?
4. What does this module expose as outputs and which ones are commonly consumed?
5. What's a working example that a new engineer can copy and modify?

The working example is the most frequently skipped and most valuable element. Not a reference to examples/ that may or may not be current: an actual, complete, copy-pasteable usage block in the README itself:

module "payments_api" {
  source = "git::https://github.com/yourorg/terraform-modules.git//ecs-service?ref=v3.1.0"

  service_name    = "payments-api"
  environment     = "prod"
  container_image = "123456789012.dkr.ecr.us-east-1.amazonaws.com/payments-api:v2.4.1"
  container_port  = 8080
  cpu             = 1024
  memory          = 2048
  desired_count   = 3

  # Networking, pull from VPC module outputs
  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_app_subnet_ids

  # Load balancer, pull from ALB module outputs
  alb_target_group_arn = module.alb.target_group_arn["payments-api"]

  # Secrets, ARNs of Secrets Manager secrets the task needs to access
  secret_arns = [
    "arn:aws:secretsmanager:us-east-1:123456789012:secret:payments/stripe-key-AbCdEf",
    "arn:aws:secretsmanager:us-east-1:123456789012:secret:payments/db-password-GhIjKl"
  ]
}

This example communicates the module's dependency model, its integration points with other modules, and the naming conventions the team uses, all without reading a single .tf file.

Tradeoffs & Decision Framework

The objection to documentation-heavy Terraform is real and worth engaging honestly: documentation takes time to write, goes stale if not maintained, and adds overhead to a PR that's already taking longer than planned.

These objections are correct as stated. They're not arguments against documentation, they're arguments for making documentation sustainable rather than heroic.

When documentation overhead is high and return is low:

• One-off Terraform that will be destroyed after a specific event (load test infrastructure, temporary migration resources)
• Internal tooling used only by the engineer who wrote it, with a clear sunset date
• Modules with fewer than four resources and no non-obvious behavior

In these cases, a brief inline comment is enough. Full README and variable descriptions are overhead that isn't proportional to the usage.

When documentation pays back immediately:

• Any module consumed by more than one team
• Any module that will be handed off to a different team or engineer
• Any module that encodes compliance requirements or security decisions that must be preserved through future changes
• Any resource configuration that deviates from the obvious default for non-obvious reasons

The sustainable documentation model: Treat documentation as part of the definition of done for a Terraform module, not a follow-up task. A PR that introduces a new module or meaningfully changes an existing one is not complete without updated variable descriptions and comments on non-obvious behavior. The discipline that makes this sustainable is reviewing for documentation in code review: not as a separate pass, but as part of the standard review checklist.

The signal that documentation is missing: a code review comment that says "what does this do?" or "why is this set this way?" Every question a reviewer has to ask in a PR review is a question an on-call engineer will also have at 2am. If the code can't answer the reviewer's question, it can't answer the on-call engineer's question either.

Lessons From the Field

1. The most expensive Terraform I've ever read was also the most terse. Inherited a 4,000-line main.tf at a logistics company where every variable was a single letter or abbreviation, no comments existed, and the only documentation was a two-line README that said "manages AWS infrastructure." Reverse-engineering the intent of specific resource configurations took weeks. The business cost: two months of slower delivery while the new team rebuilt their mental model of what the code did.

2. ignore_changes without a comment is a future incident waiting to happen. Found an ignore_changes = [engine_version] on an RDS instance with no explanation. The team had added it after a Terraform upgrade produced an unexpected plan to downgrade the engine version due to a state drift issue that had since been resolved. Nobody removed the ignore, and nobody documented why it was there. Two years later, an actual engine version security patch needed to be applied, and Terraform silently ignored it. Document every ignore_changes with the reason it was added and a note about when it's safe to remove.

3. Validation blocks catch misconfiguration before it reaches AWS, and before it wastes an apply cycle. Implemented a Terraform module for a media client that accepted a var.environment input used to determine dozens of internal behaviors. Without validation, engineers had been passing values like "Production" and "DEV" (inconsistent casing) that silently produced wrong configurations because the conditionals used exact string matching. Adding a validation block with a clear error message eliminated a class of misconfiguration that had been causing subtle bugs for months.

4. The working example in the README is what new engineers actually use. Onboarded three engineers to a platform team over six months. In every case, the first thing they did with a new Terraform module was search the codebase for an existing usage of it and copy it. Not read the README. Not read the variables file. Find a copy-paste starting point. The modules with working examples in the README got copied correctly. The ones without, where engineers found existing usages that had evolved away from best practice, propagated the anti-patterns of whatever historical usage they found first.

5. Terraform plan output is not a substitute for readable code, but it tells you when the code isn't readable enough. The diagnostic I use when reviewing Terraform: run a terraform plan on a change that should be scoped to three resources. If the plan shows more than ten resource changes, either the module is doing something non-obvious or the variable I changed has undocumented side effects. Either way, the module needs more documentation before the next engineer inherits it.

Final Thoughts

The infrastructure as code community has spent a decade arguing about which tool to use, Terraform vs. Pulumi vs. CDK vs. CloudFormation, and much less time arguing about what makes any of them worth using in the first place. The tool is the easy part. The hard part is producing infrastructure definitions that are accurate, readable, and maintainable by engineers who weren't there when they were written.

Terraform best practices are converging on this, tfdocs for automatic README generation from variable declarations, tflint for enforcement of naming conventions and deprecated syntax, terraform validate in CI before any plan runs, and Terraform Cloud's policy-as-code features for enforcing module interfaces. The tooling is closing the gap between "it's documented in the code" and "the documentation is enforced and current."

But the judgment about what to document, why a specific configuration choice was made, and what breaks if it's changed: that doesn't come from tooling. It comes from engineers who treat Terraform as the permanent record of infrastructure intent, not just the mechanism that deploys it.

That orientation (writing Terraform for the engineer who inherits it, not just for the next apply), is a core part of how we approach infrastructure work at PulseSoft. If your Terraform has become a codebase only its original authors can safely modify, let's talk.

Key Takeaways

• Terraform's primary value is documentation, not automation. Well-written HCL tells the next engineer what was built, why it was built that way, and what breaks if it changes. Automation that isn't understood is automation that can't be safely modified.
• CloudWatch Log Groups only accept specific retention_in_days values. Any value not in the accepted list will cause a perpetual diff on subsequent plans, Terraform sees the actual stored value diverging from configuration. Document this constraint in the variable description and enforce it with a validation block.
• Every ignore_changes block without a comment is a future incident. Document why each ignore_changes was added, what problem it was solving, and when it's safe to remove. Undocumented ignore rules will survive past their useful life and silently block legitimate changes.
• The working example in a module README is more valuable than the variables documentation. Engineers find existing usages and copy them. If the working example in the README is current and correct, it propagates correct usage. If it doesn't exist, engineers copy whatever historical usage they find first, including anti-patterns.
• Code review is the right time to catch missing documentation, not a separate documentation pass. Every reviewer question that starts with "what does this do?" or "why is this set this way?" is documentation debt that will become an on-call question. Flag it in review before it reaches production.
• validation blocks in variable declarations catch misconfiguration before it reaches AWS. They're also documentation: a clear error_message tells the engineer exactly what went wrong and what the valid options are, without requiring them to read the resource that consumes the variable.
• Terraform readability is an operational risk metric. The time it takes a new engineer to make a change confidently in a Terraform module is a measurable proxy for the module's documentation quality. Track the question rate on your Terraform PRs. Rising question rates mean declining documentation quality.