The Drift You Didn't Cause, Detecting Infrastructure Drift from AWS-Managed Changes

Introduction

You haven't touched the Terraform in three weeks. Your pipeline is green. No manual console changes, no emergency CLI commands, no rogue engineers with direct access. And yet terraform plan shows changes.

AWS-managed infrastructure drift is the category of configuration delta that accumulates without any human action. Auto Scaling adjusts desired capacity in response to load. RDS promotes a read replica during a failover and updates the cluster endpoint. ECS replaces a failed task with a new one running a slightly different host configuration. AWS performs maintenance on an RDS instance and modifies the latest_restorable_time attribute. Systems Manager updates the version of a managed node's SSM agent. Each of these events changes the actual state of a resource in ways that may or may not match what Terraform has in its state file.

This is the drift detection problem most blog posts ignore: not the security group rule a developer added through the console, but the continuous background mutation that AWS performs on resources you manage through IaC. Understanding which AWS-managed changes are expected, which represent genuine drift, and how to configure your IaC to distinguish between the two is one of the more nuanced parts of operating infrastructure at scale.

How AWS Mutates Resources Without You

The mental model that gets teams into trouble is treating AWS resources as static once deployed. Deploy an RDS instance, run terraform apply, done: the resource now matches your configuration and will stay that way until you change it. That model is accurate for some resources and wrong for others.

AWS continuously modifies resources across several categories:

Autoscaling systems, EC2 Auto Scaling Groups track desired capacity, minimum, and maximum as the authoritative numbers Terraform deploys. But Auto Scaling adjusts the running count in response to scaling policies, scheduled actions, and predictive scaling. If you deploy a group with desired_capacity = 3 and a load spike triggers a scale-out to 7, Terraform state still says 3. The next terraform plan will show a change: desired_capacity: 7 → 3. If you apply that plan, you've just scaled your fleet back down mid-load-event.

Database clusters, Aurora clusters track a writer instance and zero or more reader instances. During a failover, AWS promotes a reader to writer. The cluster endpoint remains stable, but the instance that it points to has changed. Terraform's state records the original writer. Post-failover, the resource attributes in AWS, specifically which instance is the writer, no longer match state, producing drift that exists entirely in AWS's internal tracking rather than in any configuration field your Terraform controls.

Managed service attributes, RDS instances have a latest_restorable_time attribute that AWS updates continuously as backup snapshots are taken. Terraform reads this attribute on refresh, but it's a computed value that changes constantly. If Terraform doesn't have ignore_changes configured for this attribute, every plan will show it as changed.

Certificate rotation and key material, ACM certificates managed by AWS have renewal dates that AWS updates automatically. Secrets Manager secrets can have last_changed_date attributes that update on rotation. Lambda function configurations can show last_modified timestamps that change on any AWS-side update.

The failure mode this creates is plan noise: a terraform plan that always shows changes, even when nothing in the configuration has changed, trains engineers to ignore the plan output or to apply reflexively without reviewing it. Once engineers stop trusting the plan, the plan loses its value as a safety check, and the genuinely important drift (the security group rule that shouldn't exist, the encryption that was disabled) gets lost in the noise of the expected drift.

The second failure mode is unintended reversions: applying a plan that contains AWS-managed changes alongside intentional changes will revert the AWS-managed changes back to the Terraform-declared values. Scaling down a fleet that AWS scaled up during peak load, or re-applying a pre-failover database configuration to a post-failover cluster, can cause incidents without any malicious intent.

AWS Deep Dive: Taming AWS-Managed Drift in Practice

ignore_changes as a Precision Tool, Not a Blunt Instrument

The standard Terraform mechanism for handling attributes that AWS manages autonomously is lifecycle { ignore_changes = [...] }. This tells Terraform to read the attribute from state but not treat differences between state and configuration as changes to apply.

The critical discipline: ignore_changes should be as specific as possible. A block that reads ignore_changes = all silently ignores every change to the resource, including intentional configuration changes you make in the future. This is how resources drift out of IaC management entirely: someone adds ignore_changes = all to stop the plan noise, and the resource becomes effectively unmanaged.

Correct, specific attribute ignores with comments:

resource "aws_autoscaling_group" "app" {
  name                = "${var.service_name}-${var.environment}"
  min_size            = var.min_size
  max_size            = var.max_size
  desired_capacity    = var.desired_capacity

  # ... other configuration ...

  lifecycle {
    ignore_changes = [
      # desired_capacity is managed by Auto Scaling policies at runtime.
      # Applying Terraform should not override capacity decisions made by
      # scaling events. Min/max are still enforced by Terraform.
      desired_capacity,

      # tag values on instances are set by EC2 launch; ignoring to prevent
      # perpetual diff from instance metadata tags injected by ASG.
      tag
    ]
  }
}

Incorrect, silencing all drift with no specificity:

lifecycle {
  ignore_changes = all  # DO NOT USE: silently ignores all future configuration changes
}

The non-obvious behavior of ignore_changes: it ignores changes in both directions. If you add an attribute to ignore_changes and then change its value in the configuration, Terraform will not apply the change. Engineers who add ignore_changes = [desired_capacity] and then try to change the desired capacity through Terraform (rather than through a scaling policy) will find their change silently ignored. Document this in a comment on the lifecycle block.

For RDS instances, the attributes that generate the most AWS-managed drift noise are latest_restorable_time, endpoint, reader_endpoint (for Aurora), and engine_version (which AWS updates during minor version auto-upgrade windows if auto_minor_version_upgrade = true):

resource "aws_db_instance" "primary" {
  identifier     = "${var.service_name}-${var.environment}"
  engine         = "postgres"
  engine_version = "15.4"

  # auto_minor_version_upgrade = true means AWS will upgrade minor versions
  # automatically. Setting ignore_changes on engine_version prevents Terraform
  # from detecting the AWS-applied upgrade as drift and attempting to revert it.
  # If you need to control minor version upgrades explicitly, set
  # auto_minor_version_upgrade = false and manage engine_version in Terraform.
  auto_minor_version_upgrade = true

  lifecycle {
    ignore_changes = [
      engine_version,          # Managed by AWS when auto_minor_version_upgrade = true
      latest_restorable_time,  # Computed by AWS; updates continuously
    ]
  }
}

Distinguishing Expected Drift from Unexpected Drift with AWS Config

terraform plan detects drift by comparing Terraform state to actual resource configuration. But it only runs when someone initiates a plan, it's a point-in-time check, not continuous monitoring. AWS Config provides continuous monitoring of resource configuration changes and can be used to distinguish between expected AWS-managed changes and unexpected human-made changes.

The pattern that makes this distinction operational: tag every Terraform-managed resource with a ManagedBy = "terraform" tag, and configure a Config rule that alerts when a resource tagged with ManagedBy = "terraform" has a configuration change that wasn't preceded by a CloudTrail event from your Terraform deployment role.

This doesn't catch all drift, Auto Scaling events don't change resource tags, and many AWS-managed attribute updates don't generate CloudTrail management events. But it catches the highest-risk category: direct human modifications to IaC-managed resources through the console or CLI.

For Auto Scaling specifically, AWS provides a CloudWatch metric (GroupDesiredCapacity, GroupInServiceInstances) that gives you real-time visibility into the current state of the group without needing a terraform plan. Setting a CloudWatch alarm on GroupInServiceInstances dropping below a threshold that your application requires for healthy operation catches the scenario where the scaling system has reduced capacity below safe levels, regardless of what Terraform state says:

resource "aws_cloudwatch_metric_alarm" "asg_min_healthy" {
  alarm_name          = "${var.service_name}-${var.environment}-min-instances"
  comparison_operator = "LessThanThreshold"
  evaluation_periods  = 2
  metric_name         = "GroupInServiceInstances"
  namespace           = "AWS/AutoScaling"
  period              = 60
  statistic           = "Minimum"
  threshold           = var.min_healthy_instances
  alarm_description   = "ASG in-service instances dropped below minimum healthy threshold"

  dimensions = {
    AutoScalingGroupName = aws_autoscaling_group.app.name
  }

  alarm_actions = [var.alert_sns_arn]
}

This alarm fires on the operational condition (too few healthy instances) rather than the configuration drift (Terraform state vs. actual desired capacity). It's the right layer for catching Auto Scaling drift that matters for service health, as opposed to drift that's expected and benign.

Scheduled Plans as a Continuous Drift Signal

The highest-signal drift detection mechanism for Terraform-managed infrastructure is a scheduled terraform plan that runs on a cadence, daily or weekly, and alerts when it produces a non-empty output.

The key is interpreting the output correctly. A scheduled plan that shows changes falls into one of three categories:

1. Expected AWS-managed drift, Auto Scaling desired capacity, RDS latest_restorable_time, attributes governed by ignore_changes that somehow appeared in the plan anyway. These should produce no output if ignore_changes is correctly configured. If they appear, it signals a missing or incorrect ignore_changes block.

2. Authorized but un-codified drift: A change was made intentionally (emergency security group update, manual scaling event) but not yet codified in Terraform. This is the primary signal the scheduled plan is designed to surface.

3. Unauthorized drift: A change was made to the infrastructure without authorization and without IaC. This requires immediate investigation and triage.

The workflow that makes this actionable:

#!/bin/bash
# scheduled-drift-check.sh, runs in CI on a schedule

set -e

terraform init -backend=true
terraform plan -detailed-exitcode -out=drift-check.tfplan 2>&1

EXIT_CODE=$?

if [ $EXIT_CODE -eq 0 ]; then
  echo "No drift detected, infrastructure matches Terraform configuration"
  exit 0
elif [ $EXIT_CODE -eq 1 ]; then
  echo "Terraform plan error, investigate pipeline configuration"
  exit 1
elif [ $EXIT_CODE -eq 2 ]; then
  # Changes detected, generate readable output and alert
  terraform show -no-color drift-check.tfplan > drift-report.txt

  # Count resource changes by type
  CREATES=$(grep -c "will be created" drift-report.txt || true)
  UPDATES=$(grep -c "will be updated" drift-report.txt || true)
  DESTROYS=$(grep -c "will be destroyed" drift-report.txt || true)
  REPLACEMENTS=$(grep -c "must be replaced" drift-report.txt || true)

  echo "Drift detected: +${CREATES} ~${UPDATES} -${DESTROYS} ±${REPLACEMENTS}"
  echo "Full report in drift-report.txt"

  # Alert, post to Slack, open GitHub issue, page on-call if replacements > 0
  if [ "$REPLACEMENTS" -gt 0 ]; then
    echo "CRITICAL: Drift includes resource replacements, immediate review required"
    # Trigger high-severity alert
  fi

  exit 2
fi

The severity routing at the end is the key addition most drift check implementations skip: a plan showing a tag update and a plan showing a resource replacement both have exit code 2, but they warrant very different responses. Route alerts based on the content of the drift, not just its presence.

Tradeoffs & Decision Framework

Drift detection rigor exists on a spectrum, and the right level depends on the team's operational maturity and the risk profile of the infrastructure.

Minimal drift detection (small team, low change frequency, non-regulated): ignore_changes for the five or six most common AWS-managed attributes (Auto Scaling desired capacity, RDS latest_restorable_time, ACM certificate renewal metadata). Scheduled weekly terraform plan with a Slack alert on non-empty output. No severity routing: everything that surfaces requires human triage.

This level is sufficient for most early-stage teams. The overhead of more sophisticated detection is disproportionate to the risk at small scale.

Structured drift detection (multiple teams, regular deployments, compliance requirements): Formalized ignore_changes policy with mandatory comments documenting the reason for each ignore. Daily scheduled plans with severity routing, replacements page on-call, updates open a GitHub issue, creates require a review before the next production apply. AWS Config aggregation to catch non-Terraform configuration changes to IaC-managed resources.

Continuous drift monitoring (regulated environment, high availability requirements): Near-real-time drift detection via EventBridge rules on CloudTrail management events for high-risk resource types (security groups, IAM policies, network ACLs), with alerting on any change not originating from the Terraform deployment role. Scheduled plans at daily cadence for all resource types, with automated categorization of detected drift into expected/authorized/unauthorized buckets. Quarterly drift remediation sprints to codify accumulated authorized-but-un-codified changes.

The honest tradeoff with aggressive drift detection: False positives erode trust. If the drift detection system fires alerts on expected AWS-managed changes because ignore_changes isn't correctly configured, engineers will start ignoring drift alerts, precisely the outcome you're trying to prevent. Calibrate the detection before operationalizing the alerts. Run the scheduled plan for two weeks and triage every alert before wiring it to an on-call page. The signal should be clean before it's amplified.

Lessons From the Field

1. ignore_changes = all is always the wrong answer, and it's always added under time pressure. At a growth-stage SaaS company, I found ignore_changes = all on six resources, all added in the same week six months prior. That week coincided with a major deployment where the plan output had been full of noise from AWS-managed attribute changes that hadn't been properly handled. Someone silenced the noise by silencing everything. Three of those resources had subsequently been changed in the console and the changes had never been codified in Terraform. The resources were effectively unmanaged. Removing ignore_changes = all from any resource requires auditing the current AWS configuration against what Terraform would apply, plan before removing, never remove blindly.

2. Auto Scaling drift during peak events is the most operationally dangerous drift to revert. A retail client ran terraform apply during their Black Friday sale to deploy an unrelated infrastructure change. The apply plan included desired_capacity: 14 → 3, their Auto Scaling group had scaled out to 14 instances to handle the load, but Terraform state said 3 from the pre-peak baseline. The engineer running the apply had reviewed the plan, seen the capacity change, and assumed it was part of the deployment. It wasn't. The apply reduced the fleet from 14 to 3 during peak traffic. Recovery took eleven minutes. Adding ignore_changes = [desired_capacity] to every Auto Scaling Group in the codebase was the same-day remediation.

3. RDS minor version auto-upgrade drift is silent until you try to control the version. Set up a new RDS instance for a fintech client with auto_minor_version_upgrade = true and engine_version = "15.3". Three months later, AWS auto-upgraded the instance to 15.4 during a maintenance window. The next terraform plan showed engine_version: "15.4" → "15.3". The engineer running the plan didn't recognize the drift, applied it, and triggered a downgrade of the RDS engine version, which AWS immediately rejected as an invalid operation (you can't downgrade a minor version). The apply failed with a cryptic error. Adding ignore_changes = [engine_version] when auto_minor_version_upgrade = true is non-negotiable; I now add it as a lint rule via tflint custom rule.

4. Scheduled drift checks surface the authorized-but-un-codified changes that everyone forgets about. Set up a weekly scheduled Terraform plan for a logistics client. First run: forty-three resource differences. Three were from unconfigured ignore_changes (expected AWS-managed attributes). Thirty-eight were authorized changes, emergency console modifications over the previous six months that had never been codified. Two were unauthorized. Without the scheduled plan, we would never have known those thirty-eight changes existed. The codification effort took two sprints but produced a Terraform state that actually reflected production for the first time in over a year.

5. The severity of drift is determined by the resource type, not the change count. A scheduled plan showing one changed attribute on a production IAM policy is a higher-severity finding than a plan showing twenty changed attributes on Auto Scaling desired capacities. Alert routing based on change count produces alert fatigue and missed incidents. Route based on resource type and change type: IAM, security groups, and encryption configuration trigger immediate review regardless of count; Auto Scaling and managed-attribute drift triggers a weekly review queue.

Final Thoughts

The infrastructure drift detection conversation is maturing alongside the IaC tooling ecosystem. Terraform Cloud's continuous runs feature monitors drift between deployments without a manually scheduled CI pipeline. AWS Config's continuous compliance evaluation catches configuration changes the moment they occur rather than at the next scheduled plan. Tools like driftctl and Steampipe provide resource-level inventory comparison that surfaces unmanaged resources and configuration deltas without requiring Terraform state.

What the tooling can't automate is the categorization problem: distinguishing expected AWS-managed drift from unauthorized human drift from authorized-but-un-codified drift requires context that only comes from knowing the infrastructure and the team's operational patterns. The tooling surfaces the signal; the engineering judgment determines the response.

The teams that manage AWS-managed drift well do two things consistently: they configure ignore_changes proactively for every resource type that AWS mutates autonomously, and they treat a non-empty scheduled plan as a question that requires an answer, not background noise. The first reduces false positives. The second ensures that real drift doesn't go unexamined because it looked like expected noise.

Getting both right, clean ignore_changes configuration and responsive drift triage, is the kind of operational discipline we build into every infrastructure engagement at PulseSoft. If your terraform plan has stopped being a trustworthy signal, let's talk.

Key Takeaways

• Not all infrastructure drift is caused by humans. AWS continuously modifies resources through Auto Scaling events, RDS maintenance and failover, minor version auto-upgrades, and computed attribute updates. These produce Terraform plan noise that, if unmanaged, trains engineers to ignore the plan output.
• ignore_changes must be specific, not global. ignore_changes = all silences all drift, including intentional future configuration changes, and is how resources drift out of IaC management entirely. Ignore only the specific attributes that AWS manages autonomously, with a comment explaining why.
• When auto_minor_version_upgrade = true on RDS, engine_version must be in ignore_changes. Without it, AWS's auto-upgrade will show as drift in the next plan, and applying the plan will attempt to downgrade the engine: an operation AWS rejects. The combination of auto-upgrade without ignore_changes on engine_version produces a plan that cannot be applied.
• Auto Scaling desired_capacity must always be in ignore_changes if scaling policies are active. Applying a plan that includes the Terraform-declared desired capacity during a scaling event reverts the fleet to pre-event capacity, potentially reducing capacity during peak load. Terraform should manage min_size and max_size; the scaling system should manage desired_capacity.
• Severity routing in scheduled drift checks should be based on resource type, not change count. One changed attribute on a production IAM policy warrants immediate review. Twenty changed Auto Scaling desired_capacity attributes warrant a weekly review queue. Routing on count alone produces alert fatigue and missed high-severity drift.
• The scheduled plan surfaces the authorized-but-un-codified drift category (emergency console changes, manual scaling adjustments, one-off configurations), that everyone forgets to codify. Run it for two weeks before wiring it to on-call alerting, to calibrate the signal and eliminate false positives from misconfigured ignore_changes blocks.
• Clean ignore_changes configuration is a prerequisite for trustworthy drift detection. A scheduled plan that fires alerts on expected AWS-managed changes trains engineers to dismiss all drift alerts. Calibrate the noise floor first. Once the plan only surfaces genuine drift, the alerts become actionable.