Drift is not an unusual situation. It’s just a normal ongoing reality of any cloud. If you work with infrastructure long enough, you stop being surprised by drift and start working on how to deal with it. In this guide we will go through the process of detection and fixing Iac Drift step-by-step.
What Is IaC Drift?
The term drift refers to a situation where there is a lack of correspondence between the actual state of your infrastructure and the state of the infrastructure that is derived from your code. Suppose someone needs to access the console at 2 a.m. because the security group is not functioning well; the change made will not be documented back in Terraform. Similarly, there could be an instance in which a Lambda function gets an unnecessary increase in memory during an incident and continues using it. It is also possible to alter auto-scaling policies with the help of different tools.
At the moment of using such tools for fixing problems and eliminating the need for documentation, all actions seem to be justified. However, when time passes, no one recalls why AWS configuration differs from the repository information, and this very gap results in the increase of risk levels.
Why Unmanaged Drift Actually Hurts Security
Security teams worry about drift because it breaks their assumptions. If your IaC states there is only private access in the database, but the console suggests something different, it means your compliance report is misleading you. Reliability may also decrease, since your next deployment can destroy someone’s emergency fix silently and bring back the bug it was aimed to fix.
Drift detection and drift assessment and drift remediation are three different tasks and most teams do only the first one. Performing terraform plan and listening to its output is drift detection. Making judgement about whether the change is risky or not is the assessment. If you want to remediate it, you have to carry out some action both ways.
Steps to Detect and Fix IaC Drifts
Step 1: Establish a Single Source of Truth
Select a single process and make it accountable for the operations. If you are using CloudFormation along with both Terraform and Pulumi in same account, the argument is lost before it even begins.
- Start by laying down the definition of what the desired state will look like, put it down in writing.
- Make the IaC repository the only location where changes get approved.
- Implement version control for everything, no exceptions for “quick fixes”.
- Make sure that each piece of the infrastructure has an owner, and not a team, but an individual.
I know teams hate the idea of one owner per resource. It feels bureaucratic. But when three people think they own a subnet, nobody actually does.
Step 2: Find the Root Cause Before You Touch Anything
Don't merely recognize that something occurred, but attempt to determine the cause for that. Was it a console click during an outage? Was it a different automation tool that was not talked about with the infra team? Or was it a provider update that changed the default value in the process?
- Use CloudTrail or any audit trail for finding the event in question
- Always gather additional information before assuming that an incident occurred due to malicious intent or negligence
- Make sure to distinguish between singular incidents from repetitive patterns.
If a certain drift happens every week, it is not an issue of an unfortunate incident, but a prevailing difference in process that will not be fixed by the use of remediation scripts.
Step 3: Detect and Prioritize by Risk, Not by Volume
A drift report that contains hundreds of entries is useless unless you know which parts of it to focus on. I learned that through a painful experience with my client’s account where a clean drift scan was reported with a lot of tag mismatches, unbalancing the situation for the entire IAM policy.
- Conduct a comparison in accounts, regions, or environments.
- Identify IAM, network rules, encryption and public exposure first.
- Put naming and tag issues to the background.
- Don’t confuse visual differences with true security exposure.
Tags will become important in the future, but they will never be as important as an opened port on the Internet.
Step 4: Decide, Reconcile or Update the Code
Every instance of drift requires a decision to be made. This can either mean that the infrastructure is incorrect, in which case it needs to align itself with the code, or that the infrastructure has become outdated and thus needs to be updated. The process of reconciling means that one will need to apply the stated config back to the infrastructure resource. On the other hand, through updating IaC, one is willing to accept the change as true and put it into the repo.
There is no one right answer here and whoever says that there is has never worked with production systems. Certain teams tend to reconcile always, while there are some others that choose to reconcile only occasionally due to its risks of eliminating a proper change done previously while working under pressure. I’d favor reviewing significant drifts by hand before reconciling them automatically, even if that would slow things down.
Step 5: Automate the Boring Parts
After recognizing what’s important, the next step is to automate its detection. Doing drift checks manually won’t work in the long run because at some point someone will forget to do them.
- Implement wire drift detection directly in your CI/CD setup.
- Plan your checks instead of leaving it to your memory.
- Use policy-as-code to flag the unsafe patterns automatically.
- Limit automatic remediation process in case of incidents to
- Keep a human in the loop for anything touching production access
While automating everything may sound good in theory, the reality is that there is a huge risk that a script will undo a legitimate change that was made purposely.
Step 6: Prevent Drift From Piling Up in the First Place
Detection and remediation approaches are generally reactive and what we actually want is to limit how much drift we create. Limit access to the console as much as it is practically possible and make sure you are always following a pre-approved process for any change.
- Remove unnecessary direct console access
- Use least privilege access approach everywhere
- Alert when there are major config changes happening
- Maintain audit logs that are used (and not just stored)
Alerts nobody reads are just noise with extra steps.
Step 7: Get Serious About State Management and Governance
An invalid or mismanaged state file results in drift that does not arise through human action. Use remote state, secure it, and keep the providers and modules up to date rather than sticking to whatever version is working two years ago.
- Monitor the frequency of drift and the elapsed time in detection and correction
- Pay attention to drift patterns since they indicate a failure in the process
- Evaluate the governance rules each quarter rather than performing the task once and forgetting about it
Governance sounds tedious until the quarter when governance review is not done and the consequences emerge.
Best Practices to Fix Drift Detection
Even though this is not a difficult concept in theory, it can be difficult to do in practice.
- Always treat your IaC repository as the actual source of truth, even though performing a quick fix via the console may seem more efficient in some instances.
- Look at the reason behind the drift before trying to fix it. There is a reason why there was drift, therefore understanding it will help fix the drift.
- Monitor constantly, since there could be silence only a couple of times, and that is exactly when the risk accumulates.
- Do not assess impact through the number of changed lines, because sometimes a change in just one line of IAM might be more important than fifty tag changes.
- Automate what can be automated safely, review other actions manually.
- Limit the number of people who are authorized to make changes in the first place, thus preventing drift effectively.
- Have a monitoring system that will be actually checked, rather than even more useless logs.
- Have a point of accountability.
The Real Takeaway
Drift is not an issue in your systems; it is a commonly encountered issue of running systems at scale. Its complete removal is an impossible task in practice. The issue is not to achieve an absolute drift free status. The objective should be to track drift that is significant enough and address it subject to the facts.
Decide what is your real reference point. Monitor the drift all the time. Understand what matters the most. Fix or revise code whenever required. Automate routine operations. Keep monitoring the situation. Manage it well since it is essential.
I have watched teams treat drift management as a one-time cleanup project, and every single one of them ended up back at square one within a year. Build it as an ongoing habit instead, and it stops being a fire drill and starts being just another Tuesday.
Also Read:
