Weekly Journal 146 - Infrastructure as Code

Large Infrastructure as Code Projects

This is a just a quick thought this week. Over the past several years I’ve noticed some recurring issues with larger infrastructure as code (IaC) projects. Namely, they become harder and harder to maintain over time in ways that are different from application code projects. As a project grows over time, I’ve noticed that it’s very easy to introduce changes that prevent using IaC to recreate the environment. For example, it’s very easy in Terraform to introduce a circular dependency. Since the system is growing incrementally, Terraform won’t see it as an issue unless you try to recreate the entire system from scratch.

As IaC projects grow they become harder and harder to understand. I wish I knew of a way to better tame that complexity, as now what would take 5-10 minutes to do manually in the AWS console can take several days to accomplish in Terraform due to the complexity and interactions of all the resources. It makes it very difficult to onboard new developers and engineers, as well as slows down development velocity.

My experience is primarily with Terraform and Ansible, so this may be different for other IaC tools. I don’t have answers to these issues, but I find them interesting to think about.