Terraform Azure DevOps Build and Release pipelines
If you use Terraform to manage your infrastructure as code, Azure DevOps for your continuous integration and continuous deployment (CI/CD) needs and you would like to combine the two, then this is the post for you.
Introduction
For the sake of keeping this post concise and to the point, I’m not going to talk about best practices and definitions but rather how I have setup Terraform and Azure DevOps Build and Release pipelines for personal and professional projects. The hope is that this post helps someone trying to do the same as it took me a minute to figure it out.
Overview of all the technologies referenced in this post
- Terraform — a tool used to manage your (usually cloud) infrastructure as code. It uses HashiCorp (creators of Terraform) proprietary programming language — HCL. The Terraform version is v0.14.2 and Terraform Azure provider version is 2.39.0
- Terraform commands — basic Terraform commands such as
init,plan,apply, etc. - Terraform workspaces — you can think of this as environment specific state. For example, there will be a state file for each environment — DEV, QA, STAGE and PROD
- Azure DevOps build and release pipelines— collaborate tool for teams. In this specific post, we’ll be looking at its build and release pipelines
- Azure Storage — Azure storage account where I store the Terraform state for each specific workspace. More specifically, blob storage
- Azure KeyVault — service primary used for storing sensitive information such as secrets
Assumptions
- You know Terraform file extensions, structures and what they’re used for (main.tf, variables.tf, etc.)
- You know basic Terraform commands and what they do
- You know what Terraform state is and how Terraform uses it
- You know what credentials Terraform needs in order to authenticate in Azure (subscription id, tenant id, client id and client secret)
Terraform
Let’s go over how I have my Terraform setup. For this demo, let’s take a look at the following scenario — you want to create an Azure Resource Group via Terraform. Thus, your main.tf file looks something like this.
Where the variables.tf file contains the following variables based on the current workspace (remember, I currently have 4 workspaces declared — dev, qa, stage and prod).
Note that the location and env variables are of type map (dictionary) where each key corresponds to each workspace I have defined.
Additionally, I have backends.tf file where I declare the definition of the azurerm backend and where the remote state files are stored.
resource_group_name— resource group of the Azure Storage accountstorage_account_name— the name of the Azure Storage account where the state files arecontainer_name— the name of the Azure Storage account (blob) container where the state files are
The remote state files are in the format of tf/terraform.tfstateenv:{env} where {env} is each workspace and prefix tf/ is the folder name. Below are examples of dev and qa. Note: if you’re starting from scratch and create a new workspace, then these are automatically created for you and you don’t have to worry about them.
That is my whole Terraform setup. Now it is time to dive into the Azure DevOps Build and Release pipelines to see how we can CI/CD the process.
How to setup your Azure DevOps build pipeline
Now that you have your Terraform scripts, it’s time to take a look at the actual Azure DevOps Build pipeline.
It should be simple and to the point. The output of it needs to be the Terraform scripts published as an artifact so they can be used in the release pipeline where the bulk of the work is done. This is what my azure-pipelines.yml file looks like.
As you can see, there are 3 steps in the process. The 1st one runs the Terraform command validate — this ensures that the scripts are valid and can be published. The 2nd command simply copies the Terraform scripts to the staging directory. Finally, the 3rd command publishes the scripts as an artifact. A typical build run would look like this.
That’s all you need in terms of the Azure DevOps Build pipeline. Now let’s take a look at the Release pipeline.
How to setup your Azure DevOps release pipeline
Now that you have your Build pipeline, it’s time to take a look at the Release pipeline where the infrastructure will be created.
For this demo, we’ll be looking at dev Release. At a high level view, this is what that looks like in Azure DevOps.
There is 1 job and 8 tasks in it. Let’s dive in.
Tasks
- Connect to Azure KeyVault where secrets are stored — such as subscription id, tenant id, client id, client secret and more. These secrets can then be referenced anywhere in the pipeline using the format
$(SECRET_NAME).
2. Write the secrets taken from Azure KeyVault to a local to the pipeline terraform.tfvars file.
3. Install Terraform v0.14.2
4. Run the init Terraform command. In the passed in arguments, you will see the $(StorageAccountKey) reference. This is the Azure Storage account key stored in Azure KeyVault (which was connected to a couple of tasks before).
5. Run the workspace select command based on the $(env) pipeline variable, in this case dev.
6. This one was the tricky one. Even though I was running terraform workspace select dev the pipeline was still unable to pick up the correct remote state file. Thus, I added this step where it writes the $(env) variable’s value, in this case dev, to the local Terraform environment file.
7. Run the plan Terraform command
8. Run the validate and apply Terraform commands
Conclusion
It’s a lot of steps to follow but this is the only way I found to make using Terraform in Azure DevOps Build and Release pipelines work. These limitations are not only because of the tool but also the Terraform extension that you need to install and use in Azure DevOps. It’s currently limited to what it can do.
Full details
IaC (Terraform) source code where these examples were taken from
IaC (Terraform) source code jump to
