A customer recently asked me for some help with setting up a controlled deployment process inside of Azure.
Here is what they asked for:
- Allow everyone to create infrastructure from a set of templates and those templates only
- Give developers the largest amount of freedom while putting controls around deployments in place
Here is the solution we came up with:
1 – Problem Statement
What the client was looking to avoid was a scenario where a customer deploys an ARM template into Azure directly without any further checks.
2 – Avoiding direct deployments
Instead of deploying directly, developers can access a central repository that contains all approved Azure ARM templates. They can submit pull request to the Git repo where these ARM templates are kept. This triggers a peer review and an automatic template validation cycle.
3 – Making templates available
Every evening, the templates are updated. The VSTS project triggers a build that verifies the templates and runs integration tests to check if a potential deployment would run to plan. If the build is successful, a new version of the approved templates gets uploaded to Azure storage. Credentials are created and they are made available to a service principal (service account) via KeyVault.
The VSTS repo also contains templates for Azure Automation Run books. These run through unit- and integration tests and are eventually uploaded to an Automation account.
4 – Enabling controlled deployments
A service principal is a non-user account in Azure AD. It can receive permissions, just like a user account, but can also be locked down to only do what it needs to do. In our case it needs to:
- Read from KeyVault
- Read templates
- Run Azure automation jobs
- Run deployments
The service principal has certificate authentication configured. The certificates are shared with an Azure Automation account. This enables it to be used as a “Run As” account during runbook executions.
The service principal is the only account in the subscription that can make deployments directly.
5 – Creating an endpoint for controlled deployments
The logic that triggers the deployment is stored in an Azure Automation run book that gets deployed, whenever a new iteration of the approved templates rolls out.
The setup – as it is – allows us to go into the automation account and kick a deployment off, but this is inconvenient for most consumers.
To make things easier, a web hook trigger can be created. This allows us to kick off run books by posting content to a URL. In the request body, we can specify deployment parameters, which allows for user customisations where required.
It is important that we validate any web hook requests coming in, and reject any deployment specifications that either do not make sense or are just invalid.
Having a single entry point for deployment requests allows you to perform additional validations on top of the normal deployment validations.
- Consumers who deploy need to send a cost centre number that eventually becomes a tag on the resource. If their cost centre number isn’t valid the deployment is rejected.
- Consumers can specify an e-mail address for updates on the deployment status. They get notified, if anything does not go to plan, or if their deployment times out.
There are some drawbacks to the approach though:
- Limited experimentation
- Team is restricted to a standard set of resources
- Requirement for additional documentation and process training