Change Control Management
Here are some of the tools and tricks at your disposal for enforcing a modern change control process.
- Use a Version Control System
- With a VCS like GitHub, you'll be able to point to every change
- With branches, you'll be able to keep pending changes out of
master
until they are ready
- Use Infrastructure as Code
- Define the business logic of infrastructure in reusable modules
- Separate the business logic from the configuration
- Stick all the code & configuration into VCS
- Use Automation (E.g. “Operations by Pull Request”)
- Eliminate humans from running commands by hand
- Pipelines promote code and configuration changes through environments
- Use a Pull Request workflow with Code Reviews & Approvals
- Use Pipeline Approval Steps
- Use Notifications
- Send a slack notification for every deployment
- Comment on GitHub Commit SHA for every deployment of that commit
- Use Branch Protections
- Require Pull Request Approvals
- Dismiss approvals if changes are pushed
- Require status checks to pass
- Enforce
CODEOWNERS
- Use
CODEOWNERS
- Use teams to connote stakeholders (e.g.
@secops
or@dba
or@qa
or@frontend
) to ensure approvals from subject matter experts - Use narrowly scoped paths for teams (E.g.
terraform/iam/* @secops
)
- Use teams to connote stakeholders (e.g.
- Use Policy Enforcement
- Tools like
Open Policy Agent
,conftest
,tfsec
help to define contracts and enforce them - Integrate the tools with your CI/CD pipelines (execute pipelines from
master
branch to enforce pipelines are not bypassed in the PR/branch)
- Tools like
- Use Multiple Accounts, Stages
- Test changes in isolation and use a formal process to promote changes
- Use Version Pinning
- Always pin your dependencies to a version (e.g. using
semver
or commit SHAs) - Never overwrite any version of the software, always create a new release/tag
- Pinning to
master
orlatest
does not count!
- Always pin your dependencies to a version (e.g. using
- Use Feature Flags
- Feature flags can ensure that functionality is only turned on when it's ready and easily disabled
- Controls around feature flag access limit who can toggle it and a change log of when it was modified
Author Details
CEO
Erik Osterman is a technical evangelist and insanely passionate DevOps guru with over a decade of hands-on experience architecting systems for AWS. After leading major cloud initiatives at CBS Interactive as the Director of Cloud Architecture, he founded Cloud Posse, a DevOps Accelerator that helps high-growth Startups and Fortune 500 Companies own their infrastructure in record time by building it together with customers and showing them the ropes.