Change Control Management

Here are some of the tools and tricks at your disposal for enforcing a modern change control process.

  • Use a Version Control System
    • With a VCS like GitHub, you'll be able to point to every change
    • With branches, you'll be able to keep pending changes out of masteruntil they are ready
  • Use Infrastructure as Code
    • Define the business logic of infrastructure in reusable modules
    • Separate the business logic from the configuration
    • Stick all the code & configuration into VCS
  • Use Automation (E.g. “Operations by Pull Request”)
    • Eliminate humans from running commands by hand
    • Pipelines promote code and configuration changes through environments
  • Use a Pull Request workflow with Code Reviews & Approvals
  • Use Pipeline Approval Steps
  • Use Notifications
    • Send a slack notification for every deployment
    • Comment on GitHub Commit SHA for every deployment of that commit
  • Use Branch Protections
    • Require Pull Request Approvals
    • Dismiss approvals if changes are pushed
    • Require status checks to pass
    • Enforce CODEOWNERS
  • Use CODEOWNERS
    • Use teams to connote stakeholders (e.g. @secops or @dba or @qaor @frontend) to ensure approvals from subject matter experts
    • Use narrowly scoped paths for teams (E.g. terraform/iam/* @secops)
  • Use Policy Enforcement
    • Tools like Open Policy Agent, conftest,  tfsec help to define contracts and enforce them
    • Integrate the tools with your CI/CD pipelines (execute pipelines from  master branch to enforce pipelines are not bypassed in the PR/branch)
  • Use Multiple Accounts, Stages
    • Test changes in isolation and use a formal process to promote changes
  • Use Version Pinning
    • Always pin your dependencies to a version (e.g. using semver or commit SHAs)
    • Never overwrite any version of the software, always create a new release/tag
    • Pinning to master or latest does not count!
  • Use Feature Flags
    • Feature flags can ensure that functionality is only turned on when it's ready and easily disabled
    • Controls around feature flag access limit who can toggle it and a change log of when it was modified

 

 

Author Details
Erik Osterman
CEO
Erik Osterman is a technical evangelist and insanely passionate DevOps guru with over a decade of hands-on experience architecting systems for AWS. After leading major cloud initiatives at CBS Interactive as the Director of Cloud Architecture, he founded Cloud Posse, a DevOps Accelerator that helps high-growth Startups and Fortune 500 Companies own their infrastructure in record time by building it together with customers and showing them the ropes.
https://osterman.com
[email protected]