Considerations for CI/CD Platforms

Must Support

  • Must support “pipelines as code” and auto-discover branches (ideally repos too)
    • E.g. Jenkinsfile, codefresh.yml, .circle.yml, travis.yml
  • Must support shared secrets and settings
    • E.g. GitHub Actions does not support shared secrets
  • Must support easy parallelization
    • Speed is critical. One way to speed up tests is to parallelize steps.
  • Must support easy integration with Kubernetes (I don’t want to manage the setup)
    • First-class Kubernetes support is essential
    • Minimal extra tooling should be required for Kubernetes-backed deployments
  • Must use container-backed steps
  • Must support webhook events from PRs originating from untrusted forks (E.g. open-source projects)
  • Must support ChatOps style requests
    • E.g. comments on PRs can trigger pipelines, slack commands can retry builds
  • Must support SSO
  • Must integrate with Slack
    • Slack notifications should be customizable
    • Slack should replace the need for most email notifications
  • Must be affordable
    • Platform should not require long-term commitments (SaaS)
    • Cost $10-20/user max
    • Support “unlimited” builds or have a pay-per-build model
    • Startup pricing preferred

Should Support

  • Should support approval steps with Slack notifications (ideally use slack buttons)
  • Should support RBAC/ABAC ACLs (e.g. for approval steps)
  • Should support local execution of pipelines for debugging
  • Should make it easy to discover all open PRs so it’s easy to re-trigger
  • Should support remote debugging (basically drop into any step via remote shell and poke around)
  • Should make it easy tag multiple versions of a docker image
  • Should support a “library” of pipelines or pipeline steps
  • Should support GitHub deployment notifications
  • Should support multiple OS-build platforms (e.g. iOS, OSX, Windows, Linux, Android, etc)