I spent some time this weekend getting caught up on the state of Jenkins in 2020. This post will focus on the pros and cons of Jenkins (not Jenkins X - which is a complete rewrite). My objective was to set up Jenkins following "Infrastructure as Code" best practices on Kubernetes using Helm. As part of this, I wanted to see a modern & clean UI throughout and create a positive developer experience. Below is more or less a brain dump of this experiment.
Jenkins has a lot of redundant plugins. Knowing which one to use takes some experimentation and failed attempts. The most common example cited is "docker." Personally, I don't mind the hunt - that's part of the fun.
Jenkins has many plugins that seem no longer maintained. It's important to make sure whatever plugins you chose are still receiving regular updates (as in something pushed within the last ~12 months).
Not all plugins are compatible with Declarative Pipelines. IMO using Declarative Pipelines is the current gold standard for Jenkins. Raw imperative groovy pipelines are notoriously complicated and unmanageable.
No less than a few dozen plugins are required to "modernize" Jenkins. The more plugins, the greater the chance of problems during upgrades. This can be somewhat mitigated by using command-line-driven tools run inside containers instead of installing some of the more exotic plugins (credit: Steve Boardwell).
There's no (maintained) YAML interface for Jenkins Pipelines (e.g. Jenkinsfile.yaml). Most modern CI/CD platforms today have adopted YAML for pipeline configuration. In fact, Jenkins X has also moved to YAML. The closest thing I could find was an alpha-grade prototype with no commits in 18 months.
The Kubernetes Plugin works well but complicates Docker Builds. Running the Jenkins Slaves on Kubernetes and then building containers requires some trickery. There are a few options, but the easiest one is to modify the PodTemplate to bind-mount /var/run/docker.sock. This is not a best-practice, however, because it exposes the host-OS to bad actors. Basically, if you have access to the docker socket, you can do anything you want on the host OS. The alternatives like running "PodMan", "Buildah", "Kaniko", "Makisu", or "Docker BuildKit" on Jenkins have virtually zero documentation, so I didn't try it.
The PodTemplate approach is problematic in a modern CI/CD environment. Basically, with a PodTemplate you have to define before the Jenkins slave starts the types of containers you're going to need as part of your pipeline. For example, you define one PodTemplate with docker, golang and terraform. When the Jenkins slave starts up, a Kubernetes Pod will be launched with 3 contains (docker, golang and terraform). One nice thing is that all those containers will be able to share a filesystem and talk over localhost since they are in the same Pod. Also, since it's a Pod, Kubernetes will be able to properly schedule where that Pod should start, and if you have autoscaling configured, new nodes will be spun up on-demand. The problem with this, however, is subtle. What if you want a 4th container to run that is a product of the "docker" container and share the same filesystem? there's no really easy way to do that. These days, we will frequently build a docker container in one step, then run that container and execute some tests in the next step. I'm sure this can be achieved, but nowhere near as easily as with codefresh.
It's still not practical today to run "multi-master" Jenkins for High Availability without using Jenkins Enterprise. That said, I think it's moot when operating Jenkins on Kubernetes with Helm. Kubernetes is constantly monitoring the Jenkins process and will restart it if unhealthy (and I tested this inadvertently!). Also, when using Helm if the rollout fails health checks, the previous generation will stay online allowing the bugs to be fixed.
Docker layer caching is non-trivial if running with Ephemeral Jenkins Slaves under kubernetes. If you have a large pool of nodes, chances are that every build will hit a new node, thus not taking advantage of layer caching. Alternatively, if using the "Docker in Docker" (dnd) build-strategy every build will necessarily pull down all the layers. This will both add considerably to transit costs and build times as docker images are easily 1G these days.
There's lots of stale/out-of-date documentation for Jenkins. I frequently stumbled on how to implement something that seemed pretty basic. Anyways, this is true of any mature ecosystem that has a tremendous amount of innovation, lots of open sources, and been around for 20 years.
The "yaml" escape hatch for defining pod
specs is sinfully
ugly.
In fact, I think it's a horrible precedent that will turn people off from Jenkins. It's part of what gives it a bad
wrap. The rest of the Jenkinsfile DSL is rather clean and readable, but embedding raw YAML into my declarative
pipelines is not a practice I would encourage for any team. To be fair, some of the ugliness could be eliminated by
using readFile or readTrusted steps (credit: Steve Boardwell), but again it’s not that simple.
I would like to end this on a positive note. All in all, I was very pleasantly surprised by how far Jenkins has come in the past few years since we last evaluated it.
Overall simple architecture to deploy (when compared to "modern" CI/CD systems). No need to run tons of backing services.
Easily extract metrics from your build system into a platform like Prometheus. Centralize your monitoring of things running inside of CI/CD infrastructure. This is very difficult (or not even possible) to do with many SaaS offerings.
Achieve greater economies of scale by leveraging your existing infrastructure to build your projects. If you run Jenkins on Kubernetes, you immediately get all the other benefits. Spin up node pools powered by "SpotInst.com Ocean" and get cheap compute capacity with preemptible "spot" instances. If you're running Prometheus with Grafana, you can leverage that to monitor all your build infrastructure.
Arguably the Jenkinsfile Declarative Pipelines DSL is very readable, in fact, it looks a lot like HCL (HashiCorp Configuration Language). To some, this will be a "Con" - especially if YAML is a requirement.
Jenkins "Configuration as Code" plugin supports nearly everything you'd need to run Jenkins itself in a "GitOps" compliant manner. And if it doesn’t there is always the configuration-as-code-groovy plugin which allows you to run arbitrary Groovy scripts for the bits you need (credit: Steve Boardwell).
Jenkins can be easily deployed for multiple teams. This is an easy way to mitigate one of the common complaints that Jenkins is unstable because lots of different teams have their hands in the cookie jar.
Jenkins can be used much like the modern CI/CD forms that use container steps rather than complicated groovy scripts. This is to say, yes, teams can do bad things with Jenkins but with the right "best practices" your pipelines should be about as manageable as any developed with CircleCI or Codefresh. Stick to using container steps to reduce the complexity in the pipelines themselves.
Jenkins Shared Libraries are also pretty awesome (and also one of the most polarizing features). What I like about the libraries is the ability for teams to define "Pipelines as Interfaces". That is, applications or services in your organization should almost always be deployed in the same way. Using versioned libraries of pipelines helps to achieve this without necessarily introducing instability.
Just like with GitHub Actions, with Jenkins, it's possible to "auto-discover" new repositories and pipelines. This is sweet because it eliminates all the ClickOps associated with most other CI/CD systems including CircleCI, TravisCI, and Codefresh. I really like it when I can just create a new repository, stick in a Jenkinsfile, and it "just works".
Jenkins supports what seems like an unlimited number of credential backends. This is a big drawback with most SaaS-based CI/CD platforms. With the Jenkins credential backends, it's possible to "plug and play" things like "AWS SSM Parameter Store", "AWS Secrets Manager" or HashiCorp Vault. I like this more than trusting some smaller third-party to securely handle my AWS credentials!
Jenkins PodTemplates supports annotations, which means we can create specially crafted templates that will
automatically assume AWS roles. This is rad because we don't even need to hardcode any AWS credentials as part of
our CI/CD pipelines. For GitOps, this is a holy grail.
Jenkins is 100% Free and Open Source. You can upgrade and get commercial support from Cloud Bees which also includes a "tried and tested" version of Jenkins (albeit more limited in the selection of plugins).
To conclude, Jenkins is still one of the most powerful Swiss Army knifes to get the job done. I feel like with Jenkins anything is possible, albeit sometimes with more effort and 3 dozen plugins. As systems integrators, we're constantly faced with yet-unknown requirements that pop-up at the last minute. Adopting tools that provide "escape hatches" provide a kind of "peace of mind" knowing we can solve any problem.
Parts of it feel dated like the GUI Configurations, but that is mitigated by Configuration as Code and GitOps. I wish some things like building and running Docker containers inside of pipelines on Kubernetes was easier. Let's face it. Jenkins is not the cool kid on the block anymore and there are many great tools out there. But the truth is few will stand the test of time the way Jenkins has in the Open Source and Enterprise space.
kubernetes (we tested 1.21.2) is what enables Jenkins Slaves to be spun upon demand. It comes preconfigured when
using the official Jenkins helm chart.
workflow-job (we tested 2.36)credentials-binding (we tested 1.20)git (we tested 4.0.0)workflow-multibranch (we tested 2.21) is essential for keeping Jenkinsfiles in your repos. The multi-branch
pipeline detects branches, tags, etc, from within a configured repository, so Jenkins works more like Circle,
Codefresh or Travis.
github-branch-source (we tested 2.5.8) - once configured will scan your GitHub organizations for new repositories
and automatically pick up new pipelines. I really wish more CI/CD platforms had this level of autoconfiguration.
workflow-aggregator (we tested 2.6)configuration-as-code (we tested 1.35) allows nearly the entire Jenkins configuration to be defined as codegreenballs (we tested 1.15) because I've always green is the color of success =Pblueocean (we tested 1.21.0) gives Jenkins a modern look. It clearly depicts stages, progress, and most other
systems we've seen like CircleCI, Travis or Codefresh.
pipeline-input-step(we tested 2.11)simple-theme-plugin (we tested 0.5.1) allows all CSS to be extended. Combined with the "material" theme for
Jenkins you get a complete facelift.
ansicolor (we tested 0.6.2) - because many tools these days have ANSI output like Terraform or NPM. It's easy to
disable the color output, but as a developer, I like the colors as it helps me quickly parse the screen output.
slack (we tested 2.35)saml (we tested 1.1.5)timestamper (we tested 1.10) - because with long-running steps, it's helpful to know how much time elapsed between
lines of output
Add the following nginx-ingress annotation to make Blue Ocean the default"
nginx.ingress.kubernetes.io/app-root: /blue/organizations/jenkins/pipelinesHide the [Pipeline] output with some simple CSS and the simple-theme-plugin:
.pipeline-new-node {
display: none;
}When researching this post and my "Proof of Concept", I referenced some links and articles.
github.com/jenkinsci/pipeline-github-plugin/blob/master/README.md
docs.cloudbees.com/docs/admin-resources/latest/plugins/github-branch-source
stackoverflow.com/questions/51110083/suppress-scripted-pipelines-output-in-jenkins
medium.com/@gustavo.guss/jenkins-building-docker-image-and-sending-to-registry-64b84ea45ee9
medium.com/@timhberry/terraform-pipelines-in-jenkins-47267129ff06
akomljen.com/set-up-a-jenkins-ci-cd-pipeline-with-kubernetes/