Codefresh Variable Substitution

JeremyRelease Engineering & CI/CDLeave a Comment

It’s important to understand the behavior of variables like ${{CF_PULL_REQUEST_NUMBER}} in Codefresh pipelines and how it interacts with shell commands.

When you use ${{CF_PULL_REQUEST_NUMBER}} in a YAML file, Codefresh handles it like this:

  • If the variable is set, the whole thing is replaced with the value of the variable
  • If the variable is not set, it is left unchanged.

When you execute a command in a freestyle step, that command is passed to bash. So when CF_PULL_REQUEST_NUMBER is set to something like 34:

cf_export IMAGE_TAG=pr-${{CF_PULL_REQUEST_NUMBER}}

becomes: cf_export IMAGE_TAG=pr-34, but when the variable is not set, the command is:
cf_export IMAGE_TAG=pr-${{CF_PULL_REQUEST_NUMBER}}
which of course is bad bash variable interpolation syntax.

We (Cloud Posse) have created a tool to help with this called require_vars we ship in geodesic. If your pipeline should not run if certain variables are not supplied, you can create an initial step like this:

validate:
    title: "Validate"
    description: "Ensure build parameters are present"
    stage: Prepare
    image: cloudposse/geodesic:0.116.0
    entry_point: /etc/codefresh/require_vars
    cmd:
      - |-
        ${{GITHUB_REPO_STATUS_TOKEN}} Personal Access Token used to give scripts
        permission to update the "status check" status on GitHub pull requests
      - ${{SLACK_WEBHOOK_URL}} Secret URL used by scripts to send updates to a Slack channel
      - |-
        ${{AWS_DOCKER_REPO_HOST}} The host hame portion of the ECR Docker repo to use.
        Typically something like 123456789012.dkr.ecr.us-east-1.amazonaws.com
      - |-
        ${{CF_PULL_REQUEST_NUMBER}} The PR number from GitHub.
        The PR number is only set if this build was triggered in relation to a PR.
        Requiring this to be present means requiring this pipeline to only work with PRs.

The PR number is only set if this build was triggered with a PR.
Requiring this to be present means requiring this pipeline to only work with PRs.

This step will verify the referenced variables are set and output the text as an error if they are not, causing the build to fail at that step. We recommend using it whenever a pipeline would not function properly with missing variables.

The other thing is that if a variable is optional, you test whether it is set or not by checking to see if the variable's value contains the variable's name. So, if a step should only run if there is not a PR number assigned, you can add a test like this:

when: 
  condition: 
    all: 
      isNotPR: 'includes("${{CF_PULL_REQUEST_NUMBER}}", "CF_PULL_REQUEST_NUMBER") == true'

(It may seem, and really should be, redundant to have == true but it seems to be required.)

It is kind of weird and confusing but makes sense once you get the hang of it. Please feel free to reach out to me if you have further questions.

PodDisruptionBudget “Gotchas”

JeremyBest Practices, DevOpsLeave a Comment

To allow the Kubernetes Cluster Autoscaler to move pods around, pods (sometimes) need PodDistruptionBudgets which can specify either minAvailable or maxUnavailable, but not both. There are a bunch of “gotchas” to look out for.

You cannot set minAvailable: 0
It's not that you can't set minAvailable it to zero, but since you are not allowed to set both minAvailable and maxUnavailable, most Helm charts have code like:

 

{{- if .Values.podDisruptionBudget.minAvailable }}
minAvailable: {{ .Values.podDisruptionBudget.minAvailable }}
{{- end  }}
{{- if .Values.podDisruptionBudget.maxUnavailable }}
maxUnavailable: {{ .Values.podDisruptionBudget.maxUnavailable }}
{{- end  }}

If you set minAvailable: 0 that is the same as not setting it. That results in neither value getting set, which is effectively the same as setting maxUnavailable: 0 (which, of course, you also cannot do).

The fix: set minAvailable: "0%". So you may wonder, “why to bother setting a PodDisruptionBudget at all, if you are going to allow all the pods to be deleted?” Well, the reason is that this gives the Autoscaler explicit permission to evict pods that it might otherwise be too cautious about evicting. For example, anything that uses emptyDir for anything. The contents of emptyDir will be lost when the pod is evicted, so the Autoscaler will not evict the pods without explicit permission to avoid deleting something important.

It is dangerous to set minAvailable: 1
It may seem innocuous to set minAvailable: 1, but frequently we scale deployments down to 1 instance, and if you combine that with minAvailable: 1 then you are stuck with a single pod that cannot be evicted, which in turn will prevent the instance from being taken out of service.

Not a fix: minAvailable: 25%. Kubernetes does not round these percentages to the nearest, it always rounds up. So with 1 replica, it will round 25% up to 1, so this is not a fix at all.

The fix: set maxUnavailable: 50%. That will avoid knocking out the service when there are replicas to spare but not prevent the service from being evicted.

Sometimes you have to set both minAvailable and maxUnavailable
While you are not allowed to set both minAvailable and maxUnavailable in the actual PodDisruptionBudget resource, if the Helm chart provides a default value for one of them and you want to use the other one, you have to set both in the helmfile. Set the one you do not want to "" to hint to readers that you are unsetting a default.