#codefresh - October 2019

Anyone else out there having issues with pushing images to AWS ECR? Codefresh support has gone dark and all my pipelines now fail to push to AWS ECR.

As if there's an outage, I don't think so, i've successfully pushed a few to ECRs in us-east-1 in the last hour or so

thanks… my pipelines push to us-west-2. All of them are failing including ones that worked before

are you using the integrated registries or

provider: ecr

Has anyone seen issues with codefresh variables interpolating with the build step?

stage: build-images
type: build
title: Build Api Image
description: build api image
working_directory: '${{CF_VOLUME_PATH}}/${{CF_REPO_NAME}}'
dockerfile: '${{dockerfile}}'
image_name: '${{image_name}}'
tag: '${{vtag}}-${{CF_PULL_REQUEST_NUMBER}}'

I am getting a scenario where the build step fails on subsequent commits to the feature branch and will not interpolate the ${{CF_PULL_REQUEST_NUMBER}} at run time.

I get the following at build time:

Building image: my_image_name:0.1.0-${{cf_pull_request_number}} using unique hash: 6903d14a13db6d2cef7a7134925ea0f68468bd00

That variable is only populated when pipeline is triggered by a Pr webhook

Are you running it manually by any Chance?

yes, i have been running automatically and manually. The manual iterations appear to be the iterations that fail. I have had to run the pipeline manually at times, since I have been experiencing intermittent Push step failures to AWS ECR. It is unfortunate that this attribute of the variable is not documented. https://codefresh.io/docs/docs/codefresh-yaml/variables/

@here public #office-hours starting now! join us to talk shop https://zoom.us/j/508587304

@Michael Kolb … I just had the same issue with variables in manual pipelines. It happens because the CodeFresh environment variables are left undefined. The key is to override the variables in the ‘options’ passed to the run

 if (servicePipelineMap.has(key)) {
          const { pipeline, trigger } = servicePipelineMap.get(key)
          const data = {
            trigger,
            sha,
            branch: targetBranch,
            variables: {
              CF_PULL_REQUEST_ID: id,
              CF_PULL_REQUEST_NUMBER: number,
              CF_BRANCH: targetBranch,
              CF_BASE_BRANCH: prBranch,
              CF_PULL_REQUEST_TARGET: targetBranch,
              CF_COMMIT_AUTHOR: user.login,
            },
          }
          app.log(data.variables)
          const workflowId = await sdk.pipelines.run({ name: pipeline }, data)
          app.log(
            `${pipeline} triggered a pipeline with id ${workflowId} on PR ${number} to ${ref}`
          )
        } else {
          app.log(`There is no pipeline associated with this service`)
        }

@Michael Kolb this is with the codefresh js sdk?

Yes @Erik Osterman (Cloud Posse) Being as I need to manually trigger pipelines outside of CodeFresh I found the JS-SDK to be the easiest to work with. They should make a point of featuring it.

I found it only by going through their github org

@Michael Frohberg Thanks for the sharing the information

Anyone else having issues with support response times from support@codefresh.io?

@Michael Kolb what is the issue?

@Oleg Sucharevich I have a technical issue with image push to ECR that is outstanding now for over 2 days with multiple emails threads

From the logging that i get the pipeline is not finding the image in r.cfcr.io/navican/ from the candidate that is built. The sha256 references are not matching up

Can you please send me the build id?

Will escalate it

Thanks!

@Oleg Sucharevich Thanks for the help

Sure 🙂

Has anyone integration CodeFresh with DataDog? This way we can see all of the events there and create monitors from it? For example, we were using this to create an alert that got sent to OpsGenie whenever our infrastructure repo failed.

You can try this. Not certain it is what you're looking for but it's what I have on hand.

 SendDatadogEvent:
    title: Sending Datadog Event
    image: concourse/datadog-event-resource:latest
    commands:
      - >-
        echo '
        {
          "source": {
            "application_key": "${{DATADOG_APPLICATION_KEY}}",
            "api_key": "${{DATADOG_API_KEY}}",
            "filter": ""
           },
           "params": {
             "title": "${{CF_REPO_NAME}}",
             "text": "${{CF_BUILD_URL}}",
             "priority": "low",
             "alert_type": "success",
             "host": "<https://g.codefresh.io>",
             "aggregation_key": "${{CF_BUILD_ID}}",
             "source_type": "codefresh",
             "tags": [
               "codefresh"
             ]
           }
        }' | /opt/resource/out ./event.json

Perfect, I can adapt that as needed. Hopefully one day there is native integration.

https://community.codefresh.io/ launched! that's nice

Codefresh still does not support limiting pipeline concurrency?

If I wanted to only ever have 1 instance of pipeline running, and all others to queue - this is still only done by a step with codefresh cli script?

I think there's an option to cancel previous build

but not limit concurrency

@dustinvb did this change?

This is found under the pipeline SETTINGS -> POLICIES tab.

can I set this from within codefresh.yml?

Yes, under the spec section of the pipeline.

spec:
  terminationPolicy:
    - type: branch
      event: onCreate
      ignoreBranch: true
  concurrency: 1
  triggerConcurrency: 1
  externalResources: []

Looks like our spec examples need to be updated. Will put in that request now.

this is great!

@Igor Rodionov @Andriy Knysh (Cloud Posse) @Jeremy G ☝️

we should drop the check for concurrency and just use this.

yea that’s good

But @Erik Osterman (Cloud Posse) aren't we generally testing for concurrency of a build in a different pipeline?

no

same pipeline

I'm thinking we check to make sure all PR build/deploys are finished before running destroy. That's the only concurrency check that comes to mind.

we check if the same pipeline is running and wait if it is

Docker Hub Incident
Oct 15, 17:24 UTC
Update - More information here:
- https://status.docker.com/pages/incident/533c6539221ae15e3f000031/5da5fc78c7b1c11cd8603721
- https://status.docker.com/Oct 15, 17:19 UTC
Update - More information here:
- https://status.docker.com/pages/incident/533c6539221ae15e3f000031/5da5fc78c7b1c11cd8603721
- https://status.docker.com/Oct 15, 17:19 UTC
Identified - Docker Hub has reported an incident:
Affected Components: Docker Hub Web, Docker Hub Registry, Docker Hub Automated...

Docker Hub Incident
Oct 15, 18:07 UTC
Resolved - Docker Hub has reported that the affected services are operational again.Oct 15, 17:24 UTC
Update - More information here:
- https://status.docker.com/pages/incident/533c6539221ae15e3f000031/5da5fc78c7b1c11cd8603721
- https://status.docker.com/Oct 15, 17:19 UTC
Update - More information here:
- https://status.docker.com/pages/incident/533c6539221ae15e3f000031/5da5fc78c7b1c11cd8603721
- https://status.docker.com/Oct 15, 17:19 UTC
Identified - Docker Hub has...

Anyone know how to have cf_export not actually echo the value? Not sure why it does this.

Docker Hub has reported an ongoing incident
Oct 15, 19:43 UTC
Investigating - Docker Hub has reported an ongoing incident

This could affect your Codefresh builds.

More information here:
https://status.docker.com/pages/incident/533c6539221ae15e3f000031/5da5fc78c7b1c11cd8603721

Docker Hub has reported an ongoing incident
Oct 15, 20:59 UTC
Monitoring - From DokerHub:
[Monitoring] Docker hub web login is working again. We will continue to monitor the situation.

More information here: https://status.docker.com/pages/incident/533c6539221ae15e3f000031/5da5fc78c7b1c11cd8603721Oct 15, 19:43 UTC
Investigating - Docker Hub has reported an ongoing incident

This could affect your Codefresh builds.

More information here:
https://status.docker.com/pages/incident/533c6539221ae15e3f000031/5da5fc78c7b1c11cd8603721

Docker Hub has reported an ongoing incident
Oct 15, 22:32 UTC
Resolved - From DockerHub:

All Systems OperationalUpdated a few seconds ago
Our system status page is a real-time view of the performance and uptime of Docker products and services.

More information here:
https://status.docker.comOct 15, 20:59 UTC
Monitoring - From DokerHub:
[Monitoring] Docker hub web login is working again. We will continue to monitor the situation.

More information here:...

Docker Hub has reported an ongoing incident
Oct 16, 15:41 UTC
Investigating - This could affect your Codefresh builds.

More information here:
https://status.docker.com/pages/incident/533c6539221ae15e3f000031/5da5fc78c7b1c11cd8603721

codefresh users, do you use a single github CI user token to add triggers on all pipelines? have you found a more secure way of doing this that is still maintainable

One thing to consider is introducing a gitops style workflow for adding pipelines.

Basically, make everyone a 'user' in codefresh with no specific permissions

cerate the github CI integration with trigger admin permissions

then use PR review/approval process to rollout the pipelines using the codefresh cli

Anyone set up Codefresh with GSuite SAML custom app for SSO? I did so a couple days ago and can't get it to work, and it's been 2 days with no response from support =/ . I followed their docs, and when I try to sync team, I get a BAD REQUEST response code. Just want to make sure I lined up the right properties in the right places.

looking for help in escaping yaml... need to run ~this in Codefresh pipeline:

commands:
      - kubectl run --env ...

and the env variable is:
LOGIN='{"randomDelay": 120000, "pauseAfter": 90000}'

how do I escape that to be valid...?

Use a block scalar with >- perhaps? https://yaml-multiline.info/

something like:

commands:
  - >-
    kubectl run --env LOGIN='{"randomDelay": 120000, "pauseAfter": 90000}'
    more command that will be folded in to a single line above
  - next new command

No clue if that'll work, but I'm pretty sure you can use : in block scalars since they are terminated by indentation

thanks, I think that may work!

has anyone tried to dynamically pass in build args in the build step (in a shared pipeline setting where you could have multiple triggers from different services)

...as in Docker build args?

yeah

prob not possible using their builtin build step huh 😅

Not likely...

I'm confused. Why would you not be able to do that? Take the "passed in" stuff, have a step that does some logic and does a cf_export on the desired build args, then in the build step use the ${{}} to snag those exported environment variables. Doesn't cf_export make the variables available to the pipeline and all future steps? Granted, I may not be understanding the initial problem.

Docker Build args have a special section

If there’s a deterministic number of args, each one could be defined and mapped to an env

But if the list I’d build args vary uncontrollably not sure how to do it

Yeah, if you want a different number of build args, that might not be possible with the build step's build args, but if you want the same build args with different values or whatever, then I think that's doable.

Docker Hub has reported an ongoing incident
Oct 17, 16:55 UTC
Resolved - Docker System Status
Current system status information:

All Systems OperationalOct 16, 15:41 UTC
Investigating - This could affect your Codefresh builds.

More information here:
https://status.docker.com/pages/incident/533c6539221ae15e3f000031/5da5fc78c7b1c11cd8603721

@Alex Siegman different number of build args and different arg names

I just have some very similar applications which i’ve wanted to share pipelines for because they build & deploy the same way

its just one builds w/ a couple build args and the others do not

i could add a conditional step (on the repo name) for that project, but at that point i think id prefer to just use a seperate pipeline

Our team has decided to bite the bullet and start using helm 3. In looking at the source code for the official helm step, it appears as though the helm version can be passed as an argument. Has anyone successfully done this or will I need to fork the step

Any way to exclude a file in the modified files expression in the triggers?

@Harrison Heck 👋 Glob expressions aren't great at excluding, but you could try !(filenamehere). Or, if you have conditions in your steps, you could run a check in one step to manually extract it and go from there.

Docker has reported an incident
Oct 18, 23:06 UTC
Identified - Docker has reported an incident.
It's affecting the following components:
- Docker Package Repositories

More information here:
- https://status.docker.com/
- https://status.docker.com/pages/incident/533c6539221ae15e3f000031/5daa35810d1eb20cb8de23ec

hi all, do you have any example of usage of codefresh to deploy on vm (not using packer, I have seen the example package-gcp) and/or aws beanstalk?

just another quick one: do you have any example of integration with monitoring system to measure quality, productivity and velocity?

Docker has reported an incident
Oct 19, 14:52 UTC
Resolved - This incident has been resolvedOct 18, 23:06 UTC
Identified - Docker has reported an incident.
It's affecting the following components:
- Docker Package Repositories

More information here:
- https://status.docker.com/
- https://status.docker.com/pages/incident/533c6539221ae15e3f000031/5daa35810d1eb20cb8de23ec

is the only way to not import every cluster in codefresh to every pipeline by being on enterprise codefresh?

Codefresh UI outage
Oct 22, 07:29 UTC
Update - We are continuing to investigate this issue.Oct 22, 07:28 UTC
Investigating - We are currently investigating this issue.

@btai every cluster is automatically available to all pipelines (even with free plans). You can just use kubectl straight in a pipeline and it will work. Could you please clarify?

@Kostis (Codefresh) yeah I know, my preference would be to not to have that by default because of security reasons. I’d prefer to explicitly allow access to specific clusters for each pipeline

Codefresh UI outage
Oct 22, 07:52 UTC
Resolved - This incident has been resolved.Oct 22, 07:29 UTC
Update - We are continuing to investigate this issue.Oct 22, 07:28 UTC
Investigating - We are currently investigating this issue.

Potential Webhook delivery issues with Github
Oct 22, 19:04 UTC
Investigating - Github reported an ongoing issue that is being investigated on their end with users experiencing difficulties with webhook delivery logs.

We are keeping a close eye on their updates and any potential disruption on the actual webhook delivery. Although at this point the report only refers to the logs on webhooks.

Potential Webhook delivery issues with Github
Oct 22, 21:28 UTC
Monitoring - GitHub updated the affected components:
- webhook delivery logs, attachments and git lfs.

Issues on lfs could affect your builds if your pipeline relies on git lfs.

More info here: https://www.githubstatus.com/incidents/fdl6khghfsp3Oct 22, 19:04 UTC
Investigating - Github reported an ongoing issue that is being investigated on their end with users experiencing difficulties with webhook delivery logs.

We are keeping a close eye on their updates and any...

Potential Webhook delivery issues with Github
Oct 23, 00:11 UTC
Resolved - Github has reported resolution of all of their webhook issuesOct 22, 21:28 UTC
Monitoring - GitHub updated the affected components:
- webhook delivery logs, attachments and git lfs.

Issues on lfs could affect your builds if your pipeline relies on git lfs.

More info here: https://www.githubstatus.com/incidents/fdl6khghfsp3Oct 22, 19:04 UTC
Investigating - Github reported an ongoing issue that is being investigated on their end with users experiencing...

Anyone know off hand where I find a Codefresh build definition example using Helmfile? Need to jump into this and would like to see an example to accelerate learning.

@dustinvb something like this...

  deploy_helmfile:
    title: Deploy with helmfile
    stage: Deploy
    image: "...."
    working_directory: /conf/
    environment:
      - 'KUBECONFIG=${{CF_KUBECONFIG_PATH}}'
      - 'REPO_ROOT=${{CF_VOLUME_PATH}}/${{CF_REPO_NAME}}'
    shell: bash
    commands:
      - kubectl config use-context "${KUBE_CONTEXT}"
      - helmfile  --environment="${FLAVOR}"
        -f "${REPO_ROOT}/deploy/helmfile.yaml" sync
    when:
      steps:
      - name: ask_for_permission
        on:
        - approved

this assumes there's a file in the repo called deploy/helmfile.yaml

this is a great way to deploy a microservice and all of it's dependencies

Cool, will wrap this up into a step and produce an image for this. Appreciate the example.

Great thanks for putting that together

@mumoshu ☝️ they are adding an official helmfile step to Codefresh :-)

Hi - I'd like to trigger a pipeline by codefresh API, with a specific branch and variable set.
The api doc https://g.codefresh.io/api/#operation/pipelines-run doesnt have an example of the acceptable body to POST - anyone got any examples?