#release-engineering - November 2018

@dustinvb just joined! he dreams in pipelines

Ah bah, no :superfresh: emoji.

haha

we can add that 😃

for those that don’t know dustin, he is an awesome support engineer at Codefresh, which is what we use for all of our CI/CD

hey @dustinvb

Hi Andriy.

Happy to help out in anyway I can.

thanks

we all love Codefresh 🙂

I do as well. Was a customer for a year before joining.

Howdy releasers

I wanted to discuss one topic that is spinning in my head for a while:
- Reusing docker or other artifacts from PRs or not

Basically, so speed up the build/release cycle, I was thinking on reusing the tested/built artifact in a PR when we deploy to the QA environment and ofcourse subsequently

Now, this is not an easy challenge as its "hard" to identify the built artifact

since we enforce "up to date branches" im testing using git rev-parse --short HEAD^2 on the merge|commit job to identify the artifact by the tag (we tag with the commit) of the branch that did the PR that was merged and created that commit.

thoughts? ideas?
what is your current CI workflow? What do you build on PR and on merge?

so our workflow is slightly different

we have “unlimited staging environments” which correspond to PRs - one per PR; those just build images pinned to 0.0.0-gitsha

entire environments or just images?

In a similar tone to my previous question, anyone has a working workflow for using terraform plan from PR build or similar?
Because while its the recommended way for terraform, I have yet to see a working workflow for it

@Gonzalo Peci if you are asking about how to trigger terraform plan/apply from an open PR, we recently used atlantis for a GitOps workflow. See #atlantis channel. Also @Erik Osterman (Cloud Posse) recently held a meetup during #connectweek in Pasadena (CA) where he gave a live demo using Atlantis with Terraform to provision AWS user accounts using only Pull Requests

No, not exactly like that, im not a big fan of what atlantis does

It applies before merge, to me that is anti-pattern

Except for terraform plans are poor

I was thinking appling the plan generated on PR, on the merge

They are optimistic at best

So if you merge and apply now what is in git is not deployed

If others are developing against it, they are just as blocked

So what we’ve reconciled is auto merge on successful apply

Well, but that is the same case for a CD container release

But for container releases it’s more stable

You have more under your control

Less value based errors

Most terraform failures in my experience are due to bad values

Containers get their values at runtime

Not at compile time (not generally but ideally)

You can achieve what you want easily with Codefresh.

Yeah, that is true, and a fair point, but not this one:

So if you merge and apply now what is in git is not deployed

If others are developing against it, they are just as blocked

In theory in the container you have the same "posibility" of failure

but I agree, is not as likely, as easiert to caught on test

TF has a lot of sideeffect failure scenarios, that plan does not catch

To me atlantis is a practical approach

Not the theoretical ideal

So rather than clutter the master commit history with a bunch of patch releases, rather get it in clean. We preserve a full transcript in the git comments so we have a record of what is deployed.

Even if the PR is closed but half applied, there’s a record of that’s

Yeah, its not a bad approach and im not saying Atlantis sucks, much the contrary, even Hashi bought them

I just feel that is a "hack" around a bigger issue

a great hack, but a hack anyway

Agreed

In my previous company we had a rake looper to get around TF Modules not having count, it worked, and it saved us a lot of time, but in "terraform" it was a hack

I just don’t see a way around it that’s practical to solve (unless your HashiCorp and have 100mil in fresh green)

Yeah true

I might give it 2nd chance, at least it will avoid those fix PRs for a stupid terraform value problem

BTW, its great to have a place to ping-pong this ideas

Yea totally!! That what this place is for

Yep, thanks 🤘 !

Run a plan, push an artefact, namely the plan output. Run an apply of said build artefact number. This could be a gitsha, PR number, Jenkins build number. I don't see a huge advantage of Atlantis... although I caveat with not having used so opinions maybe wrong. Have been doing CI/CD of terraform plan + apply for a long time in Jenkins. Not a massive fan of having output in the PR for history. Your git history lives forever, github PRs may not. Running pre merge isnt ideal for team scenario's. Have seen a lot of failed apply due to vars, state, lots of TF bugs, race conditions...

Things like tflint can help with some of these

E.g. checking a type of AMI is even available in a region

Won't catch bugs though.

Is Atlantis just like serverless Jenkins ci/cd for terraform with output posted back to the PR?

Sounds nice if you don't already have a CI/CD solution/don't need one for other things?

ok

anyone here use github actions yet? do you know if there is a way to cache dependencies between builds?

haven’t seen anything in the docs

no one @cloudposse has been invited to the beta

i reached out to some peeps at GitHub but we don't have the klout =0

@Gabe are you in the beta?

yeah we just got accepted in

just checked my inbox

nothing yet

i don’t think i got an email… just saw the new actions button on our repo

hmm yeah… it looks pretty cool so far but a few things i’ve noticed is not being able to cache dependencies between builds, no control over the size of the machine it runs on (1 cpu 3.75 gb), and only two concurrent workflows running at one time per repo

pros are that it seems simpler than circle/jenkins and you can create actions that take environment variables so it’s easier to reuse/share actions between repos

… and we just got the email saying they have enabled it for us 😆

i just got the same email a few minutes ago, and Actions on my personal GitHub account 🙂

I'm not so lucky. Still waiting

@Andriy Knysh (Cloud Posse) this screenshot from https://github.com/cloudposse/github-status-updater how do you release to a namespace?

every PR is a new k8s namespace. It’s how we do unlimited staging environments

see @Erik Osterman (Cloud Posse) presentation https://cloudposse.com/devops/unlimited-staging-environments/

@davidvasandani if you have questions or need more info, we can provide it

@Andriy Knysh (Cloud Posse) do you happen to have an example where all these pieces are glued together?

give me a few minutes

re: Self-hosted Helm Chart Registry - Codefresh added Managed Helm Repositories after the presentation, so we use it now instead of deploying our own chart museum

@davidvasandani ^