#helmfile - December 2018 | Slack Archive

121 messages

Questions and discussion around helmfile https://github.com/roboll/helmfile and https://github.com/cloudposse/helmfiles
Archive: https://archive.sweetops.com/helmfile/

Nov 2018

December 2018

Jan 2019

Darenover 7 years ago

Have you encountered issues with a chart that creates jobs? When running sync a 2nd time k8s errors with field is immutable

Shaneover 7 years ago

@Daren This would not be a issue with helmfile or helm. Certain fields are not mutable in kubernetes. What field are you trying to modify?

Darenover 7 years ago

Im changing a value which is used as an ENV by the job

Erik Osterman (Cloud Posse)over 7 years ago

so we're deploying jobs regularly with our charts - mostly to run db migrations

Erik Osterman (Cloud Posse)over 7 years ago

we've not run into that

Erik Osterman (Cloud Posse)over 7 years ago

fwiw, our envs are referring to values in configmaps and secrets

Darenover 7 years ago

hmm maybe thats the issue

Darenover 7 years ago(edited)

Im doing it directly from values

Erik Osterman (Cloud Posse)over 7 years ago

monochart

Erik Osterman (Cloud Posse)over 7 years ago

😃

Shaneover 7 years ago

@Daren If you are creating a job directly via helm you likely should have it as part of a lifecycle hook.

Shaneover 7 years ago

Jobs are not mutable and once it's create it should run and then complete

Darenover 7 years ago

Hm even when using a configmap it still fails on immutable

Darenover 7 years ago

I thought of doing init-container originally, but its tricky for this situation. Im deploying an official chart which requires that a couple scripts be run against the DB first. I wanted to avoid having to build a container just for that

Darenover 7 years ago

So I tried using a chart that runs a couple jobs using the official image.

erikover 7 years ago

Erik Osterman (Cloud Posse)over 7 years ago

Here's a job being deployed several times a day that runs migrations

Darenover 7 years ago

Heres mine, its simple:

apiVersion: batch/v1
kind: Job
metadata:
  name: {{ template "harbor-init.fullname" . }}-registry
  labels:
    <http://app.kubernetes.io/name|app.kubernetes.io/name>: {{ include "harbor-init.name" . }}
    <http://helm.sh/chart|helm.sh/chart>: {{ include "harbor-init.chart" . }}
    <http://app.kubernetes.io/instance|app.kubernetes.io/instance>: {{ .Release.Name }}
    <http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: {{ .Release.Service }}
{{- if .Values.podAnnotations }}
  annotations:
{{ toYaml .Values.podAnnotations | indent 4 }}
{{- end }}
spec:
  template:
    metadata:
      labels:
        app: {{ template "harbor-init.fullname" . }}-registry
        release: "{{ .Release.Name }}"
    spec:
      restartPolicy: OnFailure
      containers:
        - name: registry
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          envFrom:
            - configMapRef:
               name: {{ include "harbor-init.fullname" . }}
          env:
            - name: PGPASSWORD
              valueFrom:
                secretKeyRef:
                  name: {{ include "harbor-init.fullname" . }}
                  key: password
          command: ["psql"]
          args:
            - "-f"
            - "/docker-entrypoint-initdb.d/initial-registry.sql"

Darenover 7 years ago

Im not doing the checksum annotations

Darenover 7 years ago

Does it matter?

Shaneover 7 years ago

Nothing in a job's spec an change. If you want to do this on install I would recommend using a helm lifecycle hook for post-install

Shaneover 7 years ago

with your current job if the image or tag changes it will cause helm to fail

Darenover 7 years ago

ohhh a helm hook, nice

Darenover 7 years ago

nginx uses one: https://github.com/helm/helm/blob/master/docs/examples/nginx/templates/post-install-job.yaml#L17-L19

erikover 7 years ago

Shaneover 7 years ago

Erik Osterman (Cloud Posse)over 7 years ago

thanks @Shane - i didn't know why @Igor Rodionov was doing this

Erik Osterman (Cloud Posse)over 7 years ago

but i guess that's why

Igor Rodionovover 7 years ago

@Igor Rodionov has joined the channel

Shaneover 7 years ago

The hooks allow you to delete the job so when it runs again it can run the job again. if you don't delete the job it will fail if the job ever changes

Shaneover 7 years ago

Due to the immutability

Darenover 7 years ago

someone get this man a 🍺

Erik Osterman (Cloud Posse)over 7 years ago

🍻

Shaneover 7 years ago

I always accept beer as payment

Darenover 7 years ago

If you only want the job to run once, do you need the delete hook?

Darenover 7 years ago

do not run on any subsequent sync

Erik Osterman (Cloud Posse)over 7 years ago

then i thikn you want

Erik Osterman (Cloud Posse)over 7 years ago


"<http://helm.sh/hook|helm.sh/hook>": pre-install

Erik Osterman (Cloud Posse)over 7 years ago

or basically, something to indicate that it shouldn't get triggered on upgrades

Erik Osterman (Cloud Posse)over 7 years ago

i don't know the hooks off the top of my head

Darenover 7 years ago

👍️

Darenover 7 years ago

https://docs.helm.sh/developing_charts/#the-available-hooks

Igor Rodionovover 7 years ago

@Daren you can find more info here https://docs.helm.sh/developing_charts/#hooks

Igor Rodionovover 7 years ago

oh.. you found already

Darenover 7 years ago

Is there a way to bundle some resources into an official chart?

Igor Rodionovover 7 years ago

what chart?

Igor Rodionovover 7 years ago

monochart ?

Darenover 7 years ago

In this case its https://github.com/goharbor/harbor-helm

Darenover 7 years ago

I mean does helmfile allow you to attach another resource to a chart you are installing

Erik Osterman (Cloud Posse)over 7 years ago

helmfile is basically a Makefile for helm 😃

Erik Osterman (Cloud Posse)over 7 years ago

so if you can do it with helm, you can do it with helmfile

Erik Osterman (Cloud Posse)over 7 years ago

now, i know i'm going to sound like a broken record - but...

Erik Osterman (Cloud Posse)over 7 years ago

we're doing something similar. let me explain.

Erik Osterman (Cloud Posse)over 7 years ago

so we install granfa from official chart repos

Erik Osterman (Cloud Posse)over 7 years ago

but then we need some additional configmaps

Erik Osterman (Cloud Posse)over 7 years ago

in our helmfile we have one release for grafana and another release for the configmaps which get installed using our monochart

Igor Rodionovover 7 years ago

you can not inject jobs into existing chart.
We have workaround of that with helmfile and monorchat

Erik Osterman (Cloud Posse)over 7 years ago

we have an example - will show you

Darenover 7 years ago

Sounds like my use case. Im installing the Harbor chart. But I need to run a script against the DB it will use beforehand

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/cloudposse/helmfiles/blob/master/helmfile.d/0400.kube-prometheus.yaml#L262-L310

Igor Rodionovover 7 years ago

This is example how we "extend" official nginx chart with additional resources
https://github.com/cloudposse/helmfiles/blob/master/helmfile.d/0320.nginx-ingress.yaml#L156

Darenover 7 years ago(edited)

With both your examples, you are ordering the helm releases and using wait: true?

Erik Osterman (Cloud Posse)over 7 years ago

yea, concurrency 1

Erik Osterman (Cloud Posse)over 7 years ago

wait true

Igor Rodionovover 7 years ago

and release order playes role

Darenover 7 years ago

cool, we have the same approach

Igor Rodionovover 7 years ago

also I sugget to deploy it with 2 bash commands

Igor Rodionovover 7 years ago

helmfile --selector component=job
helmfile --selector component=harbor

Erik Osterman (Cloud Posse)over 7 years ago(edited)

well, or have a helmfile.yaml which includes the helmfiles in the right order

Darenover 7 years ago

My ordering is less sensitive

Darenover 7 years ago(edited)

If the pre-job runs at the same time or after its fine. Harbor will eventually recover once it does run

Igor Rodionovover 7 years ago

then helmfile fits perfect

Darenover 7 years ago

Agreed

Darenover 7 years ago

@Erik Osterman (Cloud Posse) check out harbor sometime, its working out very well

Erik Osterman (Cloud Posse)over 7 years ago

that's great to hear

Erik Osterman (Cloud Posse)over 7 years ago

i want to add more security/compliance features to our solution so it sounds like something good to have there.

Erik Osterman (Cloud Posse)over 7 years ago

did you go enterprise?

Darenover 7 years ago

Its open source...

Darenover 7 years ago

Docker registry & Chart museum: https://goharbor.io/

Darenover 7 years ago

backed by CNCF

Erik Osterman (Cloud Posse)over 7 years ago

yea, my understanding was though to get access to a larger library of vulns, you needed to go ent

Erik Osterman (Cloud Posse)over 7 years ago

though I met those guys ~2 years ago at a meetup and a lot might have changed

Darenover 7 years ago

I was not aware of an enterprise version

Darenover 7 years ago

It uses clair (https://github.com/coreos/clair) for vuln.

Erik Osterman (Cloud Posse)over 7 years ago

hah, my fault

Erik Osterman (Cloud Posse)over 7 years ago

https://anchore.com/

Erik Osterman (Cloud Posse)over 7 years ago

all these nautical product names mixed me up

Erik Osterman (Cloud Posse)over 7 years ago

this is an open core + enterprise

Erik Osterman (Cloud Posse)over 7 years ago

don't know how it compares to harbor vs clair

Erik Osterman (Cloud Posse)over 7 years ago

isn't harbor providing redundant capabilities to twistlock which you're already using?

joshmyersover 7 years ago

https://github.com/future-architect/vuls is pretty nice for a lightweight vuln scanner

Darenover 7 years ago

Ive added a chart repository that requires auth + mfa to helm. However every helm operation involving the repo (install, repo update, push) trigger an MFA verification. Is there a way for helm to cache a session?

Erik Osterman (Cloud Posse)over 7 years ago

helm-s3

Erik Osterman (Cloud Posse)over 7 years ago

or MFA with harbor?

Darenover 7 years ago

In this harbor

Darenover 7 years ago

Im wondering if its because harbor uses LDAP for auth

Erik Osterman (Cloud Posse)over 7 years ago

oh, yea, in that case no clue.

Erik Osterman (Cloud Posse)over 7 years ago

@jdolitsky probably could answer it though

jdolitskyover 7 years ago

@jdolitsky has joined the channel

Erik Osterman (Cloud Posse)about 7 years ago

@Shane what do you think about this? https://github.com/roboll/helmfile/issues/347

Erik Osterman (Cloud Posse)about 7 years ago

our use-case is we write a lot of helmfiles for our clients, but i'd like to reuse them while version pinning

Erik Osterman (Cloud Posse)about 7 years ago

https://github.com/cloudposse/helmfiles/tree/master/helmfile.d

Shaneabout 7 years ago

got ya, one issue I would see with it is helmfile is not really self contained.

Erik Osterman (Cloud Posse)about 7 years ago

hrmm

Shaneabout 7 years ago

everything would have to be in the helmfile without any external file references.

Erik Osterman (Cloud Posse)about 7 years ago

yes - it would be limited in that regard

Erik Osterman (Cloud Posse)about 7 years ago

but those external file references in our case are usually the non-portable settings

Erik Osterman (Cloud Posse)about 7 years ago

things like URLs, configuration files, keys, etc

Shaneabout 7 years ago

got ya

Erik Osterman (Cloud Posse)about 7 years ago

https://github.com/cloudposse/statup/pull/2/files#diff-c9e5b83e3877738c2e3854641c0144c5

Erik Osterman (Cloud Posse)about 7 years ago

this is the most common use-case though for us

Erik Osterman (Cloud Posse)about 7 years ago

everytime we deploy an app for some staging environment we copy and paste this over

Shaneabout 7 years ago

I'll check it out I got to jump off the wife wants to eat.

Erik Osterman (Cloud Posse)about 7 years ago

haha, no rush.

Erik Osterman (Cloud Posse)about 7 years ago

basically i don't want to copy all these snippets everywhere which add a lot of technical debt.

Erik Osterman (Cloud Posse)about 7 years ago

(heading out myself! ttyl)

Darenabout 7 years ago

Anyone know how to remove a lifecycle hook from a resource and not get the helm state out of sync? This chart https://github.com/helm/charts/blob/master/stable/sumologic-fluentd/templates/secrets.yaml#L11 has a pre hook on the secret, which is not needed and causes issues. Removing the hook and performing a sync causes helm to error out thinking the secret does not exist.

Erik Osterman (Cloud Posse)about 7 years ago

hrm... can you create a stubsecret deployed as a standalone release using helmfile?

Erik Osterman (Cloud Posse)about 7 years ago

.... not sure if i full comprehend the problem

November 2018 Browse by date January 2019