#helmfile - January 2020 | Slack Archive

199 messages

Questions and discussion around helmfile https://github.com/roboll/helmfile and https://github.com/cloudposse/helmfiles
Archive: https://archive.sweetops.com/helmfile/

Dec 2019

January 2020

Feb 2020

Zachary Loeberabout 6 years ago

So, how are we all handling the helm 3 upgrades without automatic namespace creation anyway?

Zachary Loeberabout 6 years ago

updating all helmfiles to include a presync like this?

Zachary Loeberabout 6 years ago

- events: ["presync"]
showlogs: true
command: "/bin/sh"
args:
- "-c"
- >-
kubectl get namespace "{{{{ .Release.Namespace }}}}" >/dev/null 2>&1 || kubectl create namespace "{{{{ .Release.Namespace }}}}";

Erik Osterman (Cloud Posse)about 6 years ago

See https://github.com/roboll/helmfile/issues/891

Erik Osterman (Cloud Posse)about 6 years ago

Best option I think is to use raw chart

Roderik van der Veerabout 6 years ago

I’ve had trouble with the raw chart, multiple applies fail because the namespace already exists

Zachary Loeberabout 6 years ago

https://github.com/zloeber/helm-namespace

Zachary Loeberabout 6 years ago

I decided to just whip up a point solution for the hell of it.

Zachary Loeberabout 6 years ago

Slack robocop told me not to swear. It took me a good long moment to realize I had done so... sorry I guess 🙂

Erik Osterman (Cloud Posse)about 6 years ago

lol, yes, it's a little bit strict 😉

Zachary Loeberabout 6 years ago

The helm chart I put together is so simple its not even worth publishing but hey, it does allow one to at least change the helm resource policy from 'keep' to whatever else it needs to be to allow for redeployments (if you need to do that)

Zachary Loeberabout 6 years ago

I was going to do v2 for helm3 but it should work for both helm 2 and 3 I think so I left it at v1

Zachary Loeberabout 6 years ago

@Erik Osterman (Cloud Posse), you ever not working/geeking out? 🙂

Erik Osterman (Cloud Posse)about 6 years ago

haha, not enough...

Erik Osterman (Cloud Posse)about 6 years ago

Btw, try this? https://github.com/thomastaylor312/helm-namespace

Zachary Loeberabout 6 years ago

I did, its in the notes for alternative options.

Zachary Loeberabout 6 years ago

It works fine if you are all helm 3 and willing to change your base helm commands. Honestly, its probably a better solution in general

Erik Osterman (Cloud Posse)about 6 years ago

https://github.com/zloeber/helm-namespace#alternative-1---helm-namespace

Erik Osterman (Cloud Posse)about 6 years ago

😃

Zachary Loeberabout 6 years ago

read somewhere that 3.1 will add namespace creation back anyway, so its likely a moot point 🙂

Erik Osterman (Cloud Posse)about 6 years ago

Yea, not worth investing in

Zachary Loeberabout 6 years ago

I'm looking at a pretty large stack of helm 2 charts all deployed with tillerless and helmfile that have gobs and gobs of secrets polluting the tiller namespace that makes me itchy to move to helm 3

Zachary Loeberabout 6 years ago

too many clusters for a single devops guy to look to migrate ATM so I'm using both helm3 and 2 in the same clusters like a fool

Zachary Loeberabout 6 years ago

anyone using helmfile in a gitops style deployment?

Zachary Loeberabout 6 years ago

with flux or argocd or something?

erik-stephensabout 6 years ago

Would like to automate our helmfile-centric workflow a bit more. Developing an Operator to handle the watching as well as the bits that helm/helmfile not able to perform. Would still like to leverage our helmfile effort, at least initially to quickly prototype. Anyone else go on a similar adventure? Recommendations on Operator sdk/framework (I'm currently looking at Metacontroller)?

erik-stephensabout 6 years ago

Looks like operator-sdk new --type=helm ... is designed to address this use case.

Erik Osterman (Cloud Posse)about 6 years ago(edited)

@erik-stephens Have you seen the Helmfile operator by @mumoshu ?

Erik Osterman (Cloud Posse)about 6 years ago

https://github.com/mumoshu/helmfile-operator

erik-stephensabout 6 years ago

I have not, but it's on the short list of things to evaluate. Thanks! 🙇

Zachary Loeberabout 6 years ago

I've looked at it but was unable to get it to work at the time

deftunixabout 6 years ago

@Erik Osterman (Cloud Posse) hello, do you have any example of usage helmfile to patch chart service spec without touch the chart?

Erik Osterman (Cloud Posse)about 6 years ago

You can't "monkeypatch" with helmfile as it just wraps helm

Erik Osterman (Cloud Posse)about 6 years ago

So if helm provided someway to do that, then helmfile could.

Erik Osterman (Cloud Posse)about 6 years ago

@deftunix - describe instead what you want to accomplish, and perhaps we can think of a way to do it.

deftunixabout 6 years ago

I want just add some rule in the ingress controller without change the chart to support it

Erik Osterman (Cloud Posse)about 6 years ago(edited)

And by changing the rule, you hope to accomplish what? ...what is the business objective

deftunixabout 6 years ago

Ssl redirect with alb ingress

deftunixabout 6 years ago

He needs a rule and an annotation

deftunixabout 6 years ago

It needs sorry

Erik Osterman (Cloud Posse)about 6 years ago

Does the chart support disabling the ingress?

deftunixabout 6 years ago

But the chart doesn’t support it

deftunixabout 6 years ago

Yes

deftunixabout 6 years ago

The chart support the ingress disabling

Erik Osterman (Cloud Posse)about 6 years ago

Perfect. Then you can use Helmfile

deftunixabout 6 years ago

I will disable and just add a kustomize or manifest?

Erik Osterman (Cloud Posse)about 6 years ago

Disable the ingress. Then define a new one using the raw chart and Helmfile

Erik Osterman (Cloud Posse)about 6 years ago

We have used this pattern in the past

deftunixabout 6 years ago

Do you have same repo?

Erik Osterman (Cloud Posse)about 6 years ago

You mean example?

deftunixabout 6 years ago

Yes

Erik Osterman (Cloud Posse)about 6 years ago

I couldn’t point you directly (on my phone), but you have seen our massive repo?

Erik Osterman (Cloud Posse)about 6 years ago

https://github.com/cloudposse/helmfiles

Erik Osterman (Cloud Posse)about 6 years ago

Go to the releases folder

deftunixabout 6 years ago

Thanks

Erik Osterman (Cloud Posse)about 6 years ago

Maybe search for the ingress keyword or raw

deftunixabout 6 years ago

Maheshabout 6 years ago(edited)

Is there is any tool like kube-applier to manage deployments for helm?

TBeijenabout 6 years ago

2 questions:
• What are the intended use cases for helmfile apply vs helmfile sync. If I understand correctly syncwould also remove when installed: false. Otoh, apply has the diff output which is nice feedback.
• Shouldn't helmfile have a cool logo? 😉 (Running into that every time I need to create presentations for team or business)

Zachary Loeberabout 6 years ago

Anyone out there setup the operator lifecycle manager as a helmfile?

Zachary Loeberabout 6 years ago

heck, anyone deploy/use it in any way?

deftunixabout 6 years ago

hi all, quick question about helmfile. I would like to apply a gotmpl kubernetes manifeest during the helmfile release apply phase. do you have any idea?

deftunixabout 6 years ago

I need to render the template and then apply it

Zachary Loeberabout 6 years ago

maybe use the raw chart with a values.yaml.gotmpl file?

Zachary Loeberabout 6 years ago

or delve into kustomize (something I've yet to do honestly)

deftunixabout 6 years ago

@Zachary Loeber I want just patch the ingress controller of a community helm chart without change it

deftunixabout 6 years ago

from the helmfile run

Zachary Loeberabout 6 years ago

I guess that's what you use helmfile for in general if I'm reading your statement correctly.

Zachary Loeberabout 6 years ago

I don't know about others, but I tend to completely disable ingress on all public charts then float ingress up to a custom chart of my own so I can more quickly make lateral ingress moves if so required.

Zachary Loeberabout 6 years ago

That way you aren't trying to patch 20 different ingress chart implementations which may or may not be charted out the same.

Zachary Loeberabout 6 years ago

You could do such a thing without a custom chart as well I suppose. again, maybe use the incubator/raw chart with a values.yaml.gotmpl file

Zachary Loeberabout 6 years ago

other (smarter) people on this channel may know better ways though

deftunixabout 6 years ago

yes, I fix in exactly this way

deftunixabout 6 years ago

I create a custom chart

deftunixabout 6 years ago

for the ingress

Scottabout 6 years ago

I'm trying to use ref+vault integration for secrets and I'm getting vault: get string: key "foo" does not exist in secrets foo yet it does

Scottabout 6 years ago

Has anyone used the vault integration successfully? I assume people have

Scottabout 6 years ago

I figured it out :)

Jonathanabout 6 years ago

This might not be the right place for this question, if not, sorry! I'm trying to deploy grafana, and import dashboards from a git repo. The issue is that the repository I want to import from is private, so even though the dashboards.default.local-dashboard.url is correct, I can't reach it for obvious reasons, but I cannot find anywhere in the documentation how to pass secrets/username+pass etc. to authorize myself so I can read it. Does anyone have any pointers?

Chris Maxoutisabout 6 years ago(edited)

username:password@dashboards.default.local-dashboard.url where user and password you can pass them usually from env vars or even vault.

{{ requiredEnv PASSWORD" }}

Jonathanabout 6 years ago

I'll try that out, thanks a bunch!

Matt McLaneabout 6 years ago

Morning helmfile folks.

Matt McLaneabout 6 years ago

I have been struggling the past could days on a problem related to helm file and maybe someone in here can help. Basically I am trying to get weave flux to use helmfile instead of going straight to helm. Here is what I have done so far. I have setup flux to use manifest generation which allows me to run helmfile. Using that I can actually get helmfile to run and build things, but that isn't a very good use of flux. Basically I am just using flux to clone git. What would be better is if I could get helmfile to write to stdout like kustomize build does. I am told by the folks that develop flux that it should work. It would allow us to benefit from the templating and secrets of helmfile while getting the gitops benefits of flux. Plus we could possibly use helmfile on our local systems if we wanted to.

So the question is who do I get helmfile to output to stdout like kustomize?
I am currently running the following thinking it might do the trick, but it doesn't seem to do anything in flux:

helmfile -e dev -q -f ./helmfile.d/helmfile.yaml build

Any help or suggestions would be great!

Erik Osterman (Cloud Posse)about 6 years ago

@Matt McLane have you seen the helmfile operator?

Erik Osterman (Cloud Posse)about 6 years ago

https://github.com/mumoshu/helmfile-operator

Erik Osterman (Cloud Posse)about 6 years ago

I think that might make it a simpler integration with weave flux because you can just use CRDs

Zachary Loeberabout 6 years ago

Have you used that operator yet? I've not been able to get it to compile

Matt McLaneabout 6 years ago

That is what I was worried about. it didn't look complete to me.

Matt McLaneabout 6 years ago

But I am willing to be wrong.

Erik Osterman (Cloud Posse)about 6 years ago

@mumoshu ☝️

Balaji Jabout 6 years ago

hi , is there predefined helmfile for redis native cluster(not using sentinel)? or anyone worked on creating one. Please let me know

Zachary Loeberabout 6 years ago

I could create one in about 3 minutes based on the default redis chart. Why don't you give it a whirl first as you will almost always need to customize whatever anyone else precreated anyway.

Matt McLaneabout 6 years ago

So what is the "industry standard" for pipelines to run helmfile? We are trying to move toward a gitops approach, which is why I have been looking at flux so much. We also like some of the functionality helmfile brings us. We could build something from scratch but I much rather be in line with what others are doing.

Zachary Loeberabout 6 years ago

@Matt McLane Good question, I've been trying to figure out the same. I'm looking into ArgoCD for this because it seems to have easier plugin capabilities to support helmfile. But it also seems that argo 'apps' are synonomous with helmfiles (generically) https://github.com/argoproj/argo-cd/issues/2143.

Erik Osterman (Cloud Posse)about 6 years ago

@Matt McLane we are using atlantis; i would not argue it's an industry standard, but it works well enough.

Erik Osterman (Cloud Posse)about 6 years ago

runatlantis.io

Erik Osterman (Cloud Posse)about 6 years ago

atlantis lets one define custom workflows with a plan and apply phase which we map to diff and apply in helmfile

James Huffmanabout 6 years ago

hello, i've been trying to do something that probably shouldn't be difficult but i'm struggling.
i want to do an {{ if }} block in my helmfile which checks for the existence of a file. ultimately, i only want a particular release to be deployed if a specific file exists locally. is there a standard way of doing that? (i have seen nothing of the sort in the docs/examples and have fought with my own approaches for a while now)

Erik Osterman (Cloud Posse)about 6 years ago

Hrmmmm that should be possible if Sprig supports a function for that

Erik Osterman (Cloud Posse)about 6 years ago

http://masterminds.github.io/sprig/paths.html

Erik Osterman (Cloud Posse)about 6 years ago

I don't see a function for that

Erik Osterman (Cloud Posse)about 6 years ago

@James Huffman what underlying business logic are you trying to implement? maybe there's an alternative way that doesn't depend on the existence of files.

Erik Osterman (Cloud Posse)about 6 years ago

https://github.com/roboll/helmfile/issues/766

Erik Osterman (Cloud Posse)about 6 years ago

the issue above has a workaround you can use for now

Erik Osterman (Cloud Posse)about 6 years ago


{{ if eq (exec "./fileexists.sh" (list $valueFile)) "true" }}

James Huffmanabout 6 years ago

OK, that's what i was wondering, if i needed to write a shell script to do it instead. thank you!!

vgdubabout 6 years ago

Hi folks, I am just wondering if anyone tried to add roles/clusterroles and their bindings via helm file , I am looking to apply some security polcies for multiple namespaces via helm ? is this possible ?

Zachary Loeberabout 6 years ago

try rbac-manager along with some incubator/raw charts

Zachary Loeberabout 6 years ago

that's what I've been using and it works well enough

vgdubabout 6 years ago(edited)

@Zachary Loeber looks good , I am wondering in incubator/raw under resources and templates I can specify both role/clusterrole and bindings

vgdubabout 6 years ago

my goal is to be able to specify/create only get, list,watch policies in namespaces

Zachary Loeberabout 6 years ago(edited)

- name: inv-ingest-rbac
  chart: incubator/raw
  namespace: inv-ingest
{{- if eq (env "HELM_VERSION" | default "2") "3" }}
  needs:
  - kube-system/namespace-inv-ingest
{{- end }}
  values:
  - resources:
    - kind: ClusterRole
      apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
      metadata:
        name: inv-ingest-cluster-role
        labels:
          <http://app.kubernetes.io/name|app.kubernetes.io/name>: inv-ingest
      rules:
      - apiGroups: [""]
        resources: ["pods", "services", "configmaps"]
        verbs: ["get", "list", "watch", "create", "delete", "update", "patch"]
      - apiGroups: [""]
        resources: ["secrets"]
        verbs: ["get", "watch", "list"]
    - kind: ClusterRoleBinding
      apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
      metadata:
        name: inv-ingest-role-binding
        labels:
          <http://app.kubernetes.io/name|app.kubernetes.io/name>: inv-ingest
      roleRef:
        apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
        kind: ClusterRole
        name: inv-ingest-cluster-role
      subjects:
      - name: default
        namespace: inv-ingest
        kind: ServiceAccount

Zachary Loeberabout 6 years ago

That's a fairly full example of a helmfile chart for a spark application that uses the default service account and requires more rights than I'm comfortable with

Zachary Loeberabout 6 years ago

you can pare it back to just what you'd need I'd think.

Erik Osterman (Cloud Posse)about 6 years ago

@Zachary Loeber you're doing some interesting things over there. would love a demo sometime.

Zachary Loeberabout 6 years ago

It all feels like hacks layered on hacks to me 🙂

Erik Osterman (Cloud Posse)about 6 years ago(edited)

Haha that’s the reality though... it’s why I hate these demo videos that deploy hello word apps and proclaim that victory! “Deployments made easy”

Erik Osterman (Cloud Posse)about 6 years ago

The reality is that it’s difficult. Especially when you don’t control the tool chain from top to bottom. Integration is all about hacking bits and pieces together.

DanBabout 6 years ago

I'm digging in to helmfile for the first time and trying to do something I think should be straight forward. I want to set a value that can be used "globally" in chart value files (via gotmpl). I'm not sure where to put this. If I put it in my main helmfile.yaml I get an error `line 4: field foo not found in type state.HelmState, if I put it in a file that is listed under bases, I get an error line 1: field foo not found in type state.HelmState so where does it go?

DanBabout 6 years ago

basically I want to use {{ .Value.foo }} in several chart values.yaml.gotmpl files

DanBabout 6 years ago

This is what is in my defaults.yaml base file foo: '{{ coalesce (env "FOO") (env "LOCAL_FOO") }}'

DanBabout 6 years ago(edited)

Interested in hearing the communities thoughts on the above puzzles ^^

Zachary Loeberabout 6 years ago

@DanB wouldn't you use an environment variable for that?

DanBabout 6 years ago

that is what i am trying to do

DanBabout 6 years ago

but set it once to reduce duplication

James Huffmanabout 6 years ago

so what you want is for an environment variable to be assigned to a key in your helmfile but propagate up into the base chart?

DanBabout 6 years ago

Yes but multiple charts, not just one chart

James Huffmanabout 6 years ago

so our solution to this, which may or may not work for you, is to make a values file section called global in each of our charts.
within global are all of the common values of this nature we'd like to set, initially with placeholders (since they will get overwritten).
then we have a macro file which handles setting all of the fields to their real values at run time, using env calls to collect them.
in each helmfile, we include this macro file as another values file:

values:
      - macros/deploy.yaml.gotmpl

James Huffmanabout 6 years ago

having to put the placeholders into each chart is the most annoying bit, but helm template helpers might let you avoid that

DanBabout 6 years ago

Ah hmm. So in our case the value is the same in each chart but each chart may use a different key in their values. I guess wed need the global section to duplicate the value for each possible key?

James Huffmanabout 6 years ago(edited)

to be fair, you only have to put in each chart the specific keys you care about for that chart. any keys in the macro file which don't exist in the underlying chart just won't do anything.

DanBabout 6 years ago

To put what I want in another way: I want to keep all usage of env vars out of individual chart values. I want to centralize all use of env vars to one place (environment? Base? Whatever works). Pain point I am trying to avoid is if an env var name changes or we introduce a new override env var we only have to change it in one place instead of n where n = # of charts. In our case n is 50ish

James Huffmanabout 6 years ago

you could make a single .yaml.gotmpl file containing a single parent key, under which are all of the keys you want and how to obtain their values (e.g. through env calls), then add that file to the values: array for each helmfile you're using. to make it fully work you would then update your charts to pull in that whole section. this would be a one-time deal so any time you updated this master file, all charts would see it when they render.

James Huffmanabout 6 years ago(edited)

imagine you called the section in your master values file global: and put all your keys below it. you can grab them in each of your charts with $root.Values.global i believe.

DanBabout 6 years ago

hmm, some of these charts I do not control

DanBabout 6 years ago

To me this seems like it'd be a common use case, I wonder if I am missing something or over complicating things

James Huffmanabout 6 years ago

it's mostly a consequence of how helm works. a particular value is only meaningful if it lands in the correct place within a chart. so with different charts from different sources, some of which you cannot easily modify, there's no one-size-fits-all solution. everybody writes charts their own way

DanBabout 6 years ago(edited)

ignoring env vars, is there a way for me to set a static global value I can reuse in individual chart .yaml.gotmpl via mykey: {{ .Value.someStaticGlobal }}

DanBabout 6 years ago(edited)

to give a concrete example my charts want to know the "name" of the cluster in the values file. When I run helmfile I set an envvar CLUSTER_NAME=k8s-dev-01 now I want to set some variable once in one location and then use that value in various chart values, trick is chart A may expect cluster name in a variable named clusterName chart B might define it as cluster chart C might define it as cluster-name etc

DanBabout 6 years ago

i can sprinkle the env `CLUSTER_NAME through my chart .yaml.gotmpls, but I'd really like to avoid that

James Huffmanabout 6 years ago

only way to handle that is chart by chart, unfortunately. you can't do a generic solution since the charts themselves vary so much. we've run into the same thing

DanBabout 6 years ago

ah ok, really appreciate your thoughts and time!

DanBabout 6 years ago

Oh I think I figured it out: this was the key: https://github.com/roboll/helmfile/issues/640

DanBabout 6 years ago

In my helmfile I specify values: that set values based on envvars. I simply use {{ .Values.key }} in my chart value gotmpl file. I can override these as well in my environments.yaml.gotmpl which is set as bases: in my helmfile

Erik Osterman (Cloud Posse)about 6 years ago

https://medium.com/adobetech/how-to-set-up-a-kubernetes-clusters-with-helmfile-4ab7866fc2fc

vgdubabout 6 years ago

Hi All , apologies in advance if this is not the right place to ask this , but I yesterday faced an issue with the helmfile diff --suppress-secrets . I was simply applying some RBAC policies but tiller is now giving full of errors with this.

vgdubabout 6 years ago

I can confirm that tiller has cluster-admin privileges in kube-system namespace so I am not sure why this diff is failing.

Zachary Loeberabout 6 years ago

so what rbac policies did you apply?

Zachary Loeberabout 6 years ago

may I recommend tillerless instead?

James Huffmanabout 6 years ago

another vote for tillerless

DanBabout 6 years ago

can anyone give me pointers on how to use this: https://github.com/roboll/helmfile/pull/906 but for aws secret manager? I have this in my helmfile but it just uses the literal as is, debugging doesn't indicate it tried to resolve the secret

DanBabout 6 years ago

trying to piece together info from the README and this repo: https://github.com/variantdev/vals

DanBabout 6 years ago

helmfile version v0.98.2

DanBabout 6 years ago

doc is wrong, should be ref+awssecrets://..., and depending on your secret name format it may not work at all: https://github.com/variantdev/vals/issues/18 @mumoshu

Ryan Smithabout 6 years ago

@mumoshu It would be interesting for helmfile diff to have a "hardcore" mode that compares against the k8s state instead of the helm state. Embarrassed to say, i hit cases where there's manual changes to k8s resources that aren't reflected in helm state. (If i'm misunderstanding helmfile diff -> helm diff , my bad 😅) Anyways, been using something along the lines of:

# Render all the k8s yaml
helmfile -f hello.yml template > ~/Desktop/hello.helmfile.yml

# Diff the new yaml with what's actually deployed
tail -n +2 ~/Desktop/hello.helmfile.yml | kubectl diff -f - > ~/Desktop/hello.helmfile.diff 

# If diff is acceptable, run helmfile
helmfile -f hello.yml apply

Erik Osterman (Cloud Posse)about 6 years ago(edited)

@Ryan Smith out of curiosity, are you using helm3? was wondering if it would do a better job.

erikabout 6 years ago

Ryan Smithabout 6 years ago

Solid

Ryan Smithabout 6 years ago

I need to upgrade

Erik Osterman (Cloud Posse)about 6 years ago

I'm not sure if this impacts the helm-diff plugin or not.

Ryan Smithabout 6 years ago

🤷‍♂️ but this is a great lead

Erik Osterman (Cloud Posse)about 6 years ago

let me know what you find out! 😃

Maheshabout 6 years ago

helm 3 is very buggy, we face a lot of issues, eg. https://github.com/helm/helm/issues/7426

mumoshuabout 6 years ago(edited)

helm-diff w/ helm 3 is unaffected, as it still shows the diff between the release stored in the cluster(!= the current state of k8s resources originally created for the release) and the manifests rendered by helm template.

but yeah the diff and the install/upgrade result can be much more reliable than in helm 2, as helm3 tries its best not to accidentally "revert" manual changes

mumoshuabout 6 years ago

@Mahesh does it still hang when you fix your deployment? (at glance it can happen when the k8s resources created by the chart is stuck in error or not-ready states, which is not issues in helm

mumoshuabout 6 years ago

you can just rerun it without --atomic and see if it reveals the underlying issue = cause of the hang.

helm should do a better error reporting on user errors if it's actually user error, though.

Maheshabout 6 years ago

we just do helm delete and delete k8s objects created by helm package (its very picky even for secrets)

mumoshuabout 6 years ago

to fix the hang?

Maheshabout 6 years ago

yeah, to redeploy

Ryan Smithabout 6 years ago

Oh crap... good point @Dudi Cohen you’re right. Yeah.. I think I would have to reconcile with kubectl apply with the helmfile template output

Dudi Cohenabout 6 years ago

@Ryan Smith then you won't have a release in helm

Zachary Loeberabout 6 years ago

Random helm/helmfile tip: For what its worth, if you are upgrading to helm 3 ensure you sync more than once for each new chart you deploy for the first time. The three way merge in helm 3 means certain helm constructs will be problematic (such as the autogenerated ClusterIP: "" of a service for instance). Ran into this issue a few times now without realizing it until after the fact.

Andrew Nazarovabout 6 years ago(edited)

ClusterIP was a PITA. Had to set force to false to deal with it. So far so good.

Kashif Rahmanabout 6 years ago

Guys anyone faced such issue that kubernetes job created with helm for db migrations always succeed although when we manually deploy job it shows up actual error

$ node_modules/node-pg-migrate/bin/pg-migrate -m ./migrations-app -v up
No migrations to run!
Migrations complete!
Done in 0.62s

if I manually deploy a job it will show up db migration error which is actual output
what could be the case where job from helm is passing in every case ?

Erik Osterman (Cloud Posse)about 6 years ago

Sounds like maybe the exit code from node_modules/node-pg-migrate/bin/pg-migrate is not getting returned

Erik Osterman (Cloud Posse)about 6 years ago(edited)

can you share how you call it in your docker image? for example, if you're running it in a bash script, you'll want to have set -e to ensure you exit non-zero on all errors

Cameron Boultonabout 6 years ago

It would be helpful if you would all add your commentary to that issue if you experienced it. Hopefully they will reopen or at least point to a new issue with resolution at some point.

Cameron Boultonabout 6 years ago(edited)

@mumoshu Any ideas how I might accomplish value key removals in Helmfile during the values merge? I’m looking for behavior similar to this: https://github.com/helm/helm/issues/1966 Using null however does not seem to make it past the Helmfile value merge operation (that is merging a discreet/defined value with null does not appear to be subtractive; the original value key remains)

Specifically, we have some global value keys like resources, probes, etc. that we want to remove for the default environment only (we use default for local development). Does that make sense?

Thomas Burtonabout 6 years ago

I am trying to install helmfile in a custom Docker file. This is what it looks like:

RUN apk add --update --no-cache curl ca-certificates bash && \
    curl -L ${BASE_URL}/${TAR_FILE} |tar xvz && \
    mv linux-amd64/helm /usr/bin/helm && \
    chmod +x /usr/bin/helm && \
    curl <https://github.com/roboll/helmfile/releases/download/v0.98.2/helmfile_linux_amd64> -O && \
    mv helmfile_linux_amd64 /usr/bin/helmfile && \
    chmod +x /usr/bin/helmfile && \
    helmfile --version

However the build fails with the following out put

/usr/bin/helmfile: helmfile: line 1: syntax error near unexpected token `<'
/usr/bin/helmfile: helmfile: line 1: `<html><body>You are being <a href="<https://github-production-release-asset-2e65be.s3.amazonaws.com/74499101/19b32580-317f-11ea-9dc4-79b9457abdad?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200128%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200128T124544Z&X-Amz-Expires=300&X-Amz-Signature=4504b341305adc17e28b24ce7d340e0c395b662bc7e991a6d2c856a991c17fd2&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dhelmfile_linux_amd64&response-content-type=application%2Foctet-stream>">redirected</a>.</body></html>'

Any ideas?

Graeme Gilliesabout 6 years ago(edited)

Hi! I am attempting to use the exec fuctionality in gotmpl as part of helmfile, to try and pull a value from an external source. I am doing the following in a values.gotmpl file

external_pass: {{ toJson (exec "./vault-show" (list "secrets" "stage")) }}

global:
  appConfig:
    incomingEmail:
      password:
        secret: {{ .external_pass }}

However I get an error saying

executing "stringTemplate" at <.external_pass>: can't evaluate field external_pass in type state.EnvironmentTemplateData

Graeme Gilliesabout 6 years ago

err

David Nolanabout 6 years ago

I just started looking at the cloudposse helmfiles repo, but I keep getting errors. Is there a recommended combination of versions of the repo/helm/helmfile/etc that is known to work well? I had bizarre helmfile/helm version mismatch issues which required me to downgrade helm from 3.0.2 to 3.0.0. Then I discovered that the config in the repo is trying to pull content from github which no longer exists, it looks like coreos renamed files in https://github.com/coreos/prometheus-operator/tree/master/example/prometheus-operator-crd . I updated those URLs, but now I’m hitting this:

error: error validating "<https://raw.githubusercontent.com/coreos/prometheus-operator/master/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml>": error validating data: ValidationError(CustomResourceDefinition.spec): unknown field "preserveUnknownFields" in io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.CustomResourceDefinitionSpec; if you choose to ignore these errors, turn validation off with --validate=false

Erik Osterman (Cloud Posse)about 6 years ago

@David Nolan - your best bet is to fork and use our helmfiles as a starting off point.

David Nolanabout 6 years ago

I’m trying, but I can’t seem to get the Rube Goldberg machine that is helm/hemlfile to work. I was hoping this would be a good place to start on diving into the world of helm. 😞

Erik Osterman (Cloud Posse)about 6 years ago

haha 😃

David Nolanabout 6 years ago

A friend and I were musing that there just isn’t a really good corpus of k8s example deployments, and I was saying I wanted the equivalent of the cloudposse terraform repos… and then I found your helmfiles repo and hope sprung eternal…

Erik Osterman (Cloud Posse)about 6 years ago

😊 sorry - the helmfiles are not as portable as our modules

Erik Osterman (Cloud Posse)about 6 years ago

the fact we pinned to master is wrong - we shouldn't have done that

Erik Osterman (Cloud Posse)about 6 years ago

we haven't redeployed prometheus since they moved thigns around

Erik Osterman (Cloud Posse)about 6 years ago

hence we haven't been bitten by it

David Nolanabout 6 years ago

I’ll probably send you a PR with some updates based on that coreos repo structure change. Right now I’m trying to figure out whats throwing the error about validation. Might be because I’m trying to use an EKS cluster, maybe its missing some extensions.

David Nolanabout 6 years ago

the CP modules for EKS made getting that up trivial 😉

btaiabout 6 years ago

https://github.com/cloudposse/helmfiles/pull/220

Graeme Gilliesabout 6 years ago

Do helmfile enviroments not support having a

set:

stanza? Just a

values:

stanza only?

bradymabout 6 years ago

Hey all, I've got a list of ip addresses stored as a comma separated list in aws ssm parameter store. I'm trying to get the value and split it into a list:

{{ $extIps := "<secretref+awsssm://path/to/VAR?region=us-west-1>" }}

externalIPs:
  {{ range splitList "," $extIps }}
    - {{ . }}
  {{ end }}

But this is giving me:

  externalIPs:
    - 127.0.0.1,0.0.0.0

Does anyone know if it's possible to get it to render as a list of ip addresses like this?

  externalIPs:
    - 127.0.0.1
    - 0.0.0.0

Erik Osterman (Cloud Posse)about 6 years ago

https://github.com/cloudposse/helmfiles/blob/5cff206070d49b8758915d55bf7e364b90cc8ef1/releases/nginx-ingress.yaml#L101-L103

Erik Osterman (Cloud Posse)about 6 years ago

here's how we used it

bradymabout 6 years ago

That makes sense, but it appears my issue is with the secretref+awsssm part. I can get what I want if I set $extIds to the string "127.0.0.1,0.0.0.0" but when I try to do something with the result of the call to aws ssm it doesn't seem to work. Maybe a rendering order thing?

December 2019 Browse by date February 2020