#helmfile - April 2019 | Slack Archive

123 messages

Questions and discussion around helmfile https://github.com/roboll/helmfile and https://github.com/cloudposse/helmfiles
Archive: https://archive.sweetops.com/helmfile/

Mar 2019

April 2019

May 2019

mumoshualmost 7 years ago

@Erik Osterman (Cloud Posse) managed to add helmfile destroy. also, both delete and destroy now tries to delete releases with installed: true only, as we've discussed
https://github.com/roboll/helmfile/pull/530

Erik Osterman (Cloud Posse)almost 7 years ago

Thanks @mumoshu !

Pablo Costaalmost 7 years ago

Hello, I was looking at some skaffold examples and I realized that it has similarities with helmfile sintaxe when deploying helm charts. Does anybody knows if this is just coincidence ?

Pablo Costaalmost 7 years ago

https://github.com/GoogleContainerTools/skaffold/tree/master/examples/helm-deployment

mumoshualmost 7 years ago

im not very sure but perhaps thats just coincidence

mumoshualmost 7 years ago

i do have suggested helmfile integration to skaffold before https://github.com/GoogleContainerTools/skaffold/issues/169

mumoshualmost 7 years ago

but the state of skaffold's multi helm releases support isn't something I'd suggested

mumoshualmost 7 years ago

like it misses templating

SeBalmost 7 years ago

Hello guys,
I am gad a new PR pointed me this slack repo, it is much easier to chat than creating github issues.
I'd like to contribute when I find some time. I did try on the helmfile destroy but @mumoshu was faster than me.

SeBalmost 7 years ago

thanks a lot for your effort @mumoshu

Erik Osterman (Cloud Posse)almost 7 years ago

Good to have you here @SeB

Erik Osterman (Cloud Posse)almost 7 years ago

Yes, @mumoshu is absolutely remarkable. I don’t know how he can implement so much so quickly.

SeBalmost 7 years ago

I agree, this is impressive.

Erik Osterman (Cloud Posse)almost 7 years ago

Have you checked out some of his other projects?

Erik Osterman (Cloud Posse)almost 7 years ago

Some notable ones are #variant and kubeaws

SeBalmost 7 years ago

I'll have look thanks.

SeBalmost 7 years ago(edited)

hello,
Is it possible from the same values.yaml.gotmpl file used for loading some environment values to have template check for values in this very same file?
here is an example

gatewayVersion: ~0.0.0 #may also be "local"
{{if eq .Environment.Values.gatewayVersion "local"}}
gatewayChart: ../../tdc-gateway
{{else}}
gatewayChart: talend/tdc-gateway
{{end}}

SeBalmost 7 years ago

here you can see I want to templatize the release charts name according to the version value to use a “locat” chart or the one from our registry with a given version.

Erik Osterman (Cloud Posse)almost 7 years ago

Yea makes sense

Erik Osterman (Cloud Posse)almost 7 years ago

I am not sure off the top of my head but seems reasonable.

Erik Osterman (Cloud Posse)almost 7 years ago

@Shane do you know? 👆️

Shanealmost 7 years ago

Off the top of my head I'm not sure, but I think the environment is on the second render pass

mumoshualmost 7 years ago

didn't it work? (i suppose it would work

i believe you can even write it shorter


gatewayChart: {{if eq .Environment.Values.gatewayVersion "local" }}../..{{else}}talend{{end}}/tdc-gateway

mumoshualmost 7 years ago

i'm dreaming of helmfile build, that would render all the helmfile.yaml and values.yaml.gotmpl files into plain YAML files to be commited to a git repo for auditing and gitops

mumoshualmost 7 years ago

https://github.com/roboll/helmfile/issues/388#issuecomment-480270012

SeBalmost 7 years ago(edited)

@mumoshu no I cannot make it work even with your shorter proposal

SeBalmost 7 years ago

I’ll try to create a simple example to test it.

SeBalmost 7 years ago(edited)

I have this helmfile.yaml

environments:
  default:
    values:
      - helmfile.values.yaml.gotmpl

releases:
- name: infra
  chart: {{ .Environment.Values.chartName }}

and this simple helmfile.values.yaml.gotmpl

chartVersion: ~0.0.0 #may also be "local"
{{if eq .Environment.Values.chartVersion "local"}}
chartName: ../my-chart
{{else}}
chartName: foo/my-chart
{{end}}

And I get the following errors

>helmfile -f helmfile.yaml apply
could not deduce `environment:` block, configuring only .Environment.Name. error: failed to read helmfile.yaml: failed to load environment values file "helmfile.values.yaml.gotmpl": failed to render [/Users/sgandon/git-repos/helm-charts/platform-services/helmfile/test/helmfile.values.yaml.gotmpl], because of template: stringTemplate:2:20: executing "stringTemplate" at <.Environment.Values....>: map has no entry for key "chartVersion"
error during helmfile.yaml parsing: template: stringTemplate:10:24: executing "stringTemplate" at <.Environment.Values....>: map has no entry for key "chartName"

SeBalmost 7 years ago

shall I create an issue ?

mumoshualmost 7 years ago

Ah gotcha. Unfortunately you can't self-reference environment values from within environment values file template

mumoshualmost 7 years ago

I've added it as requested but I'd admit that its very confusing

mumoshualmost 7 years ago

maybe we can add something similar to Release Template(see the helmfile best practices guide under docs/) so that it can potentially be written like:

helmfile.values.yaml (not helmfile.values.yaml.gotmpl)


chartVersion: ~0.0.0 #may also be "local"
chartName: "{{if eq .Environment.Values.chartVersion "local"}}..{{else}foo{{end}}/my-chart"

mumoshualmost 7 years ago(edited)

Back to your original work, it is really a text template. helmfile.values.yaml.gotmpl is firsly executed to generate a YAML file. At that time, .Environment.Values.chartVersion is not avaiable yet, as the YAML data isn't rendered hence not loaded yet

mumoshualmost 7 years ago

It's hard(imho) and that's why I've been saying I'd recommend using jsonnet to generate necessary files before running helmfile, or give up making it DRY too much.

But I had not received much traction about it so far!

SeBalmost 7 years ago(edited)

I am fine with this not working I just saw some examples in some github issues.

SeBalmost 7 years ago

Actually I do use the templates feature and it works fine.

SeBalmost 7 years ago(edited)

I do this

templates:
  chart: &chart
    {{ $versionKey := printf "%sVersion" `{{.Release.Name}}` }}
    chart: {{ .Environment.Values | get ($versionKey) "" | eq "" | ternary "../.." "talend" }}/{{`{{.Release.Name}}`}}

mumoshualmost 7 years ago

Great!!

mumoshualmost 7 years ago

Oh so are you saying you managed to use templates in an environment values file?

SeBalmost 7 years ago

heu no, not in a environment values file, I do that in my helm file.

SeBalmost 7 years ago(edited)

here is my helmfile

environments:
  default:
    values:
      - ../values/chart-versions.yaml

templates:
  chart: &chart
    {{ $versionKey := printf "%sVersion" `{{.Release.Name}}` }}
    chart: {{ .Environment.Values | get ($versionKey) "" | eq "" | ternary "../.." "talend" }}/{{`{{.Release.Name}}`}}
    version: {{.Environment.Values | get ($versionKey) ""}}

releases:
- name: tpsvc-crypto
  <<: *chart
...and many more

SeBalmost 7 years ago

and here is my charts-versions.yaml

##############################
# Version of the charts to deploy
# when the values are empty it will use the local chart directly

tpsvc-cryptoVersion: 
...and many more

SeBalmost 7 years ago(edited)

I just wanted to apologize to create so many tickets, but helmfile is a little rough around the edges and we at Talend are going to use it heavily internaly for our development process.
When I have finds some time I’d be happy to contribute cause I believe this a great tool.

SeBalmost 7 years ago

On question though, what is the cost of adding values file that may not exists in terms of speed ?
Cause for each of my charts I have the following values file setup using a template

templates:
  sharedvalues: &values
    missingFileHandler: warn # prevent the install to stop if some files are not found
    values:
      - "../values/{{`{{.Release.Name}}`}}.yaml"
      - "../values/{{`{{.Release.Name}}`}}.yaml.gotmpl"
      - "values/{{`{{.Release.Name}}`}}.yaml"
      - "values/{{`{{.Release.Name}}`}}.yaml.gotmpl"
      - "{{ env "TALEND_LOCAL_PATH" }}/platform-services/helmfile/hybrid-remote/values/{{`{{.Release.Name}}`}}.yaml"
      - "{{ env "TALEND_LOCAL_PATH" }}/platform-services/helmfile/hybrid-remote/values/{{`{{.Release.Name}}`}}.yaml.gotmpl"

And most of them are not there but could be. (I have to duplicate yaml and gotmpl extensions cause yaml is much nicer and simpler but it is not interpreted 😞 )

SeBalmost 7 years ago

Hello everyone,
Is there a way to load helmfile wide values without failing when the file is not found.
I am using right now the environment.default but if the values file is not found it is failing.

environments:
  default:
    values:
      - ../values/chart-versions.yaml
      - {{ env "LOCAL_VALUES_CACHE" }}/foo/helmfile/values/chart-versions.yaml

SeBalmost 7 years ago

Also it seems that the values file for an environment cannot be abosolute path but only relative ones ?????

SeBalmost 7 years ago

I’ll create 2 🙂

Andrew Nazarovalmost 7 years ago

Hi. I was wondering how installed: false works… For some reason I don’t see my release marked with installed: false among ones taken for diff (for ex. “Comparing shared-dev-elastic elastic/elasticsearch”). And my release doesn’t get uninstalled.

Andrew Nazarovalmost 7 years ago

But in helmfile’s log output I see helm list against the release I want to uninstall.

Andrew Nazarovalmost 7 years ago(edited)

Wow, I’ve been just struck by the curious failure. Not related with the aforementioned, btw. I noticed that for one of my release helmfile apply triggered helm upgrade --install three times in a row(!). There were no diffs for that particular release. One upgrade succeeded, but for another two the following error was printed:


UPGRADE FAILED
ROLLING BACK
Error: release: "XXXX-pgdump.v4" already exists
Error: UPGRADE FAILED: release: "XXXX-pgdump.v4" already exists

And it all ended up with err: release "XXXX-pgdump" in "helmfile.yaml" failed: exit status 1 and err: remove /tmp/values636482145: no such file or directory error.

It had worked ok before I bumped helmfile version to the most recent.

Kevin Gimbelalmost 7 years ago

Hello all! I’ve just started using helmfile and I was wondering if I can tell it which service account it should use for the deployments? If I run helmfile apply it automatically uses system:serviceaccount:staging:default but I’d rather have it use system:serviceaccount:staging:helmfile or some other dedicated user. Is there anyway to do this with helmfile?

Jakealmost 7 years ago

Hey, just a quickie; is there any place I can see what values I have available to me in a template? Trying to use variables like .Release.Name within the template file and not having much success.

Erik Osterman (Cloud Posse)almost 7 years ago

@mumoshu have you met @hairyhenderson?

hairyhendersonalmost 7 years ago

@hairyhenderson has joined the channel

Erik Osterman (Cloud Posse)almost 7 years ago

Just thought I'd make the intro since I saw your comment here: https://github.com/roboll/helmfile/issues/392#issuecomment-455061273

SeBalmost 7 years ago

Just a question out of the open,
What is the advantage of using helmfile versus ansible ?

Andrew Nazarovalmost 7 years ago

Experiencing the issue when my releases are not get updated via helmfile apply even though the version of a chart in helmfile.yaml is bumped. helm diff doesn’t show me a diff either. Cannot figure out what’s the problem…

Brianalmost 7 years ago

Hey! How would I be able to use the value for namespace in my chart?

Erik Osterman (Cloud Posse)almost 7 years ago

I think it was just “{{ .Namespace }}” but on my phone so hard to check

Erik Osterman (Cloud Posse)almost 7 years ago

You are talking about using the value of the —namespace arg, right?

Brianalmost 7 years ago

Here's what I just tried:

  #
  # Testing Pod
  #
  - name: testing
    namespace: blue
    chart: ./chart
    values:
    - "./values/testing-values.yaml"
    - "./values/master.yaml"
    wait: true
    set:
    - name: image.tag
      value: 74.1.192
    - name: deployment.env 
      value: {{ .Environment.Values.environment }}
    - name: deployment.replicas 
      value: {{ .Environment.Values.replicas }}
    - name: deployment.namespace
      value: {{ .Namespace }}

Brianalmost 7 years ago

I'm leaving the namespace hardcoded, although I saw that's not recommended, as we're using two separate charts for blue green deployments.

Andrew Nazarovalmost 7 years ago

Why are you trying to set the namespace twice? namespace: blue is enough to deploy your workloads in a namespace called “blue”.

Brianalmost 7 years ago

I need to assign the blue pods to an instance groups for blue, and green pods to instance group for green. I have a toleration set to noschedule based off of the namespace name.

Andrew Nazarovalmost 7 years ago(edited)

I don’t know the logic behind your helm chart, but what about just using {{ Release.Namespace }} inside your chart? It should pick the namespace defined in your helmfile.

Andrew Nazarovalmost 7 years ago

Never thought about how to do blue-green using helmfile though. That’s kinda interesting.

Erik Osterman (Cloud Posse)almost 7 years ago

https://github.com/cloudposse/example-app/tree/master/deploy

Erik Osterman (Cloud Posse)almost 7 years ago

Have an example of how we demo blue green with Helmfile

Erik Osterman (Cloud Posse)almost 7 years ago

We also use #variant by @mumoshu to define a deploy tool

Erik Osterman (Cloud Posse)almost 7 years ago

This demo uses istio gateway

Andrew Nazarovalmost 7 years ago

Btw, why have cloudposse decided to opt values embedded in helmfile.yaml?

Andrew Nazarovalmost 7 years ago

Thank you for the link.

Erik Osterman (Cloud Posse)almost 7 years ago

We use environment variables because they are interoperable with all tools

Erik Osterman (Cloud Posse)almost 7 years ago

We use direnv for configurations (non secret)

Erik Osterman (Cloud Posse)almost 7 years ago

We use chamber for secrets

Brianalmost 7 years ago

Yeah. I just started messing around with Helmfile the other day. So far, here's how I'm structuring my blue/green deployments:

├── chart
│   ├── Chart.yaml
│   └── templates
│       ├── NOTES.txt
│       ├── _helpers.tpl
│       ├── autoscale.yml
│       ├── deployment.yml
│       ├── ingress.yml
│       ├── service.yml
│       └── tests
├── enterprise-blue.yaml
├── enterprise-green.yaml
├── ingress
│   ├── Chart.yaml
│   └── templates
│       ├── _helpers.tpl
│       ├── configmap.yml
│       ├── ingress-backend.yml
│       ├── ingress-nginx.yml
│       ├── rbac.yml
│       ├── service.yml
│       └── tests
├── ingress-helmfile.yaml
├── single-helmfile.yaml

I have a separate values file for each microservice. I still have played around with it enough to determine if this is the best solution. It'll probably change later today.

erik-stephensalmost 7 years ago

Anyone know how to get --args=--context=2 relayed to the diff command when I helmfile apply?

Erik Osterman (Cloud Posse)almost 7 years ago

https://github.com/roboll/helmfile/pull/569

Erik Osterman (Cloud Posse)almost 7 years ago

@Ryan Smith ☝️

Ryan Smithalmost 7 years ago

@Ryan Smith has joined the channel

Ryan Smithalmost 7 years ago

lololol thx

Ryan Smithalmost 7 years ago

@stobiewankenobi

stobiewankenobialmost 7 years ago

@stobiewankenobi has joined the channel

Ryan Smithalmost 7 years ago

lolol ^

Erik Osterman (Cloud Posse)almost 7 years ago

That's a cool idea

Erik Osterman (Cloud Posse)almost 7 years ago

can you add an example invocation in gotemplate syntax?

Ryan Smithalmost 7 years ago

- autoDiscovery:
          clusterName: {{ requiredEnv "cluster_name" }}

AWS_REGION=us-east-1 SSM_PATH=/redacted/ ./helmfile -f $GITHUB/calm/helmfile/stable/init.yml template

Ryan Smithalmost 7 years ago

super duper small example

Ryan Smithalmost 7 years ago

but nothing fancy

Erik Osterman (Cloud Posse)almost 7 years ago

@Jeremy G (Cloud Posse) you might dig this.

Jeremy G (Cloud Posse)almost 7 years ago

@Jeremy G (Cloud Posse) has joined the channel

Ryan Smithalmost 7 years ago

just a replacement of aws-env

Erik Osterman (Cloud Posse)almost 7 years ago

why aws-env vs chamber?

Ryan Smithalmost 7 years ago

i haven't used chamber enough to have an opinion

Ryan Smithalmost 7 years ago

🤷‍♂️ but regardless, it's 2 commands

Ryan Smithalmost 7 years ago

just thinking about terraform -> terragrunt

Ryan Smithalmost 7 years ago

i was thinking.. helmfile -> helmgrunt

Ryan Smithalmost 7 years ago

but like.. naaa, just update helmfile directly

Erik Osterman (Cloud Posse)almost 7 years ago

this will make helmfile easier to automate for us since we use ENVs for so much.

Erik Osterman (Cloud Posse)almost 7 years ago

we've been calling chamber exec ... helmfile ...

Ryan Smithalmost 7 years ago

😉

Ryan Smithalmost 7 years ago

same, basically here

Erik Osterman (Cloud Posse)almost 7 years ago

@mumoshu ☝️

Ryan Smithalmost 7 years ago

and giving credit where credit's due--this was all your idea initially 😉

Erik Osterman (Cloud Posse)almost 7 years ago

aha, so I see this just loads every parameter into the env, so everything else just continues to work seamlessly.

Kevin Gimbelalmost 7 years ago

Hey everyone, I'm having sort of a crisis with helmfile. Maybe I misunderstood how it should work, but whenever I deploy a state (helmfile apply) I can not add new releases to it. If I make a change to the helmfile.yaml the new releases cannot be deployed because both helmfile sync and helmfile apply fail with an error indicating they cannot find (and therefore upgrade) the new release. Can I never again add releases to my helmfile without destroying everything first?

Andrew Nazarovalmost 7 years ago

@Kevin Gimbel What is the actual error shown? Could you please provide an example? It works for me very well.

Kevin Gimbelalmost 7 years ago(edited)

Here's the cleaned-up log I am getting when I run helmfile sync


 helmfile sync 
Adding repo my_private_repo <https://XXX>
"my_private_repo" has been added to your repositories

Updating repo
Hang tight while we grab the latest from your chart repositories...
...Skip local chart repository
...Successfully got an update from the "my_private_repo" chart repository
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈ Happy Helming!⎈ 

Upgrading my_private_repo/XXX
Upgrading my_private_repo/XXX
Upgrading my_private_repo/XXX
Upgrading my_private_repo/XXX
Upgrading my_private_repo/XXX
Upgrading my_private_repo/XXX
Upgrading my_private_repo/XXX
Upgrading my_private_repo/XXX
Upgrading my_private_repo/XXX
Upgrading my_private_repo/cerebro.s3.product.image
Upgrading my_private_repo/XXX
Upgrading my_private_repo/XXX
Upgrading my_private_repo/XXX

[...]

UPGRADE FAILED
ROLLING BACK
Error: "cerebro-s3-product-image" has no deployed releases
Error: UPGRADE FAILED: "cerebro-s3-product-image" has no deployed releases

[...]

err: release "cerebro-s3-product-image" in "helmfile.yaml" failed: exit status 1
exit status 1

Kevin Gimbelalmost 7 years ago

I've replaced the names of all non-related releases with XXX and removed all the output before and after the error

Kevin Gimbelalmost 7 years ago

If I run helmfile destroy and then helmfile apply everything installs fine

Andrew Nazarovalmost 7 years ago

It’s a known case like if your release fails being deployed for the first time and you don’t set atomic flag to true then yes, you cannot do anything except purging your failed release first.

Andrew Nazarovalmost 7 years ago

https://github.com/roboll/helmfile/issues/471

SeBalmost 7 years ago

yes the error here is missleading and should propose a solution.

SeBalmost 7 years ago

If you want to check the status of you helm releases, use helm list -a

SeBalmost 7 years ago

and you may see some in FAILED or PENDING_INSTALL that you need to delete

Kevin Gimbelalmost 7 years ago

There is indeed a Failed deployment, but this was from a helm install not from helmfile - can in theory every failed deployment fail my helmfile deployments? We're not yet running in production which is why I installed some stuff with helm and some with helmfile as of now

Kevin Gimbelalmost 7 years ago

I'll look into the atomic flag, thanks a lot!

Andrew Nazarovalmost 7 years ago(edited)

It should work ok with releases in FAILED state if it’s not their initial state. Then, atomic flag allows you to restore the previous state if something goes wrong, hence even if the very first deployment fails Helm will remove it automatically.

Kevin Gimbelalmost 7 years ago

It should work ok with releases in FAILED state if it’s not their initial state.

This is true. With atomic: true I am able to update / upgrade the releases as I'd expect to, but only if nothing failed with the initial deployment

Lee Skillenalmost 7 years ago

Nice, helmfile supports private repositories out of the box - We (Cloudsmith) recently added support for Helm repositories and we're looking at how we can document integration with a few of the solutions that are out there. Looking at helmfile and landscaper atm. 🙂 This is mostly for user reference, but everyone likes a worked example. We've only got a very minimal example at the moment.

Erik Osterman (Cloud Posse)almost 7 years ago

@zadkiel I'm trying ot use helm-git but running into errors

Erik Osterman (Cloud Posse)almost 7 years ago


helm repo add t1 "git+<https://github.com/stakater/Forecastle@deployments/kubernetes/chart/forecastle?ref=master>"

Erik Osterman (Cloud Posse)almost 7 years ago

i'll move to #helm

March 2019 Browse by date May 2019