#kubernetes - December 2019

#first ☝️ 😃

@Andriy Knysh (Cloud Posse) ☝️

i really don't like that pattern of mapping a namespace

surely affinities and taints would be a much better way of doing that

Hey folks. I tried a few searches, but I apologize if this question is answered elsewhere. I'm about to move the POC certmanager deployment which can't perform dns001 chalenges to https://github.com/cloudposse/charts/tree/master/incubator/cert-manager but before I do I was curious if there was anyone here who is using it on EKS with a service account & aws-iam-authenticator (aka not kiam)?

I'm getting this error without using a chart and am struggling to see what I've misconfigured:

controller.go:131] cert-manager/controller/challenges "msg"="re-queuing item  due to error processing" "error"="error instantiating route53 challenge solver: unable to assume role: WebIdentityErr: unable to read file at /var/run/secrets/eks.amazonaws.com/serviceaccount/token\ncaused by: open /var/run/secrets/eks.amazonaws.com/serviceaccount/token: permission denied" "key"="edx..."

Anyone have good structural patterns to follow for node labeling?

For instance, do you label specific nodes for general applications or are they app specific or both?

A tip that might save you the same 6 hours of your life that I lost: It appears that a) you can overload the metrics-server pod on GKE by calling it too many times and b) helm will stops working when any api is down (fixes for 2.16 and 3.1 in progress/merged). Tips to work around it here: https://github.com/helm/helm/issues/6361

so what’s this Kustomize I am hearing about? Worth looking into? :)

@mumoshu I think uses it

i'd say it's a valid next step after you get tied of kubectl+sed

Cuelang looks great, recommend watching the tgik8s episode with Joe Beda just released on the weekend.

I don't see an #istio channel or anything, so I'll post here

whyyyyyyyyyyyy?!?!?!

Wow, that sucks.

We also saw they were investing in a new helm chart that we've been waiting for in parallel to this istioctl tool

@Jeremy G ☝️

this is what i was thinking of: https://github.com/istio/installer/tree/master/istio-telemetry

but i guess that's just the telemetry portion which is built on helm charts

though this is wierd...

and it's actively being developed

@Rhooker @Adam Carlile heads up

@Rhooker has joined the channel

@Adam Carlile has joined the channel

this just means they're carving out a niche for cloudposse to maintain working helm charts for helmfile deploys 🙌 🙌

haha - really hoping not to! 😛 my preference is deferring thought leadership as much as possible, especially with something as complicated as istio

so if I am redeploying clusters on a bi weekly basis how does istio want me to manage that?

If we set SSL termination at the cloud provider elb level. The traffic from our node to our ingress controller should be unencrypted right. hypothetically if i wanted to have it encrypted through to the Ingress controller is the best way to turn off ssl termination on the cloud provider elb?

you can re-encrypt traffic to the backend without needing to turn off termination at the elb

common pattern is to encrypt with private certificates internally and then a publicly trusted wildcard at the elb

would i be able to figure out what causes my worker node reboot by looking at /var/log/syslog and /var/log/messages ? I took an AMI copy of the worker node after i cordoned+drained it and it doesn’t save the dmesg logs from the actual instance. I do have those other two logs @Erik Osterman (Cloud Posse)

i can track down to the time in which i believe the kubelet starts up again in syslog but theres approx 1 minute of nothing that gets logged to syslog right before that point

Thanks for the invite @Ryan Smith!

https://www.signadot.com/ ^^^^ is Arjun's Company 🙂 🙌

Welcome @Arjun Iyer !

Looking for recommended way to get kubernetes events into our elasticsearch log cluster. eventrouter looks good but referred to upstream effort 2.5 years ago (https://github.com/heptiolabs/eventrouter/issues/26#issuecomment-318472723). Anyone happen to know what that upstream effort is?

Related but not what you are looking for: https://github.com/getsentry/sentry-kubernetes

We use this... happy with it

This is for shipping to sentry rather than ES

We've thought about deploying sentry for application level errors. Maybe this will help tip the scales 🙂

Do you use #helmfile?

Yup!

Ours is here

For sentry

And the forwarder

You know of any issues with your charts working in air-gapped (no internet access) environments?

Just curious, cuz we typically have to fork charts to get them working here.

Nevermind, those are helmfiles :)

Unfortunately, no experience working in airgapped environments

@erik-stephens totally randomly (just right now) I stumbled across this tool: https://github.com/max-rocket-internet/k8s-event-logger

it's a pretty smart solution.

basically you just deploy a pod that echos the events to stdout

that way whatever logging solution you already have in place will capture them 😃

What is the best way to test helm charts (on travisci)? I was trying to do helm install --dry-run --generate-name but it needs a k8s cluster which is a lot of setup. Does helm lint also generate the yaml files?

try template , it is very useful to see how the files will be generated

cool, i went with helm lint "$d" && (helm template "$d" | kubeval)

Nice tip

---
https://learnk8s.io/kubernetes-ingress-api-gateway

other than this feature introduced as alpha in k8s 1.16, is there an simple way to evenly distribute a type of pod onto a cluster (i.e. 100 redis pods distributed onto a 10 node cluster = 10 redis pods per node) currently on 1.13/1.14 clusters https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/