#kubernetes - November 2020

Anyone ever have a zombie k8s resource problem? We have a resource that we cannot delete (the delete processes, but the resource doesn’t go anywhere). Any debugging / troubleshooting tips to get at the internals of k8s to address that? More info in thread.

What is the goto PVC -> S3 backup solution (helm chart) nowadays?

Qovery Engine – open-source multi-cloud deployment - https://github.com/Qovery/engine

Hello Experts.

we are running Kubernetes in AWS deployed using Kops,
for backup purpose we are using Velero with Restic integration,
I am new to Velero.
And I wanted to know under what condition Velero will take EBS snapshot and and under what condition Velero will backup using Restic repo. for PV’s.
Because we are having multiple PV’s and all the PV’s are annotated with kubectl annotate pod/<pod-name> <http://backup.Velero.io/backup-volumes=|backup.Velero.io/backup-volumes=><pvcname>, but some PV’s are backed-up using EBS Snapshot and some are backed up using Restic.

Thank you.

Hey @Andriy Knysh (Cloud Posse) @Erik Osterman (Cloud Posse) — Low priority, so take your time in getting back, but I see you guys use {{event.tags.cluster_name}} a bunch in your DD monitors.yaml. I’m not finding that variable available in the message content for my monitor, but my metrics/events do have that tag. Did you folks have to do something specific to enable more variables in scope of that message content? I’m struggling with that right now.

Both of our ingresses are set up like this: ELB -> service -> pods. Does the ingress just pass the request to the service and let the service determine which node the pod will run on? I'm trying to get requests that come in to a node to pass the request to a service pod on the same node.

I recall way back a really nifty example kubernetes service someone built... it was a simple http server responding back with the request headers, content etc, including extra detailed info about the current pod which responded.... anyone have a clue which one it was?.. podinfo of course... my brain suddenly woke up from being teased enough

crossposting as might be relevant to kube too. Hope it's useful to someone. 👋

Is anyone using https://github.com/kubernetes-sigs/external-dns with a private cluster?

If so, I assume this will only work using AWS CNI? So that the DNS can resolve to the private vpc IP, which (some) other places can then route too

Am trying cluster migration in AWS, Both  k8s clusters are in same region.
Cluster 1 : Deployed 2 Application with PV reclaim policy one as Delete and another as Retain, and annotated so it will take Restic backup.
Cluster 2: Restored those 2 applications, worked fine.

again
Cluster 1: Deployed same 2 application with Reclaim policy as Delete and Retain but not annotated so it took snapshot when i backup.
Cluster 2: Restore did not work as PV volume is failed to attach with the following Warning FailedAttachVolume pod/<pod-name> AttachVolume.Attach failed for volume "pvc-<id>" : Error attaching EBS volume "vol-<id>" to instance "i-<instance-id>": "UnauthorizedOperation: You are not authorized to perform this operation.

So, Snapshot restore feature will work in the same AWS region or am only getting this error????

Are there any advantages on placing stuff like cert-manager, cluster-autoscaler , external-dns , aws-load-balancer-controller into the kube-system namespace, or isolate all this stuff into it's own namespace?
Almost all tutorials and example code use as default for this services/controllers the kube-system namespace, however there is are advantages on splitting everything into namespaces.

https://github.com/hashicorp/terraform-provider-aws/issues/13643 🤯 anyone have any workarounds for this?

Best way to manage EKS?

Hi guys, how could I fix this base64 decoding? It spews out this gibberish from Jenkins’ pod :)

printf $(kubectl get secret --namespace default jenkins-160573443 -o jsonpath="{.data.jenkins-admin-password}" | base64 --decode);echo
�K��ly�޷�jg���u�ں"�ϵ�N{߯5��#��

Hey folks — I want to switch a project off of DataDog log management to ELK due to cost. I’m looking for the best resources to do that — Any recommendations?

I’ve asked this here lightly before and I know the Cloud Posse approach is FluentD => Firehose => ElasticSearch. I’d like to implement something similar with FluentBit > FluentD (project is running Fargate so smaller sidecar containers + aws-for-fluentbit is attractive), but before I dive into implementing all of that I figured I should ask what’re the best resources / OSS / possible terraform modules I should pick up to accomplish this with the least amount of pain.

While upgrading EKS from 1.15 to 1.18 using eks module, do we have to upgrade step by step like 1.15 ==> 1.16 ==> 1.17 ==> 1.18 or I can directly modify cluster_version = "1.18" and tf apply will do all magic?

Did anybody else get hit by the 🐛 EKS AMI - https://github.com/awslabs/amazon-eks-ami/releases/tag/v20201112 - that was fun 😬

Greetings! I was wondering if anyone had any chance to play with crossplane? Would you know if it is somewhat comparable to TF cloud operator?

Hey all, working with creating a new cluster, basically running into an issue where basically the alb-ingress-controller can’t see any subnets on an ingress being created despite those subnets existing, any ideas?

aws-load-balancer-controller-5d96f6c4f6-vq86z controller {"level":"error","ts":1606243773.951048,"logger":"controller","msg":"Reconciler error","controller":"ingress","name":"grafana","namespace":"monitoring","error":"couldn't auto-discover subnets: unable to discover at least one subnet"}

Hi guys - can someone hep explain to me how port-forward works under the hood. I got a bit confused when I saw that the rbac permissions required need “create” permissions:

 rule {
    apiGroups = [""]
    resources = ["pods/portforward"]
    verbs     = ["get", "list", "create"]
  }

Hi all, is anyone know any tools that can provide SAML authentication in Kubernetes EKS cluster
It may possible using HashiCorp Boundary but I want to explore other tools...

Any idea, if we corrupt the configmap aws-auth, then how to recover it?
Once configmap aws-auth settings got corrupted no one can access EKS Cluster. Any workaround?

Do you guys (iptables or network policy) block the EC2 metadata api or redirect to a metadata proxy for containers to remain “sane” when providing iam roles via the “native” eks service role method?

Hi all - what are some approaches for application config management in kubernetes? A few topics I’m interested in
• dynamic configuration (say for example configuration a feature flag in an app)
• deploying applications that share configuration (or secrets)
Just looking for some projects to look into to get a feel for what people are doing

Has me nodding along, kube already feels like legacy to me, it will be replaced by something sooner than later (by the founder of tailscale, which just came up in #random)... https://blog.dave.tf/post/new-kubernetes/