#kubernetes - June 2024

Hi all.

What are you using to display the current cluster details in the terminal?
As in something that might help you tell if you're in prod or some other environment?

Guys, what's the best practice for resource allocation to deployments/statefulset? Should I not put any allocations at all and just rely on it's horizontal scaling? I feel like adding memory and CPU resources defeats the purpose of auto scaling? Please correct me if I'm wrong

I have a (I think pretty nooby) question, but for some reason can’t find the right phrase to google to work it out, or I’m skimming over the answer when reading docs somehow…

If I have a Deployment with replicas set to 4, and a HorizontalPodAutoscaler with minReplicas set to 2, which value takes precedence? I’m assuming the HPA will attempt to scale down to 2 when there isn’t enough utilisation to justify it, but I just can’t tell for sure!

Sorry if this is super obvious, but my brain rot from endless meetings this week won’t let me see the answer, lol

👋 Hello, team!

Does anyone have any experience or ideas about running a k8s control plane in our account, and having k8s worker nodes in different cloud environments (keeping multi-tenancy in mind).

Our idea is having a product control plane (k8s), and each customer would have worker node groups in their respective cloud accounts. The control plane would ensure our services, license management, sharing configurations from us to the customer all happens there.

Is multiple node-workers spanning across multiple accounts possible with a single control plane?

Issue: Application Performance
Explanation: We have deployed all our microservices on AWS EKS. Some are backend services that communicate internally (around 50 services), and our main API service, "loco," handles logging and other functions. The main API service is accessed through the following flow:
AWS API Gateway -> Nginx Ingress Controller -> Service.
In the ingress, we use path-based routing, and we have added six services to the ingress, each with a corresponding resource in a single API gateway.
Our Angular static application is deployed on S3 and accessed through CloudFront. The complete flow is as follows:
CloudFront -> Static S3 (frontend) -> AWS API Gateway -> VPC Link -> Ingress (Nginx Ingress Controller with path-based routing) -> Services -> Container.
Problem: Occasionally, the login process takes around 6-10 seconds, while at other times it only takes 1 second. The resource usage of my API services is within the limit.
Below are the screenshots from Datadog traces of my API service:
• Screenshot of the API service when it took only 1 second
• Screenshot of the API service when it took 6-10 seconds
Request for Help: How should I troubleshoot this issue to identify where the slowness is occurring?

We are updating our terraform-aws-eks-node-group module to support EKS on AL2023. With that, it will support AL2, AL2023, Bottlerocket, and Windows Server. Each has different ways to configure kubelet, and kubelet itself is moving from command line args to KubeletConfiguration. I am seeking community input on how the module should take configuration parameters and deploy them properly to each OS.

Goals:
• We want to limit maintenance, so we only want to support the most used configuration options. I'm thinking just kube-reserved system-reserved eviction-hard eviction-soft, although we also need to support taints applied before the node joins the cluster.
• We want it to interoperate well with EKS defaults. In particular, I'm concerned with using --config <file> with bootstrap.sh. Has anyone tried that?
• We will always allow you provide the complete userdata for the Node, so advanced use cases can be handled via that escape route.
One other question: AMI IDs in the Launch Template. Previously, if you just wanted the latest AMI for your nodes, we left the AMI ID out of the launch template and let EKS handle updates when new versions were released. I thought we might change that to always specify the AMI ID, providing a consistent experience, getting the AMI ID from the Public SSM Parameter for the latest ("recommended") AMI. However, it does seem we lose some features that way, like visibility of the availability of updates via the AWS Web Console. Thoughts?

Hi,
We are having ingress manifest which contain a cognito authentication as a default that we don't want, so after deployment we have removed it manually from console, however since we are managing resources argocd, the rule getting added back. I have disabled the automatic sync, but it's doesn't help. Need suggestions!

Hi, I'm using Postgres-HA chart of Bitnami and after using it for a while, we decided that we don't really need HA and a single pod DB is enough without the pg_pool and the issues it comes with, right now we're planning to migrate to normal Postgres chart of Bitnami, I would like to know how to have the database's data persisted even if we switch to the new chart ?

I got another noob question about K8S... I was testing a cluster update, that happened to cycle the nodes in the node group. I'm using EKS on AWS, with just a single node, but have three AZs available to the node group. There's a StatefulSet (deployed with a helm chart) using a persistent volume claim, which is backed by an EBS volume. The EBS volume is of course tied to a single AZ. So, when the node group updated, it seems it didn't entirely account for the zonal attributes, and cycled through 3 different nodes before it finally created one in the correct AZ that could meet the zonal restrictions of the persistent volume claim and get all the pods back online. Due to the zonal issues, the update took about an hour. The error when getting the other nodes going was "1 node(s) had volume node affinity conflict." So basically, any pointers on how to handle this kind of constraint? Is there an adjustment to the helm chart, or some config option I can pass in, to adjust the setup somehow to be more zonal-friendly? Or is there a Kubernetes design pattern around this? I tried just googling, but didn't seem to get much in the way of good suggestions... I don't really want to have a node-per-AZ always running...