#kubernetes - July 2020

When I set the kubelet-certificate-authority flag in kube-apiserver.yaml, I am running into the following message when trying to start a pod. I am using kubespray to provision the AWS cluster.

Error from server: Get <https://10.250.205.173:10250/containerLogs/default/bash-shell-d8bd1/bash-shell-d8bd1>: x509: cannot validate certificate for 10.250.205.173 because it doesn't contain any IP SANs

I think this is the controller node getting status from a worker node. The information that I found about this issue is:

That message is coming from the master trying to connect to the node (the flow of traffic 
is kubectl -> master API -> kubelet -> container). When starting the master, are you 
setting --kubelet_certificate_authority? If so, the master expects to be able to validate 
the kubelet's serving cert, which means it needs to be valid for the hostnames/IP addresses 
the master uses to connect to it.

Any help to resolve this issue would be appreciated.

I question why the master is using an IP address for the worker node. I've been trying to find information about kubelet-preferred-address-types. I wonder if I can change that setting.

I set --kubelet-preferred-address-types=InternalDNS (and just this value). Then tried to start a pod. This error was displayed:

Error from server: no preferred addresses found; known addresses: [{InternalIP 10.250.205.173} {Hostname ip-10-250-205-173.ec2.internal}]

Edit: The InternalIP and Hostname are literally telling me what is acceptable as address type. When I add Hostname, the error changes:

Error from server: Get <https://ip-10-250-205-173.ec2.internal:10250/containerLogs/kube-system/nodelocaldns-s8mfk/node-cache>: x509: certificate signed by unknown authority

Is there a way I can prevent creation of new config maps every time I deploy through helm.

Hi, I’m trying https://github.com/cloudposse/terraform-aws-eks-cluster and for an unknown reason, even if all my ressources seem OK, my cluster has 0 nodes.

Hey all!
Is anyone deploying to AWS EKS with the readinessGates in the K8s Deployment?

The issue TL;DR
• ALB (ingress) is being managed as a separate deployment with multiple host paths
• When deploying to AWS EKS with readinessGates the Service does not register the pods; The deployment then times out and fails
• When deploying to AWS EKS without readinessGates the Service registers the pods; The deployment is successful BUT podDistributionBudget issue arises when nodes are rotated during upgrade.

Hey everyone. my pod refuses to launch due to an x509: certificate signed by the unknown authority . I use an AWS Certificate for my docker registry and it looks like I should perform a simple update-ca-certificates to solve my issue; I added it in my gitlab-runner as a step before doing the helm install... but it doesn’t help (actually, this commands returns a skipping so not even sure it works). i’m on this error since yesterday and I have the feeling to have tested everything… could somebody help me on this?

Hey, all. Looks like my DD cluster agent is not configured properly for the liveness and readiness probes. First, no ready pods:

$ k get pods
NAME                                          READY   STATUS    RESTARTS   AGE
datadog-cluster-agent-6cf486544f-gvv9s        0/1     Running   0          6d

And from describe , it looks like these tests are misconfigured (no host), but i haven't found a way of specifying that -- which I should not have to do:

    Liveness:   http-get http://:5555/live delay=15s timeout=5s period=15s #success=1 #failure=6
    Readiness:  http-get http://:5555/ready delay=15s timeout=5s period=15s #success=1 #failure=6

Events:
  Type     Reason     Age                        From                                                Message
  ----     ------     ----                       ----                                                -------
  Warning  Unhealthy  4m42s (x34539 over 5d23h)  kubelet, ip-10-6-32-102.us-east-2.compute.internal  Readiness probe failed: HTTP probe failed with statuscode: 500

Any idea what i'm missing here?

Anyone using the IstioOperator CRD? The Gateway spec does not include SDS so I’m trying to figure out during migration where to specify the SDS container? The new Istio Helm Gateway chart does not seem to have the option either.

Anyone doing container forensics?

Adding the EventRateLimit admission control to my api-server manifest file results in the api-server not restarting but I don't know where to find any error messages. Why would this fail? How can I debug the issue?

Guys, would you expect any issues with EKS pods/containers if you encrypt node’s root volumes?

Hello,
I am looking to use AWS API Gateway with my EKS cluster. I found this -> https://aws.amazon.com/blogs/containers/api-gateway-as-an-ingress-controller-for-eks/. I am looking for some feedback from folks if they have tried this out. In particular I would like to know if the AWS API Gateway when deployed as an EKS ingress, supports Custom Authorizers.

Kube2IAM or Kiam

you can check the closed PRs/issues

What are people doing to monitor worker health before it's connected to a control plane? Custom Cloudwatch metrics based on kubelet /healthz ? Something else?

Quick question for those who had to convert deprecated APIs when moving to Kubernetes v1.16:

Do I need to worry about converting ReplicaSets objects? My intuition says no and that I only need to worry about their corresponding Deployment objects.

Am I correct in assuming that?

@Somnath has joined the channel

Has CloudPosse come up with a turn-key way to do IAM Roles for Service Accounts (IRSA)? I need to start looking at doing that now for an EKS cluster we have

I can't speak for CloudPosse but I'm using it from the terraform-aws-eks module and it was pretty straight forward

Quick question - does anyone know if liveness probes continue to execute once a pod enters the Terminating state? If they do, and if they fail, will the pod be forcibly terminated and/or rescheduled? (https://github.com/kubernetes/kubernetes/issues/52817 looks somewhat related to my question)