#kubernetes - February 2020 | Slack Archive

109 messages

Archive: https://archive.sweetops.com/kubernetes/

Jan 2020

February 2020

Mar 2020

Chris Fowlesover 6 years ago

So i had a thought on the train this morning when I was thinking of writing some yoman generators for templating out gitops repos for flux so that teams don't need to know exactly where to put what in which file.

Chris Fowlesover 6 years ago

And then i thought "you know what's really good at templating yaml? helm!"

Chris Fowlesover 6 years ago

Is it crazy to use helm template to generate repos for flux (which will create flux helm crds)? I can't see anything really stupid at face value - but feels like the kind of idea that requires a sanity check

Zachary Loeberover 6 years ago

I wouldn't think so. Jenkins-x does exactly that if you opt to not use tiller for deployments I believe.

Chris Fowlesover 6 years ago

writing helm charts and wanting some instant feedback ended up with this to check my rendered templates: watch -d 'helm template my-chart-dir | kubeval && helm template my-chart-dir'

Erik Osterman (Cloud Posse)over 6 years ago

nice trick

Roderik van der Veerover 6 years ago

I’m a bit stuck with GKE persistent disks that cannot be found:

│  Type     Reason              Age   From                     Message                                                                                                                                                                                                                                                                                │
│  ----     ------              ----  ----                     -------                                                                                                                                                                                                                                                                                │
│  Normal   Scheduled           73s   default-scheduler        Successfully assigned coral-firefly/coral-firefly-ipfs-ipfs-0 to gke-shared-europe-main-3d862732-mq45                                                                                                                                                                                  │
│  Warning  FailedAttachVolume  73s   attachdetach-controller  AttachVolume.Attach failed for volume "pvc-c018de57-476a-11ea-af39-42010a8400a9" : GCE persistent disk not found: diskName="gke-shared-europe-9030-pvc-c018de57-476a-11ea-af39-42010a8400a9" zone="europe-west1-b"

Roderik van der Veerover 6 years ago(edited)

while the disk is just there in the google console

Roderik van der Veerover 6 years ago

https://www.dropbox.com/s/nkyv4uhrad4p6zw/Screenshot%202020-02-04%2017.49.43.png?dl=0

Roderik van der Veerover 6 years ago

Not sure where to look for an error message i can actually use

Roderik van der Veerover 6 years ago

the pvc even says that it is bound

Roderik van der Veerover 6 years ago

ipfs-storage-coral-firefly-ipfs-ipfs-0               Bound    pvc-c018de57-476a-11ea-af39-42010a8400a9   10Gi       RWO            standard       27m

Roderik van der Veerover 6 years ago

anyone have an idea where to look?

Roderik van der Veerover 6 years ago

hmmm, seems like provisioning too many disks at the same time is the culprit. is there a way to configure that in k8s, helm or helmfile?

Erik Osterman (Cloud Posse)over 6 years ago

Are you running helmfiles serially?

Roderik van der Veerover 6 years ago

i recently moved to a lot more parallel to speed it up (we deploy clusters + services as a service, it needs to be as fast as possible)

Ryan Smithover 6 years ago(edited)

https://www.reddit.com/r/Terraform/comments/axp3tv/run_terraform_under_kubernetes_using_an_operator/ <--- nice!!!!

@Erik Osterman (Cloud Posse) do you have a recommendation for using a k8s operator to manage aws resources?
https://github.com/amazon-archives/aws-service-operator has been archived and uses cfn under the covers. Lol. Have you had luck with this or anything else?

Erik Osterman (Cloud Posse)over 6 years ago

@Ryan Smith not from first hand accounts, however, there have been a number of new operators to come out to address this lately

Erik Osterman (Cloud Posse)over 6 years ago

I did see the aws-service-operator was deprecated the other day when I was checking something else out.

Erik Osterman (Cloud Posse)over 6 years ago

sec - I think we talked about it recently in office hours

Erik Osterman (Cloud Posse)over 6 years ago

https://sweetops.slack.com/archives/CHDR1EWNA/p1579117464023000

Erik Osterman (Cloud Posse)over 6 years ago

What's interesting "outwardly speaking" regarding kubeform is that it is by the people behind appscode

Erik Osterman (Cloud Posse)over 6 years ago

appscode is building operators for managing all kinds of durable/pesistent services under kubernetes

Erik Osterman (Cloud Posse)over 6 years ago

like https://kubedb.com/

Erik Osterman (Cloud Posse)over 6 years ago

I don't have any first hand accounts though of using it or other appscode services

Ryan Smithover 6 years ago

Ah, very nice reference

Erik Osterman (Cloud Posse)over 6 years ago

Rancher has 2 projects they are working on.

Erik Osterman (Cloud Posse)over 6 years ago

Both are alpha grade and don't know how practical

Erik Osterman (Cloud Posse)over 6 years ago

https://github.com/rancher/terraform-controller

Erik Osterman (Cloud Posse)over 6 years ago

https://github.com/rancher/terraform-operator

Erik Osterman (Cloud Posse)over 6 years ago

ohhhhh

Erik Osterman (Cloud Posse)over 6 years ago

they renamed it

Erik Osterman (Cloud Posse)over 6 years ago

never mind.

Ryan Smithover 6 years ago

My feelings were hurt by rancher's product in 2016. I'm sure they've improved vastly since then though, lol. 🤣

Ryan Smithover 6 years ago

Yeah, II'm curious about the kubeform one

Ryan Smithover 6 years ago

will probably give that a try

Ryan Smithover 6 years ago

very nice references man

Erik Osterman (Cloud Posse)over 6 years ago

https://github.com/danisla/terraform-operator

Ryan Smithover 6 years ago

this is why cloudposse slack is best-in-class 🙌

Ryan Smithover 6 years ago

Yeah, I saw that one too

Ryan Smithover 6 years ago

Just have no clue what people are actually usinig

Erik Osterman (Cloud Posse)over 6 years ago

Please report back if you get a chance to prototype/poc one of them.

Erik Osterman (Cloud Posse)over 6 years ago

I've wanted to do this for some time.

Erik Osterman (Cloud Posse)over 6 years ago

I think it could simplify deployments in many ways by using the standard deployment process with kubernetes

Ryan Smithover 6 years ago

this is actually a part of a bigger discussion around ephemeral environments

Erik Osterman (Cloud Posse)over 6 years ago

Ryan Smithover 6 years ago

all the helmfile stuff is super straight forward

Ryan Smithover 6 years ago

it's the infrastructure beyond it tho

Erik Osterman (Cloud Posse)over 6 years ago

so while I love terraform, terraform is like a database migration tool that doesn't support transactions

Erik Osterman (Cloud Posse)over 6 years ago

so when stuff breaks, no rollbacks

Ryan Smithover 6 years ago(edited)

heh, yea

Erik Osterman (Cloud Posse)over 6 years ago

while webapps are usually trivial to rollback

Erik Osterman (Cloud Posse)over 6 years ago

so coupling these two might cause more instability where before there was none.

Erik Osterman (Cloud Posse)over 6 years ago

also, this is what was kind of nice (perhaps) with the aws-service-operator approach deploying cloudformation since cloudformation fails more gracefully

Erik Osterman (Cloud Posse)over 6 years ago

did you listen to this weeks office hours recording?

Erik Osterman (Cloud Posse)over 6 years ago(edited)

I think it's relevant to the ephemeral preview environments conversation, which is also one of the reasons were exploring terraform workspaces more and more as one of the tricks to help solve it.

johncblandiiover 6 years ago

Anyone address kubelet-extra-args with eks node groups?

Ryan Smithover 6 years ago(edited)

Thought experiment..

Is there a fundamental difference between slow+smart rolling deployment vs. canary deployment?

Ryan Smithover 6 years ago

Curious if I can hack the health checks of a rolling deployment to convert the behavior into a canary deployment

Erik Osterman (Cloud Posse)over 6 years ago

Canary can be restricted (e.g. with feature flags) to a segment of users.

Erik Osterman (Cloud Posse)over 6 years ago

of course, it could be defined that that segment is just some arbitrary % of users based on the number of updated pods

Erik Osterman (Cloud Posse)over 6 years ago

however, that can lead to inconsistent behavior

Erik Osterman (Cloud Posse)over 6 years ago

I think it's hard to generalize for all use-cases, but some kind of "canary" style rolling deployment can be achieved using "max unavailable"

Ryan Smithover 6 years ago

people always harp on canaries around "well you gotta make sure you're running health checks on the right stuff.. p99, queue depth, DD metrics"

Erik Osterman (Cloud Posse)over 6 years ago

ya, btw, you've seen flagger, right?

Ryan Smithover 6 years ago

sounds familiar, but I'll just say no

Julian Gindiover 6 years ago

Has anyone run into issues with downtime when nginx pods are terminated? I am testing our downtime during rollouts and are noticing a small blip with our nginx pods. Our normal API pods cause no downtime when rolled. Anyone have experience with this?

grvover 6 years ago

Hey all, I have ran into this problem before and wanted to know if someone has a good solution for it. I run kube clusters using KOPS. For infrastructure (VPC), I create them using cloudposse terraform scripts, route 53 stuff and all using tf as well. Then I create kube cluster using KOPS. While everything works fine, I always run into trouble when deleting kube clusters

grvover 6 years ago

never mind, I found solution to my own problem 😄

Alex Siegmanover 6 years ago

I use kops, but haven't been in the habit of deleting clusters; out of curiosity was it just some setting or config somewhere?

grvover 6 years ago

so, when I setup my infra (vpc, route 53 etc) using old tf versions, there were some documented issues in github that the kops delete cluster was trying to delete everything, but then used to fail cz resources like VPC was created using Terraform. In an ideal scenario, kops delete should not even look at those resources

grvover 6 years ago

Whatever new stuff i put in (vpc etc) using Cloudposse, is working well (kops delete only deleting resources such as security groups, nodes, masters and other instance groups) and not touching VPC config, Elastic IP's and all

grvover 6 years ago

My guess is, whatever we put up using standard terraform earlier, has incorrect tags or something attached, becuase those resources like vpc, does come in the list opf resources to delete when i do kops delete cluster without the --yes flag, to validate what all will be deleted

johncblandiiover 6 years ago

Has anyone seen their a pod not be able to talk to itself?

Context: Jenkins on K8s runs perfectly fine for building images, but the rtDockerPush call does not allow a connection.

<http://java.net|java.net>.ConnectException: connect(..) failed: Permission denied
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from ip-10-60-177-129.us-west-2.compute.internal/10.60.177.129:34956
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
		at hudson.remoting.Channel.call(Channel.java:955)
		at org.jfrog.hudson.pipeline.common.docker.utils.DockerAgentUtils.getImageIdFromAgent(DockerAgentUtils.java:292)
		at org.jfrog.hudson.pipeline.common.executors.DockerExecutor.execute(DockerExecutor.java:64)
		at org.jfrog.hudson.pipeline.declarative.steps.docker.DockerPushStep$Execution.run(DockerPushStep.java:96)
		at org.jfrog.hudson.pipeline.declarative.steps.docker.DockerPushStep$Execution.run(DockerPushStep.java:71)
		at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
		at hudson.security.ACL.impersonate(ACL.java:290)
		at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Caused: io.netty.channel.AbstractChannel$AnnotatedConnectException: connect(..) failed: Permission denied: /var/run/docker.sock

Andrew Rothover 6 years ago

I got tired of reading the stupid guide every time I wanted to spin up Rancher on my local machine so I made a thing!

https://github.com/RothAndrew/k8s-cac/tree/master/docker-desktop/rancher

dalekurtover 6 years ago

@Karoline Pauls You’re trying to determine what those values should initially, or how much headroom to between request and limits?

dalekurtover 6 years ago

That’s pretty good. I would be interested in hearing other methods

Zachary Loeberover 6 years ago

I'd give https://github.com/FairwindsOps/goldilocks a whirl. It uses a controller to deploy the VPA in recommendation mode on labeled namespaces' deployments to attempt to calculate required resources for pods.

Erik Osterman (Cloud Posse)over 6 years ago

Neat! Hadn't seen goldilocks before

dalekurtover 6 years ago

@Zachary Loeber If you are in NY or ever in NY I owe you a beer

Zachary Loeberover 6 years ago

ha, maybe if I actually authored the thing. I'm just a toolbag good sir. One tip with the chart since we are talking about it, don't sweat setting up VPA ahead of time (there isn't a good chart for it anyway), just use the InstallVPA: true value and it will run the deployment for the VPA for you. Also, the github repo docs are a bit out of date. The actual helm repo for goldilocks is stable and at https://charts.fairwinds.com/stable

kskewesover 6 years ago

We always set same value for memory as not compressible or shareable. Headroom depends on app peak usage then healthy bit more as memory is cheap.
CPU moved everything to burst with 3x.
Because CFS issue causing high P99, limit unnecessary throttling, drive higher usage of worker nodes, spikey nature of requests.

Andreasover 6 years ago

What are you guys using for multi cluster monitoring?

Pierre Humberdrozover 6 years ago

Prometheus. 1x Cluster Internal Prometheus for each cluster -> External Prometheus capturing all other prometheus via federation.

Zachary Loeberover 6 years ago

Interesting article on pros/cons of various kube cluster architecture strategies: https://learnk8s.io/how-many-clusters

BitsnBitesover 6 years ago

Hi all. Any ideas on why we get basically a 503 trying to expose the port for WebStomp with RabbitMQ? It has us perplexed. Here is the YAML;

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
field.cattle.io/creatorId: user-vnwtv
field.cattle.io/ingressState: '{"c3RvbXAvcmFiYml0bXEtaGEvc3RvbXAudGV4dHJhcy5jb20vL3N0b21wLzE1Njc0":""}'
field.cattle.io/publicEndpoints: '[{"addresses":["xxx.xx.xxx.x"],"port":80,"protocol":"HTTP","serviceName":"rabbitmq-ha:rabbitmq-ha-prod","ingressName":"rabbitmq-ha:stomp","hostname":"stomp.ourdomain.com","path":"/stomp","allNodes":true}]' nginx.org/websocket-services: rabbitmq-ha-prod creationTimestamp: "2020-01-29T05:39:40Z" generation: 2 labels: cattle.io/creator: norman name: stomp namespace: rabbitmq-ha resourceVersion: "1653424" selfLink: /apis/extensions/v1beta1/namespaces/rabbitmq-ha/ingresses/stomp uid: f15178c6-3d10-4202-adbd-d9d1cc123bc5 spec: rules: - host: stomp.ourdomain.com http: paths: - backend: serviceName: rabbitmq-ha-prod servicePort: 15674 path: /stomp status: loadBalancer: ingress: - ip: xxx.xx.xxx.x

It looks as though the RMQ http API used by our NodeJS code is fine as I see exchanges being created. So there is service running on that port.

BitsnBitesover 6 years ago

Using Rancher btw

Roderik van der Veerover 6 years ago

Anyone managed to get nginx-ingress on GKE to work when both exposing custom TCP and UDP ports? It complains about LoadBalancer not being able to mix protocols

Erik Osterman (Cloud Posse)over 6 years ago

Alright, time to level up. Inviting @UUB28NLDS !

btaiover 6 years ago

@UUB28NLDS help

Erik Osterman (Cloud Posse)over 6 years ago

anyone using kudo?

Erik Osterman (Cloud Posse)over 6 years ago

https://kudo.dev/

Andrew Rothover 6 years ago

@UUB28NLDS winning

curious deviantover 6 years ago

Hello Kubernetes Experts,

We have a K8s Deployment that scales up in response to the number of messages in a queue using HPA. There is a requirement to have the HPA scale up the pods however the scaling down must be handled by some custom code. Is it possible to configure the HPA to only handle the scale-up ?

Zachary Loeberover 6 years ago

https://cloud.google.com/kubernetes-engine/docs/best-practices/enterprise-multitenancy#checklist <- A worthy checklist regardless of the cloud vendor

Zachary Loeberover 6 years ago

@curious deviant I'm reading it that you don't want autoscaling, you want scale up. If you have the scale down being handled via custom code what is preventing you from wrapping that all into a controller of your own to handle both? I know that you can disable cluster autoscale down with an annotation now, not certain about HPA though

Zachary Loeberover 6 years ago

Anyone read through this nifty guide yet? https://kubernetes-on-aws.readthedocs.io/en/latest/index.html

btaiover 6 years ago

I understand the nature of k8s is to be able to deploy onto different underlying platforms with ease, but any gotchas for peeps that made the move from kops on AWS to EKS?

Zachary Loeberover 6 years ago

@btai Technically, if the hype is to be believed, you just went to an easier to manage cluster 🙂. I'm not going to profess to be an aws eks master but I'd say that possibly managing your own storage maybe simplified by using ebs or efs within eks for your persistent volumes. On the other hand, you may see those storage costs go up if you were using local ec2 instance stores in your kops clusters (presented using something like minio) for high disk IO driven workloads and now are using EKS EBS presented storage instead.

Zachary Loeberover 6 years ago

I may be way off though and welcome corrections by smarter people in this channel

Hemanthover 6 years ago

anyone running kubernetes on raspberry pi's here ? just have few newbie questions - my intention is just for the fun/kicks on it for weekends

Maheshover 6 years ago

Hello all, I would like to set nologin in one of Deploy agent Pod so that no one can do kubectl exec --- Is there a way we can achieve this without having to look at k8s RBAC? I want to avoid k8s RBAC because we use EKS platform and not using RBAC extensively yet.

Zachary Loeberover 6 years ago

afaik autoscaler does not auto-apply node taints from the last time I looked into it on aks clusters.

Zachary Loeberover 6 years ago

but I was not being extremely exhaustive on what I was doing either, I found that using taints for steering deployments was not super intuitive and leaned into using node affinities to force preferences for workload node pools/groups instead.

Zachary Loeberover 6 years ago

annotations seemed to carry over to autoscaled nodes just fine

Zachary Loeberover 6 years ago

Is anyone out there building up managed kube clusters, running a workload, then tearing them down as part of a deployment pipeline?

Erik Osterman (Cloud Posse)over 6 years ago

@btai ☝️

btaiover 6 years ago

@Zachary Loeber i’m doing similar. but not just running a workload. our applications live on ephemeral clusters, when we are upgrading to a new k8s version we spin up a new cluster (i.e. 1.11 -> 1.12, CVEs) and for other things like helm3 migrations

Zachary Loeberover 6 years ago

@btai So you spin up the cluster, baseline config it, then kick off some deployment pipelines to shove code into it?

January 2020 Browse by date March 2020