#kubernetes - May 2020 | Slack Archive

kubernetes

95 messages

Archive: https://archive.sweetops.com/kubernetes/

Apr 2020

May 2020

Jun 2020

Zachary Loeberabout 6 years ago

Passed the CKA, whoop

AugustasVabout 6 years ago

configuring ingress-nginx on kubernetes, but page still can't see the endpoints - backend is working, showing this is nginx page

AugustasVabout 6 years ago

404 Not Found

Omer Senabout 6 years ago

404 is being taken always if there is domainname mismatch

Omer Senabout 6 years ago

though nginx ingress has a default ingress

Omer Senabout 6 years ago

can you check pods logs or using stern

AugustasVabout 6 years ago(edited)

the thing is that it was working before, I just wanted to reinstall ingress. I can ping and reach services inside cluster

AugustasVabout 6 years ago

I only see 404 error, cant reach other pages. maybe nginx ingress doesnt use ingress file or something? logs show nothing

curious deviantabout 6 years ago

Hello,

Does anyone have experience using Rancher to manage their EKS clusters ? What I am specifically looking for is if there's a way for me to specify namespaces and resource quotas in a yaml etc. for my clusters and then feed that into Rancher ? I am just getting started with this tool and looks like everything happens through the UI thus far.

Omer Senabout 6 years ago

Hello may I ask what do you use to authenticate against AWS EKS so you can use kubectl etc. Is there an alternative to https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html ?

Omer Senabout 6 years ago

for example i want to use AD or SSO or what do you use generally.

Zachary Loeberabout 6 years ago

anyone here ever futz around with the Istio ingress deployment before?

btaiabout 6 years ago

can you increase memory limit on pods w/o triggering a restart?

btaiabout 6 years ago

or daemon set

Zachary Loeberabout 6 years ago

Not that I'm aware @btai That would be considered vertical pod autoscaling

Zachary Loeberabout 6 years ago

start your search with that term and let me know if I'm wrong (I'd rather be corrected than make wrong assumptions :))

btaiabout 6 years ago

what does everyone else do when their ingress controller is nearing a mem limit w/o incurring downtime 🤔

Erik Osterman (Cloud Posse)about 6 years ago(edited)

and no HPA?

btaiabout 6 years ago

personally it’s easy for us to provision a new cluster w/ new limits and move traffic to that

btaiabout 6 years ago

but i was wondering for those that don’t do ephemeral clusters

Erik Osterman (Cloud Posse)about 6 years ago

is every nginx-ingress pod reaching memory limit?

Erik Osterman (Cloud Posse)about 6 years ago

what's considered "downtime" in this situation? no request dropped?

btaiabout 6 years ago

yeah all my pods are close 😛

Erik Osterman (Cloud Posse)about 6 years ago

isn't it enough to ensure you have a rolling-update strategy in place and then change the limits?

Erik Osterman (Cloud Posse)about 6 years ago

that should be ~zero downtime (possibly some dropped connections)

Zachary Loeberabout 6 years ago

I would almost certainly deploy ingress with autoscaling moving forward

btaiabout 6 years ago

@Erik Osterman (Cloud Posse) you would think but we have had alot of connection errors doing tat in the past. will need to add some telemetry around it, very possible not at fault of the ingress controller

Erik Osterman (Cloud Posse)about 6 years ago

https://cloudposse.com/devops/pod-disruption-budget-gotchas/ by @Jeremy G

erikabout 6 years ago

EKS users:

bradymabout 6 years ago

I'm curious how others handle deploying things like an nginx config for a reverse proxy to a third-party endpoint for something like email click tracking or subdomain redirects. Do you just add it to the ingress-nginx config? Deploy a separate nginx pod with its own config file? Something else? Any input?

Scottabout 6 years ago

Does anyone have input or a link to what could be a checklist of tasks for setting up a basic EKS/k8s cluster? I'm generally bad at abstracting ideas away until I'm knee deep in them, I can only think of general tasks like 1) build nodes 2) user auth 3) logging 4) metrics collection

Scottabout 6 years ago

^^ This is with the presumption the vpc/infra is all setup

Zachary Loeberabout 6 years ago

steps: 1.) Build EKS cluster, 2.) ???, 3.) Profit!

Scottabout 6 years ago

^^ Pretty much hah

Andriy Knysh (Cloud Posse)about 6 years ago(edited)

assuming that you provisioned not only EKS cluster, but all the IAM roles and SSO stuff for humans (to be able to access the cluster) and for apps (EKS service account IAM roles), then we usually deploy these k8s releases:

Andriy Knysh (Cloud Posse)about 6 years ago(edited)

- `external-dns`
- `nginx-ingress`
- `cert-manager`
- `reloader`
- `metrics-server`
- `kubernetes-dashboard`
- `efs-provisioner`
- `aws-secret-operator`
- `prometheus stuff

timduhenchanterabout 6 years ago

Anyone have experience exposing status.hostIP to a third-party application to hit a DaemonSet that does not natively support referencing environment variables with Downward API (use-case: Datadog Agent and Kong)? https://github.com/kubernetes/kubernetes/issues/74265 asked for this to be revisited but in the meantime…

Zachary Loeberabout 6 years ago

A new blog I dropped recently about screwing around with Istio on k3d and kind: https://zacharyloeber.com/2020/05/the-istio-rabbithole/

Joeyabout 6 years ago(edited)

Hey guys, devops noob here...

I followed this tutorial to dockerize a react app https://mherman.org/blog/dockerizing-a-react-app/
and it turns out that -it (interactive mode) is required for docker run commands

Now I'm trying to follow this tutorial to deploy via Kubernetes dashboard https://www.youtube.com/watch?time_continue=279&v=je5WRKxOkWQ
My app keeps crashing with the same error that occurs if I don't add -it to docker run

So does anyone here know how I could add -it when deploying via Kubernetes dashboard?

Joeyabout 6 years ago

@Andriy Knysh (Cloud Posse)

Josephs-MacBook-Air:web josephbennett$ docker-compose up -d --build
Building sample
Step 1/9 : FROM node:13.12.0-alpine
 ---> 483343d6c5f5
Step 2/9 : WORKDIR /app
 ---> Using cache
 ---> 961768ca865e
Step 3/9 : ENV PATH /app/node_modules/.bin:$PATH
 ---> Using cache
 ---> 8c67f044ee11
Step 4/9 : COPY package.json ./
 ---> Using cache
 ---> 38c4fc32e5b5
Step 5/9 : COPY package-lock.json ./
 ---> Using cache
 ---> 1e03505a795d
Step 6/9 : RUN npm install --silent
 ---> Using cache
 ---> 4c69c439a90f
Step 7/9 : RUN npm install react-scripts@3.4.1 -g --silent
 ---> Using cache
 ---> 8de2388a9ba5
Step 8/9 : COPY . ./
 ---> Using cache
 ---> 2cc9fce978f4
Step 9/9 : CMD ["npm", "start"]
 ---> Using cache
 ---> 1b726fc6c6ce

Successfully built 1b726fc6c6ce
Successfully tagged web_sample:latest
my-react-app is up-to-date
Josephs-MacBook-Air:web josephbennett$ docker run web_sample:latest

> my-app@0.1.0 start /app
> react-scripts start

ℹ ｢wds｣: Project is running at <http://172.17.0.2/>
ℹ ｢wds｣: webpack output is served from 
ℹ ｢wds｣: Content not from webpack is served from /app/public
ℹ ｢wds｣: 404s will fallback to /
Starting the development server...

Josephs-MacBook-Air:web josephbennett$ docker run -it web_sample:latest

Joeyabout 6 years ago

^ it works only after that last line

Joeyabout 6 years ago

& that's what I can't seem to replicate in the Kubernetes console, just figuring out how to add the -it

Joeyabout 6 years ago

thanks for the help btw

Maarten van der Hoefabout 6 years ago(edited)

@Joey The documentation you posted https://mherman.org/blog/dockerizing-a-react-app/ explains at "What’s happening here?" #2

-it starts the container in interactive mode. Why is this necessary? As of version 3.4.1, react-scripts exits after start-up (unless CI mode is specified) which will cause the container to exit. Thus the need for interactive mode.

You will need to enable the ci mode and then it will work without -it, you can do that by setting the env variable CI to true.

Joeyalmost 6 years ago

Hey guys, do you know if it's possible to specify the DNS whenever you create a new service in Kubernetes? If not, then what's a stable way to stay routed if your service fails?

e.g. let's say we create a service with the DNS abc123-456us-east-2.elb.amazonaws.com:3000, and we have our URL www.mysite.com pointing to it... Then we create a backup service with the DNS def456-789us-east-2.elb.amazonaws.com:3000. Now we have to take the time to go on godaddy or wherever, to direct www.mysite.com to point to our backup DNS...

Sorry if this is a super noob question, but is there something that I can do in case our service fails? Or should I just assume that the service will live on?

Chris Fowlesalmost 6 years ago

move your dns to something like route53 for a start

Chris Fowlesalmost 6 years ago

then you can use something like https://github.com/kubernetes-sigs/external-dns

joeyalmost 6 years ago

and then you can do even more fancy stuff like using weighted records and associating healthchecks with cname's, because you probably want to have www.mysite.com CNAME to something like point to primariy.cluster.mysite.com weight 100 and backup.cluster.mysite.com weight something else each with their independent healthchecks and maybe you want to delegate only cluster.mysite.com to route 53 instead of your whole domain

joeyalmost 6 years ago

i probably wouldn't want to point my mysite.com cnames directly to my clb/alb/nlb's because they're probably going to get clobbered at some point and the beauty of alb-ingress-controller and external-dns is that'll all just magically work 🎉

Joeyalmost 6 years ago

thanks for all the suggestions guys, I'll do the fancy stuff later. I'm googling what a route 53 is and how to move my DNS to it 😅 thanks!

joeyalmost 6 years ago(edited)

just be sure to do all your testing in production 😂 (LOTS OF SARCASM HERE)

Chris Fowlesalmost 6 years ago

i'm having trouble keeping my joey's straight here

Joeyalmost 6 years ago

You can just call me deadpool

Pierre Humberdrozalmost 6 years ago

Interesting Project:
https://github.com/spotahome/service-level-operator

dalekurtalmost 6 years ago

Has anyone done feature branch deployments within AWS EKS? I'm planning on doing this, but I see a potential issue with AWS ALB route limits.

AugustasValmost 6 years ago

Want to create nodegroup via eksctl create nodegroup command, got error

Error: timed out (after 25m0s) waiting for at least 1 nodes to join the cluster and become ready in "standard-workers"

any ideas? It didnt appear in AWS EKS panel
Error: timed out (after 25m0s) waiting for at least 1 nodes to join the cluster and become ready in "

Erik Osterman (Cloud Posse)almost 6 years ago

---

Erik Osterman (Cloud Posse)almost 6 years ago

https://github.com/awslabs/amazon-eks-serverless-drainer

Joeyalmost 6 years ago(edited)

Does anyone here know of an easy way for me to pull from a private docker repo, via dashboard?

Joeyalmost 6 years ago(edited)

I keep getting permission denied and idk how to log in to docker on my kubernetes dashboard

Joeyalmost 6 years ago(edited)

nvm figured it out!

kubectl create secret docker-registry regcred --docker-server=<http://docker.io/|docker.io/><user>/<image> --docker-username=<your-username> --docker-password=<your-pword> --docker-email=<your-email>

then in the deployment's .yaml:

      imagePullSecrets:
        - name: regcred

Erik Osterman (Cloud Posse)almost 6 years ago

https://cdk8s.io/

Joeyalmost 6 years ago

Hey guys, I'm trying to make my web page only viewable by me. So I'm modifying the ELBs security group....

This works:

Type - All Traffic
Protocol - All
Port range - All
Destination - 0.0.0.0/0

This doesn't work:

Type - All Traffic
Protocol - All
Port range - All
Destination - <my ip address>/32

& I don't know why. This ELB was generated from Kubernetes, so is there anything that I can do in that application?

Neil Gealyalmost 6 years ago

Anybody have experience with pods that are crashing due to heap out of memory, and had to try to capture the core dump?

Erik Osterman (Cloud Posse)almost 6 years ago

https://news.google.com/articles/CBMigQFodHRwczovL3RlY2hjcnVuY2guY29tLzIwMjAvMDUvMTQvdmVuYWZpLWFjcXVpcmVzLWpldHN0YWNrLXRoZS1zdGFydHVwLWJlaGluZC10aGUtY2VydC1tYW5hZ2VyLWt1YmVybmV0ZXMtY2VydGlmaWNhdGUtY29udHJvbGxlci_SAYUBaHR0cHM6Ly90ZWNoY3J1bmNoLmNvbS8yMDIwLzA1LzE0L3ZlbmFmaS1hY3F1aXJlcy1qZXRzdGFjay10aGUtc3RhcnR1cC1iZWhpbmQtdGhlLWNlcnQtbWFuYWdlci1rdWJlcm5ldGVzLWNlcnRpZmljYXRlLWNvbnRyb2xsZXIvYW1wLw?hl=en-US&gl=US&ceid=US%3Aen

Sergeyalmost 6 years ago(edited)

Hi all!
What do you advise me, guys, for rotating logs that are being stored in hostPath directory? After pod is terminated, logs retain on a node and will be removed only after node is
terminated. Currently we are not able to change log streams to stdout, but we are working on it.

Erik Osterman (Cloud Posse)almost 6 years ago

https://blog.pipetail.io/posts/2020-05-04-most-common-mistakes-k8s/

Adefemi Michealalmost 6 years ago

Hi all, does anyone know what could lead to podinitializing state even after increasing storage, more cpu and memory?

sahil kambojalmost 6 years ago

Hey Guys
Need help regarding traffic load
i have websites running on K8 which are sql oriented , need to test how they will behaves in heavy user load and how my cluster handle that(need to upgrade node or not)
how should i implement it and what tools should i use

Pierre Humberdrozalmost 6 years ago

this sucks..

 rpc error: code = Unknown desc = Error response from daemon: Get <https://quay.io/v2/pusher/oauth2_proxy/manifests/v4.0.0>: received unexpected HTTP status: 500 Internal Server Error

How are you all managing external docker images ? Are you putting them in your own registry?

Zachary Loeberalmost 6 years ago

I converted localstack to run in kubernetes for locally testing out AWS scripts on kind clusters. Example includes the use of kompose, helmfile, the raw helm chart, and my own little framework for stitching it all together. https://zacharyloeber.com/2020/05/aws-testing-with-localstack-on-kubernetes/

Erik Osterman (Cloud Posse)almost 6 years ago

wow, rad idea

Mike F.almost 6 years ago

Super cool stuff!

Erik Osterman (Cloud Posse)almost 6 years ago

https://cloud.google.com/kubernetes-engine/docs/add-on/config-sync/overview

Zachary Loeberalmost 6 years ago

well that's useful. kind of like argocd or flux?

Matt Gowiealmost 6 years ago

Hey k8s folks — Is there a defined best place to get spun up on k8s? I’ve put off ramping up on the topic for a while, but I’d like to dive in while I have some downtime from clients. I’m sure some folks here have good resources or strong opinions on where to head for info.

Andrew Rothalmost 6 years ago

Andrew Rothalmost 6 years ago(edited)

Can I get a dummy check on my plan for deploying kiam to my K8s cluster?

1. Add an additional node pool of small instances to each cluster. 1-2 instances is really all that is needed
2. Apply the Instance Profile to the new node pool
3. Apply a [Taint](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) to the new nodes that tells the cluster not to schedule any pods to them
4. Deploy the [kiam](https://github.com/uswitch/kiam) server to the new nodes using a [Toleration](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/)
5. Deploy the kiam agent to all nodes
6. Annotate namespaces that are allowed to use IAM with the <http://iam.amazonaws.com/permitted|iam.amazonaws.com/permitted>: <regex that matches allowed roles> annotation
7. Annotate pods inside the permitted namespaces with <http://iam.amazonaws.com/role|iam.amazonaws.com/role>: <role name>

Zachary Loeberalmost 6 years ago

seems quay is down again 😞

Zachary Loeberalmost 6 years ago

Are there any projects out there that make it easy to report on all of the cached images in a cluster, pull them into some other registry, then apply a mutating admission webhook to rewrite the container registry source when the deployments get applied to point to the new image source?

Zachary Loeberalmost 6 years ago

I guess that is not 'easy' at all but there should be such a project if not to help avoid things like this

Zachary Loeberalmost 6 years ago

I'd specifically target things like cert-manager, kafka, or any other vendor(ish) images that would typically go through a review and testing process before simply upgrading them (so core services in a deployment, not developer workloads that might get updated multiple times a day)

Zachary Loeberalmost 6 years ago

I have peeked over kube-fledged and it seems to be on the right path towards something like this https://github.com/senthilrch/kube-fledged

Zachary Loeberalmost 6 years ago

(too bad there isn't a simple helm chart deployment for the thing, it is either straight yaml or the flux helm-operator it seems....)

Zachary Loeberalmost 6 years ago

In fact, that is the only project I've seen that allows for any direct image cache manipulation within a cluster

btaialmost 6 years ago

wonder what they’re doing at quay to be causing this many major outages

Erik Osterman (Cloud Posse)almost 6 years ago

quay acquired by coreos

Erik Osterman (Cloud Posse)almost 6 years ago

coreos acquired by redhat

Erik Osterman (Cloud Posse)almost 6 years ago

coreos (the os) now EOL

Erik Osterman (Cloud Posse)almost 6 years ago

quay the registry? not sure, but seeing how acquisitions go - the project leads are probably all gone. the project itself open sourced. who knows? maybe it's on life support?

Erik Osterman (Cloud Posse)almost 6 years ago

#codefresh decide to deprecate their registry

Erik Osterman (Cloud Posse)almost 6 years ago

my guess is these registries cost a fortune to operate

Zachary Loeberalmost 6 years ago

I imagine they would eat bandwidth and cloud storage space like no tomorrow

Zachary Loeberalmost 6 years ago

I've seen builds based on upstream public images as the base image before and cried on the inside (then promptly eliminated those outside deps...)

btaialmost 6 years ago

i feel like many of the major kubernetes tools have been hosting their images on quay.

April 2020 Browse by date June 2020