#kubernetes - September 2018 | Slack Archive

kubernetes

143 messages

Archive: https://archive.sweetops.com/kubernetes/

Aug 2018

September 2018

Oct 2018

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/clems4ever/authelia/blob/master/README.md

Erik Osterman (Cloud Posse)over 7 years ago

Good alternative for bitly oauth2 proxy?

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/cmattoon/aws-ssm

Erik Osterman (Cloud Posse)over 7 years ago

@Michal Matyjek have you guys tried https://github.com/hypnoglow/helm-s3

Erik Osterman (Cloud Posse)over 7 years ago

Michal Matyjekover 7 years ago

we have not, the built-in codefresh one seems to be good enough for now

Ryan Smithover 7 years ago

to helm or not to helm... (as a n00b).. that is the question

Max Moonover 7 years ago

im a fan of helm, but i would suggest not using it until you're extra comfy with kubectl and writing manifests yourself

Max Moonover 7 years ago

it can feel like a bit of a black box, even if you wrote the charts yourself

Ryan Smithover 7 years ago

Ah, interesting

Erik Osterman (Cloud Posse)over 7 years ago(edited)

so we recently did an engagememnt with caltech students in a research lab

Erik Osterman (Cloud Posse)over 7 years ago

what worked well for them was to first write all the resources by hand to get comfy with it - the way @Max Moon describes

Erik Osterman (Cloud Posse)over 7 years ago

then write the charts later

Erik Osterman (Cloud Posse)over 7 years ago

also, i’d like to point out https://github.com/cloudposse/charts/tree/master/incubator/monochart

Erik Osterman (Cloud Posse)over 7 years ago

which is our declarative helm chart. this means it will work for the most common use-cases and you won’t need to write a custom chart. you just define all the settings in values.yaml

Max Moonover 7 years ago

the above is what i recommend as well, every company i've used helm at, i've used the same approach

Ryan Smithover 7 years ago

That's awesome--I appreciate the direction

Max Moonover 7 years ago

np!

Max Moonover 7 years ago

also think about your CICD tool(s) of choice

Max Moonover 7 years ago

for instance, last i used it, spinnaker was pretty opinionated and wanted control over deployments and deployment management, so we couldnt use helm

Max Moonover 7 years ago

that might have changed, but just something to keep in mind

Max Moonover 7 years ago

pretty much everything else should play along fine

Ryan Smithover 7 years ago

gotcha--yeah totally

Ryan Smithover 7 years ago

kind of doing some skunkwork at the moment

Ryan Smithover 7 years ago

Just want to get looking at the right stuff in the right way (that has worked for others)

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/mcuadros/terraform-provider-helm

Erik Osterman (Cloud Posse)over 7 years ago

terraform alternative for helmfile

Erik Osterman (Cloud Posse)over 7 years ago

hadn't seen that before, but looks to be identical in interface to chamber

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/segmentio/chamber

Rohit Vermaover 7 years ago

can someone point me towards changing timezone on kops created instances

Andriy Knysh (Cloud Posse)over 7 years ago

@Rohit Verma i think you can use https://github.com/kubernetes/kops/blob/master/docs/cluster_spec.md#hooks

Andriy Knysh (Cloud Posse)over 7 years ago

as described here https://github.com/kubernetes/kops/issues/1794

Ryan Smithover 7 years ago

Really liking cloudposse/charts/monochart

Ryan Smithover 7 years ago

https://github.com/cloudposse/charts/pull/171 😉

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/heptio/ark

Max Moonover 7 years ago

love ark

Erik Osterman (Cloud Posse)over 7 years ago

have you used it?

Max Moonover 7 years ago

yes indeedy

Erik Osterman (Cloud Posse)over 7 years ago

was it easy to get up and running?

Max Moonover 7 years ago

oh yeah

Erik Osterman (Cloud Posse)over 7 years ago

are you guys using it now?

Max Moonover 7 years ago

not yet

Max Moonover 7 years ago

it was the backup tool of choice at my last company, used it on every single cluster, took me a morning to put in place

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/helm/charts/tree/master/stable/ark

Max Moonover 7 years ago

whenever i had issues with it, the main guy, andy, was very quick to help

Max Moonover 7 years ago

they have a channel on the main k8s slack

Erik Osterman (Cloud Posse)over 7 years ago

https://www.zdnet.com/article/netapp-buys-stackpointcloud-launches-kubernetes-as-a-service/

Max Moonover 7 years ago

curious if anyone has given creating an Operator themselves a go: https://github.com/operator-framework

Max Moonover 7 years ago(edited)

chances are you are (either knowingly or unknowingly) already using one or many in your cluster

Erik Osterman (Cloud Posse)over 7 years ago

we're using all the prometheus operators in our latest rollouts

Erik Osterman (Cloud Posse)over 7 years ago

I would love to see a Terraform operator

Erik Osterman (Cloud Posse)over 7 years ago

haven't yet considered taking the plunge to write one

Erik Osterman (Cloud Posse)over 7 years ago

is there something in particular you want to build?

Max Moonover 7 years ago

nothing concrete yet, have been trying to think of some ideas

Erik Osterman (Cloud Posse)over 7 years ago

for context, we're moving this discussion here: https://sweetops.slack.com/archives/CB2PXUHLL/p1537386287000100

Max Moonover 7 years ago(edited)

I'd be curious to know, are you getting that composer error from the log from kubectl logs <pod name> or elsewhere?

Matthewover 7 years ago

Getting it from the container that is being build

Matthewover 7 years ago

built*

Matthewover 7 years ago

image: report-portal:develop built locally works 100% of the time. Deployed using this, somehow libraries are lost and dropped.

Erik Osterman (Cloud Posse)over 7 years ago

what happens if you pull and run that ECR image locally?

Matthewover 7 years ago

When i build the ECR image locally, same error is produced

Matthewover 7 years ago

But when I build the image that is being uploaded, it works smoothly

Matthewover 7 years ago

I have 0 idea why my ECR image would be corrupt but the one building that image works

Andriy Knysh (Cloud Posse)over 7 years ago

@Matthew please explain it again step-by-step for people to be able to help you, something like this…

Andriy Knysh (Cloud Posse)over 7 years ago

1. I have a Dockerfile (show it here) which I build locally and then start the container and it works locally

Andriy Knysh (Cloud Posse)over 7 years ago

2. Then I push the already built image to ECR manually

Andriy Knysh (Cloud Posse)over 7 years ago

3. When the image gets deployed from the ECR repo to Kubernetes, it throws errors

Andriy Knysh (Cloud Posse)over 7 years ago

explain what works and what does not, where you build it and how

Erik Osterman (Cloud Posse)over 7 years ago

https://aws.amazon.com/blogs/opensource/provision-aws-services-kubernetes-aws-service-broker/

Erik Osterman (Cloud Posse)over 7 years ago

“And that’s all folks” - wasn’t that easy? :P

Erik Osterman (Cloud Posse)over 7 years ago

Joking aside - pretty cool. Basically let’s you provision AWS backing services from within Kubernetes

Erik Osterman (Cloud Posse)over 7 years ago

H/T @Florian Drescher

Florian Drescherover 7 years ago

@Florian Drescher has joined the channel

Erik Osterman (Cloud Posse)over 7 years ago

We’ve released our EKS terraform modules for Kubernetes this week.
- https://github.com/cloudposse/terraform-aws-eks-cluster
- https://github.com/cloudposse/terraform-aws-eks-workers

Welcome feedback

Ryan Smithover 7 years ago

@stobiewankenobi 😏 ^^^

stobiewankenobiover 7 years ago

@stobiewankenobi has joined the channel

Ryan Smithover 7 years ago

it looks clean! 🙂

Ryan Smithover 7 years ago

@Erik Osterman (Cloud Posse) should post on Reddit if you don't already

Erik Osterman (Cloud Posse)over 7 years ago

Thanks @Ryan Smith yes - plan to do this soon 😃

Erik Osterman (Cloud Posse)over 7 years ago

I think @Andriy Knysh (Cloud Posse) is doing some more testing today

Jeremyover 7 years ago

Hi there, I was wondering whether the prometheus-to-cloudwatch solution can be adapted to scape metrics from the metrics server instead of kube-state-metrics.

Andriy Knysh (Cloud Posse)over 7 years ago

ohh, the module was created just as an experiment, tested a little bit, and then forgotten (meaning not supported anymore b/c there many more official solutions to do prometeus-to-cloudwatch) 🙂

Andriy Knysh (Cloud Posse)over 7 years ago

@Erik Osterman (Cloud Posse) can explain the whole situation

Jeremyover 7 years ago

thanks @Andriy Knysh (Cloud Posse). Can you direct me to where I can find those other solutions?

Jeremyover 7 years ago

I saw an exporter for exporting CW metrics to prometheus but not the other way around.

Andriy Knysh (Cloud Posse)over 7 years ago

https://github.com/prometheus/cloudwatch_exporter
https://groups.google.com/forum/#!topic/prometheus-developers/3n7n0PGG7Vw
https://medium.com/@griggheo/initial-experiences-with-the-prometheus-monitoring-system-167054ac439c

Jeremyover 7 years ago

So far, your solution was the only solution I found for exporting prometheus metrics to CW.

Andriy Knysh (Cloud Posse)over 7 years ago

@Erik Osterman (Cloud Posse) ^ 🙂

Andriy Knysh (Cloud Posse)over 7 years ago

who knew 🙂 https://groups.google.com/forum/#!topic/prometheus-developers/3n7n0PGG7Vw

Andriy Knysh (Cloud Posse)over 7 years ago

this should be done in a more official way by using Prometheus Operator

Andriy Knysh (Cloud Posse)over 7 years ago

this tool to do it was already mentioned https://github.com/operator-framework

Jeremyover 7 years ago

That makes sense.

Andriy Knysh (Cloud Posse)over 7 years ago

Operator will allow a much better integration with Prometheus

Andriy Knysh (Cloud Posse)over 7 years ago

(but I agree, our tool is simpler 😉 )

Andriy Knysh (Cloud Posse)over 7 years ago

but @Jeremy, what you asked (to scape metrics from the metrics server instead of kube-state-metrics) could be done by installing kube-prometheus and then scraping it, no?

Jeremyover 7 years ago

yeah, i guess that's what I'm asking. do i simply need to change the url in the values.yaml?

Jeremyover 7 years ago

(I haven't had time to look at the code you're using to scape the metrics yet)

Andriy Knysh (Cloud Posse)over 7 years ago

look at this https://github.com/kubernetes/kube-state-metrics#kube-state-metrics-vs-heapstermetrics-server

Andriy Knysh (Cloud Posse)over 7 years ago

Additionally, some monitoring systems such as Prometheus do not use Heapster(metrics-server) for metric collection at all and instead implement their own, but Prometheus can scrape metrics from heapster itself to alert on Heapster(metrics-server)’s health. Having kube-state-metrics as a separate project enables access to these metrics from those monitoring systems

Jeremyover 7 years ago(edited)

i'll have a look. I'm interested in surfacing the metrics described in this blog https://blog.freshtracks.io/a-deep-dive-into-kubernetes-metrics-part-4-the-kubernetes-api-server-72f1e1210770 to CW.

Jeremyover 7 years ago

i believe i should be able to get these without running the metrics server.

Andriy Knysh (Cloud Posse)over 7 years ago

yea, it’s a very convoluted topic

Andriy Knysh (Cloud Posse)over 7 years ago

metrics server is part of Kubernetes https://kubernetes.io/docs/tasks/debug-application-cluster/core-metrics-pipeline/#metrics-server

Andriy Knysh (Cloud Posse)over 7 years ago

so I think you need to install Prometheus (via kube-prometheus https://github.com/coreos/prometheus-operator/tree/master/helm/kube-prometheus for example ) and then will be able to scrape it using a scraping tool

Andriy Knysh (Cloud Posse)over 7 years ago(edited)

or via prometheus-operator chart https://github.com/coreos/prometheus-operator/tree/master/helm/prometheus-operator

Andriy Knysh (Cloud Posse)over 7 years ago

in our collection of helmfiles, we have examples on how to do it https://github.com/cloudposse/helmfiles/tree/master/helmfile.d

Erik Osterman (Cloud Posse)over 7 years ago

@Jeremy G (Cloud Posse) we've added better formatting for prometheus alerts. See this PR by @Igor Rodionov https://github.com/cloudposse/helmfiles/pull/48

Jeremy G (Cloud Posse)over 7 years ago

@Jeremy G (Cloud Posse) has joined the channel

Erik Osterman (Cloud Posse)over 7 years ago

Also, deploying Grafana dashboards with configmaps: https://github.com/cloudposse/helmfiles/pull/18

Andriy Knysh (Cloud Posse)over 7 years ago

I think it’s a different Jeremy :)

Erik Osterman (Cloud Posse)over 7 years ago(edited)

@Jeremy G (Cloud Posse) is with PopChest <--- using our older versions of kube-prometheus

Jeremy G (Cloud Posse)over 7 years ago

But, @Erik Osterman (Cloud Posse) There is also @Jeremy Cowan, who I think you meant to be referring to.

Erik Osterman (Cloud Posse)over 7 years ago

ahk

Erik Osterman (Cloud Posse)over 7 years ago

I see.

Erik Osterman (Cloud Posse)over 7 years ago

well, welcome to #kubernetes 😃

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/kubernetes/node-problem-detector/blob/master/README.md

Erik Osterman (Cloud Posse)over 7 years ago

node-problem-detector aims to make various node problems visible to the upstream layers in cluster management stack. It is a daemon which runs on each node, detects node problems and reports them to apiserver. node-problem-detector can either run as a DaemonSet or run standalone. Now it is running as a Kubernetes Addon enabled by default in the GCE cluster.

Erik Osterman (Cloud Posse)over 7 years ago

Works with draino

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/negz/draino

Erik Osterman (Cloud Posse)over 7 years ago

Draino automatically drains Kubernetes nodes based on labels and node conditions. Nodes that match all of the supplied labels and any of the supplied node conditions will be cordoned immediately and drained after a configurable drain-buffer time.

Erik Osterman (Cloud Posse)over 7 years ago

@Daren see this: https://github.com/heptiolabs/eventrouter

Erik Osterman (Cloud Posse)over 7 years ago(edited)

(this was the project I was thinking of... came across it today by accident looking at heptio projects)

Danover 7 years ago

i saw the eks tf files that were created

Danover 7 years ago

do we have anything for aws-config

Danover 7 years ago

aws-auth

Andriy Knysh (Cloud Posse)over 7 years ago

the eks project is being tested now and the modules will be updated to the latest version this week

Andriy Knysh (Cloud Posse)over 7 years ago

what are you doing with aws-config?

Danover 7 years ago

for aws-auth

Danover 7 years ago

i can authorize specific users to be able to make changes to the clusters and deploy environments

Erik Osterman (Cloud Posse)over 7 years ago

@Dan are you referring to EKS?

Max Moonover 7 years ago

Does anyone else follow this blog? https://blog.jessfraz.com/ tons of good stuff regarding containers, security, and k8s

Max Moonover 7 years ago

Same person is responsible for: https://github.com/genuinetools

Max Moonover 7 years ago

again, lots of really good/cool/interesting tools there too

Andriy Knysh (Cloud Posse)over 7 years ago

thanks for the links @Max Moon

Max Moonover 7 years ago

Erik Osterman (Cloud Posse)over 7 years ago

She’s awesome! I have been following her for years on twitter

Erik Osterman (Cloud Posse)over 7 years ago

https://docs.cloudposse.com/assets/4132caf-image_3.png

Erik Osterman (Cloud Posse)over 7 years ago

She runs Coreos on the desktop

Erik Osterman (Cloud Posse)over 7 years ago

Has containerized everything

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/jessfraz/dockerfiles

Erik Osterman (Cloud Posse)over 7 years ago

On an unrelated note...

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/gofunct/cloudnative-engineer/blob/master/CKA/README.md

Erik Osterman (Cloud Posse)over 7 years ago

Certified Kubernetes Administrator Exam Prep

August 2018 Browse by date October 2018