#sre - February 2020 | Slack Archive

sreArchived

25 messages

Archive: https://archive.sweetops.com/monitoring/

Jan 2020

February 2020

Mar 2020

Erik Osterman (Cloud Posse)over 6 years ago

https://www.transposit.com/blog/2020.01.30-writing-runbook-documentation-when-youre-an-sre/

btaiover 6 years ago(edited)

those of you using efs-provisioner helmfile for prometheus on k8s, did you need to install nfs-utils on your worker nodes? @Andrew Roth @Erik Osterman (Cloud Posse)

Erik Osterman (Cloud Posse)over 6 years ago

Nope - here's our kops manifest: https://github.com/cloudposse/reference-architectures/blob/master/templates/kops/kops-private-topology.yaml.gotmpl

Erik Osterman (Cloud Posse)over 6 years ago

(we're using kops debian)

Zachary Loeberover 6 years ago

Interesting site idea: https://awesome-prometheus-alerts.grep.to/

Zachary Loeberover 6 years ago

So if I do happen to get apiserver high utilization alerts, how would I track down what might be causing it?

joshmyersover 6 years ago

“apiserver high utilization alerts” is pretty vague…

Zachary Loeberover 6 years ago

Yeah, you are telling me. I got prometheus alerts for it and found I couldn't complete kubectl commands but couldn't really find a cause of the issues. Even looking through local host logs for kubelet was rather futile (which feels like defeating the purpose of having managed kubernetes to me btw)

Zachary Loeberover 6 years ago

finally going on a hunch I rebooted one of the nodes and stabilized the api services. Solving things by gut instinct always feels kinda wrong though 🙂

joshmyersover 6 years ago

Well, given I have 0 context one your setup/app/what you are talking about

joshmyersover 6 years ago

First step would be to go check what metrics that alert is looking at…

joshmyersover 6 years ago

Should be pretty obvious at least where the smell is coming from

joshmyersover 6 years ago

Maybe not the root cause

Zachary Loeberover 6 years ago

alert: APIServerErrorsHigh
expr: rate(apiserver_request_count{code=~"^(?:5..)$"}[5m])
  / rate(apiserver_request_count[5m]) * 100 > 5
for: 10m
labels:
  severity: critical
annotations:
  description: API server returns errors for {{ $value }}% of requests

Zachary Loeberover 6 years ago

My question is how can I tell what (if anything) within a cluster might be causing a high request rate to the kubernetes api services?

Zachary Loeberover 6 years ago

I have a feeling that a controller or deployment was going haywire but there must be a quicker way to isolate such deployments I'd think

Zachary Loeberover 6 years ago

here is another prometheus rule that was triggering:

Zachary Loeberover 6 years ago

alert: APIServerLatencyHigh
expr: apiserver_latency_seconds:quantile{quantile="0.99",subresource!="log",verb!~"^(?:WATCH|WATCHLIST|PROXY|CONNECT)$"}
  > 1
for: 10m
labels:
  severity: warning
annotations:
  description: the API server has a 99th percentile latency of {{ $value }} seconds
    for {{$labels.verb}} {{$labels.resource}}
  summary: API server high latency

joshmyersover 6 years ago

Is this the k8s API, or your application API?

Zachary Loeberover 6 years ago

k8s api

joshmyersover 6 years ago

I assume you have done the usual thing of Googling k8s API latency/issues?

Arjun Iyerover 6 years ago

Folks, anybody have experience with distributed tracing tools like Zipkin or Jaeger? We are building an automated solution that would greatly reduce the time spent in searching for the right traces which point to the root cause. We would love to talk to you for 15 mins. Feel free to DM me if you are interested.

Erik Osterman (Cloud Posse)over 6 years ago

@UUB28NLDS help keep tabs! 😉

grvover 6 years ago(edited)

Hi guys, anyone using Datadog? Currently we are setting it up, and I am bit confused if it needs to be paired with oob Kubernetes monitoring tools like Kube state metrics

grvover 6 years ago(edited)

We are running KOPS based kube clusters, with Datadog agents deployed as daemonsets. My question is, is there a benefit of pairing Kube state metrics with Datadog agents? The current dashboard is showing duplicate values of instance types (probably cz of metrics gathering from two sources)

January 2020 Browse by date March 2020