#sre - June 2020

nice - it'd be good if they could make it available as a git repo ala pagerduty

anyone doing long term storage for prometheus?

we have some async tasks. maybe around 20+ or so. Some of them run at odd hours in the middle of the night and some of them can take up to 20 min to run. I want to get super alerted if something doesn’t run or fail to run. Looking for advice on dashboarding versus alerting. Currently, the team has been trained to keep a close eye on Sentry alerts that comes in thr Slack. We had email alerts from AWS in the past but the team got tuned out.

anyone have an example of dropping prometheus labels (i.e. pod name, ip) from some of my custom prometheus metrics with a specific prefix? I can’t tell from the relabeling config whether I can drop it from a subset of metrics

heres the part of my helmfile that i’m attempting to do this:

        serviceMonitor:
          metricRelabelings:
          - targetLabel: pod
            replacement: ''
          - targetLabel: instance
            replacement: ''
          - sourceLabels: [__name__]
            regex: '(container_tasks_state|container_memory_failures_total)'
            action: drop

I currently have prometheus running on an EC2 instance for my scrapes, as the rest of our applications are all on EC2 still. It has a detachable EBS volume so that I can destroy the instance and spin up a new one for upgrades, but this leaves me with a few-minutes gap in the data when the old instance is down and waiting for the new one to finish attaching the EBS and reading back the WAL.
Is there some way (and I’m open to moving this to ECS if that would solve it) of doing this with (near) zero downtime in my metrics?