sreArchived
25 messages
Erik Osterman (Cloud Posse)over 5 years ago
For those of you who missed our #office-hours , here's an explanation of how we're managing opsgenie with terraform:
https://www.youtube.com/watch?v=fXNajuC4L1o
https://www.youtube.com/watch?v=fXNajuC4L1o
Erik Osterman (Cloud Posse)over 5 years ago
Jayesh Patelover 5 years ago
Does anyone know how to set this Prometheus alert for specific time duration using
inhibition_rule ? - alert: TestEventBacklogCritical
expr: sum(test_depth{topic="events",paused="false"}) >= 150000
for: 15m
labels:
severity: page
annotations:
description: Event queue has reached 150K for at least 15m
summary: Event queue has reached 150K
Ian Bartholomewover 5 years ago
Has anyone worked with either DataDog synthetics or AWS CloudFront synthetics? We are evaluating them, and are curious if anyone has prior experience one way of the other with them. Thanks!
Chris Fowlesover 5 years ago
are you already using datadog?
Zachover 5 years ago
cloudwatch synthetics seemed ridiculously expensive when it was released
Chris Fowlesover 5 years ago
yeh - that's kind of where i was going to lead. i think our cost estimates had datadog at 1/3 to 1/2 the price of cloudwatch
Zachover 5 years ago(edited)
and datadog is like 2-3x as expensive as other synthetic providers 😉 I guess it makes sense if you already use them
Chris Fowlesover 5 years ago
i'd probably lean into datadog over cloudwatch
Chris Fowlesover 5 years ago
so the catch is, cloudwatch you need to pay also for: alarms, logs, lambda, and s3
Chris Fowlesover 5 years ago
If you create 5 canaries that run once every 5 minutes, add alarms on 5 of the metrics generated by the canaries, and store the data for 1 month, your monthly bill will be calculated as follows:
5 canaries * 12 runs per hour * 24 hours per day * 31 days per month = 44,640 canary runs
Monthly CloudWatch charges
Canary run charges = 44,640 canary runs * $0.0012 per canary run = $53.57 per month
5 alarms per month = 5 * $0.10 = $0.50 per month
Total monthly CloudWatch charges = $53.57 + $0.50 = $54.07
Monthly additional charges
Each canary run also runs an AWS Lambda function and writes logs and results to CloudWatch Logs and the designated Amazon S3 bucket. For details on AWS service pricing such as AWS Lambda, Amazon S3, and CloudWatch Logs, see the pricing section of the relevant AWS service detail pages.
Lambda charges = requests charges + duration charges
= requests from 44,640 runs * $0.2 per 1,000,000 + duration of 20 seconds * 44,640 canary runs * 1 GB memory size * $0.000016667 per GB per sec
= $0.01 + $14.88 = $14.89 per month
CloudWatch Logs charges = collection charges + storage charges
= collection of 0.00015 GB per run * 44,640 runs * $0.5 per GB + storage of 0.00015 GB per run* 44,640 canary runs * $0.03 per GB per month
= $3.35 + $0.20 = $3.55 per month
S3 charges = put request charges + storage charges
= put requests of 44,640 runs * $0.005 per 1,000 requests + storage of 0.001 GB per run * 44,640 canary runs * 1 month * $0.023 per GB per month
= $0.22 + $1.03 = $1.25 per month
Additional monthly charges = $14.89 + $3.55 + $1.25 = $19.69
Total monthly charges = $54.07 + $19.69 = $73.76
Pricing values displayed here are based on US East Regions. Please refer to pricing tabs for most current pricing information for your respective region(s).Zachover 5 years ago
yah a single synthetic canary run on Cloudwatch per minute was going to be like an entire month on other services
Zachover 5 years ago
I honestly don’t understand who is paying/using that
Chris Fowlesover 5 years ago
people who have approval to spend money on AWS but not enter new contracts with third parties
Zachover 5 years ago
Ah, I have a little bit of that going on. VPs don’t raise an eyebrow at the aws bill but if I ask for a new purchase I have to prove that I can’t build it for cheaper
Chris Fowlesover 5 years ago
this is what AWS marketplace is for 😉
Chris Fowlesover 5 years ago(edited)
Chris Fowlesover 5 years ago
😄
Chris Fowlesover 5 years ago(edited)
or maybe you want to roll out sumologic? https://aws.amazon.com/marketplace/pp/B06XXVNPN2?qid=1599609541958&sr=0-1&ref_=srh_res_product_title
Zachover 5 years ago
AWS Marketplace mostly feels like “AMIs that I could build with a simple userdata script that fetches a binary from github”
Chris Fowlesover 5 years ago
yeh - but also i can put things in my aws bill that finance won't raise an eyebrow at 😉
Zachover 5 years ago
or “a cloudformation stack that I almost certainly will regret deploying later”
btaiover 5 years ago(edited)
is this tutorial still relevant for deploying prometheus operator w/ thanos? https://medium.com/@kakashiliu/deploy-prometheus-operator-with-thanos-60210eff172b
sheldonhover 5 years ago
Deep into datadog trial. Had another division propose site24x7. Any general impression on it for a comprehensive platform tool? Anything will help.
Looking to centralized logs, apm, monitors etc for AWS.
Datadog is so expensive due to per host costs so it's causing some to want more competitor evaluation.
Looking to centralized logs, apm, monitors etc for AWS.
Datadog is so expensive due to per host costs so it's causing some to want more competitor evaluation.
muhahaover 5 years ago
Guys? I have a question ... What are You using for logging in k8s ( forwarders ) ?
I am using Loki & Opendistro ( Elasticsearch ), problem is that I want to use Fluentbit + FluentD combo ( tls forward, exposed separate loadbalancer ), what is problem that there is not complete & matured solution for it, which is weird ..
• fluentbit -> there isnt support for hotreloading, nor API endpoints, signaling option in application
• fluentd -> there is no good helm chart with elasticsearch & loki output and sidecar for reloading after config change ( not only config, but mainly secret (tls) change )
• logging-operator by banzaicloud -> systemd and host logging is behind paywall via logging-opeator-extensions, which is nogo for me
• kubesphere/fluentbit-operator -> seems unfinished ( no helm chart ), but promising
• vmware/kube-fluentd-operator -> helm chart available, its promising
Any other alternatives? I can probably use beats & logstash, but whole community is using fluentbit/fluentd combo..., but this ecosystem is not matured yet...
Ideas? Thanks
I am using Loki & Opendistro ( Elasticsearch ), problem is that I want to use Fluentbit + FluentD combo ( tls forward, exposed separate loadbalancer ), what is problem that there is not complete & matured solution for it, which is weird ..
• fluentbit -> there isnt support for hotreloading, nor API endpoints, signaling option in application
• fluentd -> there is no good helm chart with elasticsearch & loki output and sidecar for reloading after config change ( not only config, but mainly secret (tls) change )
• logging-operator by banzaicloud -> systemd and host logging is behind paywall via logging-opeator-extensions, which is nogo for me
• kubesphere/fluentbit-operator -> seems unfinished ( no helm chart ), but promising
• vmware/kube-fluentd-operator -> helm chart available, its promising
Any other alternatives? I can probably use beats & logstash, but whole community is using fluentbit/fluentd combo..., but this ecosystem is not matured yet...
Ideas? Thanks