#prometheus - March 2019 | Slack Archive

282 messages

Archive: https://archive.sweetops.com/prometheus/

Feb 2019

March 2019

Apr 2019

Glenn J. Masonabout 7 years ago

Hey folks, I'm deploying Prometheus (with helm and the prometheus-operator chart from stable) into several clusters, but wondering how best to set up the alerting rules? I've added a Slack receiver to Alertmanager, and I'm getting alerts there, but ... any clues on the process I would use to ensure good alerting on my clusters?

Erik Osterman (Cloud Posse)about 7 years ago

Hrmm @Glenn J. Mason can you add some more context around the process piece?

Erik Osterman (Cloud Posse)about 7 years ago

Process for updating alerts?

Erik Osterman (Cloud Posse)about 7 years ago

Where to define them?

Glenn J. Masonabout 7 years ago

More like ... how do I go about discovering what metrics mean "bad, or going bad"? Like ... do I just monitor in e.g. grafana and/or make up "sensible" defaults that I think might work? The human process for tuning I mean.

Erik Osterman (Cloud Posse)about 7 years ago

Have you started by looking over the many community provided dashboards?

Gonzalo Peciabout 7 years ago(edited)

Anyone running Pushgateway in HA "HA"?

Gonzalo Peciabout 7 years ago

We are evaluating multiple options, but could not find a lot of info about it, as I can see we:
- Replicate traffic to multiple PG from another proxy
- Send reports to multiple PGs

tamskyabout 7 years ago

@Gonzalo Peci what are your goals with running pushgateway with an HA setup?

Gonzalo Peciabout 7 years ago

to avoid downtime during deployments/etc so it always gets the metrics

Gonzalo Peciabout 7 years ago

EG: i roll a new update, config is bad, i broke the PG, and now Im losing all PG metrics coming in (backups, etc)

Gonzalo Peciabout 7 years ago

if this was HA, a rolling/blue-green would be really simple

tamskyabout 7 years ago

have you considered using either DNS + a load balancer -- or service discovery in clients -- which should help?

tamskyabout 7 years ago

I've never run PG in production, but if I did, I would avoid running it as a singleton.
Is there anything non-trivial involved with running more than one PG (replicas with load-balancing, healthchecks)?

Gonzalo Peciabout 7 years ago

We do have a LB and DNS

Gonzalo Peciabout 7 years ago

but the problem is that you need the data in both places

Gonzalo Peciabout 7 years ago(edited)

is like having 2 postgres database, wihtout replication, and trying to read form the 2nd one if hte first one goes down, you dont have the same data

tamskyabout 7 years ago

to avoid downtime during deployments/etc so it always gets the metrics

EG: i roll a new update, config is bad, i broke the PG, and now Im losing all PG metrics coming in (backups, etc)

if this was HA, a rolling/blue-green would be really simple

I think I'm having trouble understanding what "always gets the metrics" means for your situation.

is like having 2 postgres database, wihtout replication, and trying to read form the 2nd one if hte first one goes down, you dont have the same data

Are you sure you're using pushgateway to solve a problem for which it is designed & recommended ?
- https://prometheus.io/docs/practices/pushing/

If not, please describe what you're trying to solve for.

tamskyabout 7 years ago

if hte first one goes down, you dont have the same data

If the first one goes down, you shouldn't care, assuming both are equally scraped by prometheus.
Prometheus reads the metrics from the first (down) and second (up) and merges metrics+{labelsets} into the same metrics timeseries history.

Gonzalo Peciabout 7 years ago

I think I'm having trouble understanding what "always gets the metrics" means for your situation.

You have only 1 PG, you do a failed deployment or it goes down for any reason, now any job pushing to PG cant push to it, because its down until you fix it/rollback/etc

Are you sure you're using pushgateway to solve a problem for which it is designed & recommended ?

Yes im 100% sure, eg scheduled jobs like backups or other croned or event triggered things.

If the first one goes down, you shouldn't care, assuming both are equally scraped by prometheus.

It does matter, because if you have 2 PG, unless you push the same data to both, by either a proxy/LB that replicates the data to both or by pushing to both from the clients, you have different data

Gonzalo Peciabout 7 years ago(edited)

eg. a backup job runs, after it ran it pushes a metrics to PG through a LB.
- PG1 gets the connection and stores the info
- PG2 does not, as the LB only sent to PG1

PG1 goes down, for whatever reason, Prom starts reading from PG2, which doesnt contain the metric for the backup. now your metrics show odd stuff, and might trigger an alert about a missing backup in our case

tamskyabout 7 years ago

Let me respond to each of those separately:

You have only 1 PG,

Is something forcing you to create a single point of failure?
By deploying 1 PG, you create a SPOF.
Have you considered deploying N+1 ?

you do a failed deployment or it goes down for any reason, now any job pushing to PG cant push to it, because its down until you fix it/rollback/etc

If, as you say, you only have 1 PG and you accidentally deploy a "failed deployment" (read: "a blackhole for metrics"), yes, you're going to lose data.
Load balancers alone won't help, but healthchecks do.
If your infrastructure is always testing that your PG instances are able to accept POST requests, then you have some assurances that active PG are able to accept metrics.

Gonzalo Peciabout 7 years ago(edited)

Is something forcing you to create a single point of failure?

Are you even reading my messages?! that is exactly what im doing, and asking how others are doing that....

do you even run PG? because tbh it seems you have no clue what you are talking about

Gonzalo Peciabout 7 years ago

do you understand PG only keeps the data at the receiving end?

tamskyabout 7 years ago

yes

Gonzalo Peciabout 7 years ago

and if you have 2, unless you send the data to the second nodea s well, you are not going to have redundant data

Gonzalo Peciabout 7 years ago

To be honest, it seems like you are trolling at this point

tamskyabout 7 years ago

are you looking to use pushgateway as a continuous scoreboard? because that's what it sounds like -- but that's not what it's designed for.

Gonzalo Peciabout 7 years ago

Again, do you even read?

tamskyabout 7 years ago

absolutely I read and have been trying to understand how you're using it and why exactly you think it's a problem to have different data on different PGs

Gonzalo Peciabout 7 years ago(edited)

Why dont you explain how YOU run your N+1 reployment instead?

tamskyabout 7 years ago

sure

tamskyabout 7 years ago

I was in the middle of composing this when you replied, so I'll send it now:

Yes im 100% sure, eg scheduled jobs like backups or other croned or event triggered things.

If your backups and crons run in the same place (host/instance) then I'd suggest using the Textfile collector [1]
I've definitely used it to export the status of a backup script... the script would even update metrics during execution,
eg: it updated a start_time metric before triggering the backup
and additional detailed metrics at exit: {exitcode, completion_time, last_backup_size_bytes}

If using textfile metrics is not possible, that's fine -- just hoping to make you aware it is option, if you weren't already.

[1] https://github.com/prometheus/node_exporter#textfile-collector

Gonzalo Peciabout 7 years ago(edited)

Not an option, and the textfile option is just a hack tbh, you are basically building metric state in your node

tamskyabout 7 years ago

now you sound trollish 😃

Gonzalo Peciabout 7 years ago

but not an option, this is lambda

tamskyabout 7 years ago

ok great - thanks for sharing the environment in which you're trying to operate

Gonzalo Peciabout 7 years ago

im waiting for you N+1 deployment,instead of how to save metrics to disk

Gonzalo Peciabout 7 years ago

pushing lambda/cron/job style metrics is EXACTLY what PG is there for

tamskyabout 7 years ago

sure.

tamskyabout 7 years ago

An ALB with N+1 PG's in the targetgroup can receive metrics from your ephemeral jobs.

tamskyabout 7 years ago

Your prometheus server collects metrics from each PG in the TG individually -- it does not scrape them via the ALB

tamskyabout 7 years ago

rewinding a bit from the scenario above -- are you doing anything currently to manage the metrics lifecycle on your current PG's ?

tamskyabout 7 years ago

do you delete metrics at all?

Gonzalo Peciabout 7 years ago(edited)

On PG, no I do not delete them, but that has not been an issue nor the topic of my question

Gonzalo Peciabout 7 years ago

now, in your scenario

Gonzalo Peciabout 7 years ago

you have a lambda, that runs, pushes a eg, timestamp of lastsuccess run (like many metrics do) to PG, through LB. It only lands in 1 of the PG, lets say PG1
Now PG1 is down, your new metrics would go to PG2.
Now you have an alert or want to check something like:
((time() - pushgateway_thismetric_last_success) / 3600) > 24 to check last daily backup did run

Gonzalo Peciabout 7 years ago

wouldnt the 2PG metrics be 2 different metrics (due to the node they come from) and make it so you get a trigger

Gonzalo Peciabout 7 years ago

because PG2 might not have data for pushgateway_thismetric_last_success yet

tamskyabout 7 years ago

https://github.com/prometheus/pushgateway#about-the-job-and-instance-labels

tamskyabout 7 years ago

so the only situation where what you've described creates the situation you sound concerned about would be:
your lambda pushes via the LB to pg1, after which pg1 dies without being scraped by your prometheus server.
in all other situations where lambda pushes via the LB to pg1 and prometheus scrapes it, you're safe.

Gonzalo Peciabout 7 years ago

because of merges metrics+{labelsets} into the same metrics timeseries history?

tamskyabout 7 years ago

because your query for pushgateway_thismetric_last_success should return the latest metric, whether it arrived via pg1 or pg2

Gonzalo Peciabout 7 years ago

Yeah, that makes sense, cool

Gonzalo Peciabout 7 years ago(edited)

I was under the impression it would not do that due to instance being different basically

tamskyabout 7 years ago

you have to make sure the instance label is being set to ("") on scrape

Gonzalo Peciabout 7 years ago(edited)

Yep, they act as basically the same metric

tamskyabout 7 years ago

that's the behavior you want 😃

Gonzalo Peciabout 7 years ago

Indeed

tamskyabout 7 years ago

if you do have "can't lose" metrics -- you'll want a more durable path to getting the metrics to prometheus

Gonzalo Peciabout 7 years ago

Yeah, tbh in the rare case of the metric getting pushed to PG1 and prom not scraping in time

Gonzalo Peciabout 7 years ago

i can tolerate the bad alert

Gonzalo Peciabout 7 years ago

Or I guess there the persisting to disk or some other exporter would make more sense

tamskyabout 7 years ago

your questions here got me thinking of a setup : api gateway -> lambda sns writer -> sns readers -> dynamodb ) + ( /metrics scrape endpoint -> api gw -> lambda dynamo reader )

Gonzalo Peciabout 7 years ago

that would be better than PG tbh

tamskyabout 7 years ago

right ? glad you agree 😃

Gonzalo Peciabout 7 years ago(edited)

you could even ALB -> lambda -> dynamo

tamskyabout 7 years ago

I searched around and didn't find anything even discussing that setup

Gonzalo Peciabout 7 years ago

without SNS and api gate

tamskyabout 7 years ago

I was thinking of a fanout for writes to go to multiple readers, in case you wanted cross-region HA

Gonzalo Peciabout 7 years ago

I guess you could make it have 2 endpoints

Gonzalo Peciabout 7 years ago

one for direct push, 1 for fanout or similar

Gonzalo Peciabout 7 years ago

or make it extensible for fanout

tamskyabout 7 years ago

😃
maybe there's a market for an HA pushgateway

Gonzalo Peciabout 7 years ago

as for many people ALB->lambda->dynamo might be enought

tamskyabout 7 years ago

I was also thinking that having TTLs on the dynamo entries would kind-of automate the metrics deletion aspects that you've waved your hands at

Gonzalo Peciabout 7 years ago

if the entire ALB and lambda of region goes down, i have probably bigger issues 😛

Gonzalo Peciabout 7 years ago

Yeah, tbh we dont have that issue right now, as most of our metrics are not "stale"

tamskyabout 7 years ago

auto-metrics-deletion gives a lot of rope to folks to create high cardinality issues for themselves.

Gonzalo Peciabout 7 years ago

but if you run most of your stuff in serverless, I guess you do have that problem

tamskyabout 7 years ago

running your stuff in serverless isn't a fixed concern -- it's always how you've designed your labelsets. you have to be cognizant of their cardinality.

Gonzalo Peciabout 7 years ago(edited)

Yeah, but pushed on button, as we only run satellite jobs and some simple (but important) metrics from them (backups, cleanups, checks, auto handle of failed events, etc) we dont have too many labels or cardinality

tamskyabout 7 years ago

you might benefit from having instance on a pushed metric in the pushgateway -- so you can quickly find+debug the job that pushed it.
but when you scrape it, that instance label is no longer useful, so it should be dropped/nulled.

Gonzalo Peciabout 7 years ago

Yeah, that basically was my entire confusion with this PG stuff at the end

tamskyabout 7 years ago

yay!

tamskyabout 7 years ago

glad to help

Gonzalo Peciabout 7 years ago

that is why I wanted to check how others were doing, because just discarding random stuff on the prometheus config seemed odd, but we needed a way to handle more than 1PG so we HA

tamskyabout 7 years ago

If you don't yet, I'd suggest you run a blackbox prober scheduled lambda that pushes metrics on an interval, both individually to each PG (avoiding the ALB), as well as through the ALB.
This gives you an alertable metric that your PG is not functioning.

Gonzalo Peciabout 7 years ago

but you are also correct, once Prom got the metric from one PG, and as long as your query "accepts" or "discards" lables that make the metrics from any of the PGs different, it should work

Gonzalo Peciabout 7 years ago

ill do a local test with compose,thanks for the help

Gonzalo Peciabout 7 years ago

glad we could understand each other at the end

tamskyabout 7 years ago

same here.

Gonzalo Peciabout 7 years ago

I have one more @tamsky if you have a minute

Gonzalo Peciabout 7 years ago(edited)

if you have
PG1 last_success = 123
PG2 last_success = 124
(because the LB sends one to each)
dont they get overwriten on the prom side after each scrape and make for odd metrics?

tamskyabout 7 years ago

timestamps should distinguish the two

tamskyabout 7 years ago

when you push, you push with a timestamp?

tamskyabout 7 years ago

https://github.com/prometheus/pushgateway#about-timestamps

tamskyabout 7 years ago

reading that, maybe I'm confusing it with the timestamps in the textfile exporter

tamskyabout 7 years ago

rereads that link

tamskyabout 7 years ago

do you have a push_time_seconds for your last_success metrics group?

Gonzalo Peciabout 7 years ago

push_time_seconds gets generated automatically

Gonzalo Peciabout 7 years ago

afaik

Gonzalo Peciabout 7 years ago

there is a timestime field tho, which in my test I did not push

Gonzalo Peciabout 7 years ago

push_time_second

Gonzalo Peciabout 7 years ago


/tmp/tmp.Aj9RooPWfR on  master [!] on 🐳 v18.09.3 took 2s 
❯ echo "some_metric 3.14" | curl -u "admin:admin" --data-binary @- <http://localhost:9091/metrics/job/some_job>

/tmp/tmp.Aj9RooPWfR on  master [!] on 🐳 v18.09.3 
❯ echo "some_metric 3.15" | curl -u "admin:admin" --data-binary @- <http://localhost:9092/metrics/job/some_job>

Gonzalo Peciabout 7 years ago

the sample metric looks like

tamskyabout 7 years ago

I see what's going on. thinking

Gonzalo Peciabout 7 years ago

that is the problem i have with my backups metrics as well, that is why i was trying to basically "replicate" traffic to both PGs

Gonzalo Peciabout 7 years ago

I might be missing something in the config tho idk

Gonzalo Peciabout 7 years ago

  - job_name: "pushgateway"
    scrape_interval: 10s
    honor_labels: true
    static_configs:
      - targets: ["pushgateway1:9091", "pushgateway2:9091"]

Gonzalo Peciabout 7 years ago

just in case

tamskyabout 7 years ago

what does a graph of push_time_seconds look like?

Gonzalo Peciabout 7 years ago

the same

tamskyabout 7 years ago

should be two flat lines

Gonzalo Peciabout 7 years ago

no, its just one

Gonzalo Peciabout 7 years ago

because they dont have any labels that distinguishes them

tamskyabout 7 years ago(edited)

and you pushed at nearly the same time to both 9091 & 9092 ?

Gonzalo Peciabout 7 years ago

some seconds after

Gonzalo Peciabout 7 years ago

no, not at the same time

Gonzalo Peciabout 7 years ago(edited)

If you have an LB one can come now, the other in 10 minutes and hit the other node

tamskyabout 7 years ago

that is the problem i have with my backups metrics as well, that is why i was trying to basically "replicate" traffic to both PGs

I understand that you want to try to replicate PUT/POST to the PG...
but I'm now thinking I'm wrong about HA being a valid design pattern for this.

I was thinking that the PG timestamps operated similarly to the file exporter / text format
- https://prometheus.io/docs/instrumenting/exposition_formats/#text-format-details

tamskyabout 7 years ago

rewinding to the decision of "I need HA for PG" -- what's driving that decision?

Gonzalo Peciabout 7 years ago

Dont you run PG in HA? and even you said

By deploying 1 PG, you create a SPOF. Have you considered deploying N+1 ?

Gonzalo Peciabout 7 years ago(edited)

dont you have the same issue?

tamskyabout 7 years ago

I've run PG in multiple regions, but haven't needed HA, single node ASG has been fine.

Gonzalo Peciabout 7 years ago

So basically no

tamskyabout 7 years ago

it's possible HA can still work, you'd add a non- job/instance label on scrape of each PG with a unique value and then can query for the most recent group of metrics across all scrapes

Gonzalo Peciabout 7 years ago

Yeah

tamskyabout 7 years ago

and your metrics query looks like a join

Gonzalo Peciabout 7 years ago

Now to Why, as explained in the question before

Gonzalo Peciabout 7 years ago

when you asked that

Gonzalo Peciabout 7 years ago

what if your deploy a bad version?

Gonzalo Peciabout 7 years ago

etc

tamskyabout 7 years ago

yes.

Gonzalo Peciabout 7 years ago

maybe this https://www.robustperception.io/aggregating-across-batch-job-runs-with-push_time_seconds

Gonzalo Peciabout 7 years ago

will help

tamskyabout 7 years ago

if you deploy a metrics blackhole, you're going to lose pushes 😃

Gonzalo Peciabout 7 years ago

Im surprised im the only one having this "issue" with PG not supporting HA

tamskyabout 7 years ago(edited)

The thing you seem concerned with ("bad config/deploy") is not unique to PG

tamskyabout 7 years ago

What if you deployed a bad version of prometheus? Or your lambda functions?

tamskyabout 7 years ago(edited)

But you might be able to get there with max_over_time(push_time_seconds{job="myjob"}[5m])

Gonzalo Peciabout 7 years ago

that is why you run prom in HA

Gonzalo Peciabout 7 years ago

with 2+ nodes and even federation

Gonzalo Peciabout 7 years ago

and a DNS/Server/Client based balance for EG grafana

tamskyabout 7 years ago

yes, but nobody gets concerned if you lose one of 2 nodes, and that a new node doesn't have exactly the same set of data that the older node does

Gonzalo Peciabout 7 years ago

but quite similar

Gonzalo Peciabout 7 years ago

its clearly not the same, prometheus has a pseudoHA config

tamskyabout 7 years ago(edited)

I don't agree with the statement: prometheus has a pseudoHA config

Gonzalo Peciabout 7 years ago

where you scrape same targets, and the only difference is due to timing, which you have to live with or not use prometheus

tamskyabout 7 years ago

that I agree with.

Gonzalo Peciabout 7 years ago

that is their way of HA

tamskyabout 7 years ago

it's redundancy

tamskyabout 7 years ago

when running multiple promservers nobody expects them to store identical timeseries history to disk

Gonzalo Peciabout 7 years ago

And I never said that

tamskyabout 7 years ago

but that's HA

Gonzalo Peciabout 7 years ago

Where does it say that is HA?

Gonzalo Peciabout 7 years ago

that is maybe your personal take on HA

tamskyabout 7 years ago

that seemed to be the desired outcome for the PG -- that an HA version of PG would replicate writes to all PGs

tamskyabout 7 years ago

that sort of replicated write does seem like a possible way to configure PG

Gonzalo Peciabout 7 years ago

max_over_time(push_time_seconds[1m]) would work for evergrowing metrics, but not for a gauge

Gonzalo Peciabout 7 years ago

if you push something like items_processed you can do that

Gonzalo Peciabout 7 years ago

and you get a flapping metric again

tamskyabout 7 years ago

lets get an example going for max_over_time

tamskyabout 7 years ago

push_time_seconds{job="myjob",pg="pg1"} 111
push_time_seconds{job="myjob",pg="pg2"} 122

Gonzalo Peciabout 7 years ago

where is pg= coming from?

Gonzalo Peciabout 7 years ago(edited)

are you adding instance labels?

Gonzalo Peciabout 7 years ago

because the client pushing, is not aware to which PG is pusshing

tamskyabout 7 years ago

I mentioned it earlier:

it's possible HA can still work, you'd add a non- job/instance label on scrape of each PG with a unique value and then can query for the most recent group of metrics across all scrapes

Gonzalo Peciabout 7 years ago

Yeah, Prom side

tamskyabout 7 years ago

yup

Gonzalo Peciabout 7 years ago

yeah, doing a join does work, I meant direct max_overtime does not

Gonzalo Peciabout 7 years ago

quite... hacky imho

tamskyabout 7 years ago

I'd think you'd use max_over_time to select the series and pg label that has the most recent timestamp (you're running ntp right 🙂 )

Gonzalo Peciabout 7 years ago(edited)

yeah, that could work as a join, but its quite a hacky way as said before

tamskyabout 7 years ago

I agree hacky

tamskyabout 7 years ago

instead of engineering the PG to be HA have you considered making the PG clients more robust? what fallback mechanisms do you have if PG is down?

Gonzalo Peciabout 7 years ago(edited)

Yes, I did, as explained when I started this, I could have 2 targets per job

tamskyabout 7 years ago

that's just 2x fire-and-forget

tamskyabout 7 years ago

I'm thinking a separate backup path if the primary is down. Backup might not be HTTP.

Gonzalo Peciabout 7 years ago

I think at that point i would rather add a proxy that replicates the data to multiple PGs

Gonzalo Peciabout 7 years ago

as its a solution that applies to all my lambdas/jobs instead of an implementation per job

tamskyabout 7 years ago

how do you ameliorate the situation where the proxy misses a write to one of the PGs and the datasets diverge?

tamskyabout 7 years ago

you'll wind up with a fluctuating graph, like you shared

tamskyabout 7 years ago

unless you run one prom server per PG ?

Gonzalo Peciabout 7 years ago

but then its the same, 1 alrets one doesnt

Gonzalo Peciabout 7 years ago

multiple prom will not fix it

Gonzalo Peciabout 7 years ago

but yeah you are right, if the node returns, we might be back to square one

tamskyabout 7 years ago

I think "be careful" will still apply

Gonzalo Peciabout 7 years ago

unless we have non persistent metrics

tamskyabout 7 years ago

I wasn't thinking the PG node goes up/down/up

Gonzalo Peciabout 7 years ago

which then the new pg does not have a metrics for the metrics that the other PG does

tamskyabout 7 years ago

I was simply thinking the proxy fails to communicate a PUT/POST for any number of network reasons -- and it doesn't retry, so that PUT/POST gets dropped on the floor.

Gonzalo Peciabout 7 years ago(edited)

I would ask, how often is that bound to happen on an internal network?

tamskyabout 7 years ago

a non-zero number of times

tamskyabout 7 years ago

😃

Gonzalo Peciabout 7 years ago

I didnt ask how many times, I asked how often

tamskyabout 7 years ago

divide by choice of duration

Gonzalo Peciabout 7 years ago(edited)

which is, if this affects only 1% of the cases, im not really that worried for this scneario

Gonzalo Peciabout 7 years ago

how do you ameliorate the situation where the proxy misses a write to one of the PGs and the datasets diverge?

In a number of ways, for the number of problems

Gonzalo Peciabout 7 years ago

-.-

tamskyabout 7 years ago

I'm thinking from all this that you have different classes of metrics that you're pushing -- some are frequent and business critical and are categorized "can't lose",
and others are batch processes, like backups where the metric is useful but losing it isn't intrinsically harmful.

is that correct?

Gonzalo Peciabout 7 years ago

Well that is the thing, a metric for backups going missing is quite harmful depeding on the period

Gonzalo Peciabout 7 years ago

as it could trigger a number of alerts

Gonzalo Peciabout 7 years ago

OFC, if the soltuion to HA PG is more complex than just a couple of silences here and there

Gonzalo Peciabout 7 years ago

ill just silence them when the event presents

Gonzalo Peciabout 7 years ago

but Im suprised this is not a bigger problem for other usecases, eg. heavy serverless users

tamskyabout 7 years ago

triggering alerts definitely has a cost, but maybe there's a different metric, (for example, via a blackbox prober) that reports the actual status of the latest backup

Gonzalo Peciabout 7 years ago

you mean <https://github.com/prometheus/blackbox_exporter>?

tamskyabout 7 years ago

yes

Gonzalo Peciabout 7 years ago

Yeah, i mean that is basically saying "dont use PG if you need PG HA" which is ok

Gonzalo Peciabout 7 years ago

maybe PG is not the tool for this

Gonzalo Peciabout 7 years ago

but im asking in a more generic way

tamskyabout 7 years ago

a blackbox example might be: a script that reads the state of all backups and reports the most recent timestamp

Gonzalo Peciabout 7 years ago(edited)

how do you run PG in HA or some way of redundancy

Gonzalo Peciabout 7 years ago

and there does not seem to be a way to do that

Gonzalo Peciabout 7 years ago

Just to be clear, I know i can solve this in a number of different ways

Gonzalo Peciabout 7 years ago(edited)

but in this case this is not my question, it is rather, how would you HA PG?

tamskyabout 7 years ago

then I agree, there is no out of box clustered PG that offers a singular HA datastore

Gonzalo Peciabout 7 years ago

or replicated, even if delayed

Gonzalo Peciabout 7 years ago

like gossip based for example

Gonzalo Peciabout 7 years ago

Yeah, out of the box nothing, but maybe there are some clever ways to do this

tamskyabout 7 years ago

I'd use the existing singleton PG pattern, use ASG, and put it behind an ALB.

Gonzalo Peciabout 7 years ago

EG: a replication proxy (altho your comment makes a fair point against it)

Gonzalo Peciabout 7 years ago(edited)

Yeah, we do that already basically with a container

tamskyabout 7 years ago

I'd implement a backup channel in the clients to write metrics to SNS and have a persistent reader dequeue and PUT SNS data->PG

tamskyabout 7 years ago

clients would use the backup channel if the primary PG failed or timed out

Gonzalo Peciabout 7 years ago

The thing is the problem is not the channel tbh, if that was the case, you could argue the same point about any HTTP communication

Gonzalo Peciabout 7 years ago

but it could be a good solution to have some sort of queue

tamskyabout 7 years ago

it depends on the criticality of the data -- if you have a requirement that you can never lose writes, you need more backup channels

Gonzalo Peciabout 7 years ago

so client -> SNS/SQS/whatever -> deque -> PG

Gonzalo Peciabout 7 years ago

so if PG goes down, it will just put the metrics once its back up

tamskyabout 7 years ago

I'd throw in a S3 writer if I thought SNS wasn't reliable enough

Gonzalo Peciabout 7 years ago

Yeah, i agree, but again, I guess you could say the same about any communication

tamskyabout 7 years ago

maybe an out-of-region backup PG ALB as an alternate

Gonzalo Peciabout 7 years ago

and I dont see this being a pattern for any REST service for example

tamskyabout 7 years ago

you have this problem even in kafka writers

tamskyabout 7 years ago

kafka guarantees writes, but only after ACK. what do you do as a writer with critical data that doesn't receive an ACK.

tamskyabout 7 years ago

1. keep trying
2. queue to a different system
3. all of the above

tamskyabout 7 years ago

4. fall over and ask for help

Gonzalo Peciabout 7 years ago

yeah but you dont com over a different system

Gonzalo Peciabout 7 years ago

normally, you retry or buffer and retry

Gonzalo Peciabout 7 years ago

or fallover

tamskyabout 7 years ago

you don't have as many options in lambda

Gonzalo Peciabout 7 years ago

Exactly, but I think its not a great architecture patter to say

Gonzalo Peciabout 7 years ago

"all lambdas HTTP to PG or send over SNS if that fails"

Gonzalo Peciabout 7 years ago

because the same could be said about any HTTP connection in the same way

Gonzalo Peciabout 7 years ago

waht about lambda calling another service?

Gonzalo Peciabout 7 years ago

should it SNS as well?

Gonzalo Peciabout 7 years ago

I would rather just make SNS or a queue the com channel

Gonzalo Peciabout 7 years ago

and if that fails, then fallover and cry

tamskyabout 7 years ago

I can agree with that decision... just use SNS for all metrics writes

Gonzalo Peciabout 7 years ago(edited)

(given you can async)

tamskyabout 7 years ago

HTTP over SNS

Gonzalo Peciabout 7 years ago(edited)

Yeah, queuing the PG metrics might be a sane idea, that mitigate 90% of the issues, for not that much extra work

tamskyabout 7 years ago

actually I'm thinking SQS is the thing we're talking about, not SNS

Gonzalo Peciabout 7 years ago

Yeah, I said SNS as you said SNS before, but yeah im talking about a queue

tamskyabout 7 years ago

yeah - we're talking same

Gonzalo Peciabout 7 years ago(edited)

so client -> SNS/SQS/whatever -> deque -> PG

as said before

tamskyabout 7 years ago

the question isn't how to handle REST failures in lambda, it's how to deliver unique unreplicated data that will be otherwise lost unless written somewhere

Gonzalo Peciabout 7 years ago

Yep, i think that is accurate

tamskyabout 7 years ago

my mistake using SNS 😃

tamskyabout 7 years ago

and hopefully you can engineer it so that some amount of loss is acceptable

tamskyabout 7 years ago

as opposed to "no amount of loss is acceptable" 😃

Gonzalo Peciabout 7 years ago

Yeah, that i guess depends on the use case, as said for backups/simple stuff, it might be the silence-work<architectural-complexity

Gonzalo Peciabout 7 years ago

but i was curious about how this is tackled across places, as i can think of many ways/things where this becomes more critical

tamskyabout 7 years ago

hear me out for another minute

tamskyabout 7 years ago

lets say your cron job that performs backups fails to report its status to PG

tamskyabout 7 years ago

and you have an alarm NoRecentBackupsHaveCompleted which fires after N+1 hours

tamskyabout 7 years ago

I think it's also wise to have another job that reads the most recent backup and exports what it read.

tamskyabout 7 years ago

lets say that other "backup-reader" job has a metric alarm NoRecentValidBackup -- which would be in state OK if the backup worked ok

tamskyabout 7 years ago

you can combine those two alerts to inhibit. you don't need silences.

tamskyabout 7 years ago

lmk if you need more detail.

tamskyabout 7 years ago

how this is tackled across places, as i can think of many ways/things where this becomes more critical

I haven't worked with enough shops that were totally serverless to know the pitfalls.
I do know some folks are very happy with couchbase for event metrics.
I've also seen mondemand used as an event bus (in both multicast and anycast modes).
At some point, durable logging becomes your primary concern, and metrics ingestion becomes secondary.

tamskyabout 7 years ago

nsq is pretty awesome

Abel Luckabout 7 years ago(edited)

anyone using prometheus to monitor an aws stack with the cloudwatch exporter? is it worth using?

Abel Luckabout 7 years ago

I'm also curious how costs compare to using cloudwatch alone vs the api request costs of the exporter

Gonzalo Peciabout 7 years ago

We use CW-Exporter, but only for things we cant get native metrics for. I dont know if that helps you much

Abel Luckabout 7 years ago

by native metrics, you mean the host metrics from node_exporter?

Abel Luckabout 7 years ago

what would those metrics be that you use CW exporter for?

tamskyabout 7 years ago

random guess where CW-Exporter might be useful: ALB metrics

February 2019 Browse by date April 2019