#sre - November 2018

What are you guys using with ECS and prometheus for SD?

We had our own SD in python, but im always afraid of hitting the api limits as we scale, as we did before

(we had a DNS SD based on lambda and ECS events)

At previous client who were so big they ended up having to pay for AWS API requests a tool was written to do a single lookup a la ec2_sd, write it to a file and the file gets mounted inside k8s prom, where k8s prom was multi team and if each team did their own lookups, would bust the limit

Yeah, we did a tool for that in Lambda, tbh is not that complicated

but I was looking for a "simpler" solution

I have a similar situation for uploading new prometheus configs, without doing a docker deployment, since albeit incorrectly that was the "easy" start for us but it sort of sucks

@Gonzalo Peci ☝️

what's wrong with Consul for SD ?

You need consul? 🙂

Maybe not what you want if that is all you are going to use Consul for

@joshmyers so what are your reasons for not using Consul if used strictly for SD ?

ease of use, setup, deficiencies in AWS SD options, yeah, Consul is great

I don’t have any. I’m just saying folks may not want to run a 3 node etcd cluster when they have been using AWS API as cheap service discovery

Good news, Consul is not etcd 😄 😄 😄

hah, oops, same thing. It is a thing you need to manage?

of all the services I've operated/managed since 2014, consul is the least needy service I've met

Nice

self-bootstrapping EC2 ASG cluster FTW

Have used with Nomad before and not had any issues with it, but it isn’t a managed type service, was my only point 🙂

managed type services are good for getting started -- one should have a plan for when your org's needs or skills outgrow a managed service offering from anyone

...such as multi-cloud

Aye, multi cloud is hard though 😉

yea, the all elusive multi-cloud strategy

@mrwacky yeah we know about consul and registrator, but as explained by @joshmyers that is ofc the option if you have Consul, we dont

and while it is a good easy discovery once you have that, the question would be you do if you dont hve consul

at the moment our services dont use mesh, as we dont need/want that yet

so consul will be there ONLY to support prometheus discovery, and that seemed overkill to me, but maybe its the only option

I'm sure there's other options..

Tamsky, I know the options, as I mentioned we are already using this, and we had our own lambda for SD

I was trying to ping/pong how others were doing it

(we had a DNS SD based on lambda and ECS events)

but I was looking for a "simpler" solution

I guess I was trying to help out re: "simpler" solutions.

I have a similar situation for uploading new prometheus configs, without doing a docker deployment, since albeit incorrectly that was the "easy" start for us but it sort of sucks

how do you handle persistent storage for prometheus in your docker setup -- that answer might guide us toward an easy process that can update your prometheus configs.