63 messages
General Discussions
S
SweetOpsalmost 6 years ago
Hey everyone, give a warm welcome to our newest members!
- @kenny
- @Michael S
- @Padarn
- @Eric Malenfant
Good to have you here =)
- @kenny
- @Michael S
- @Padarn
- @Eric Malenfant
Good to have you here =)
Ryan Smithalmost 6 years ago
I actually don't know where else to ask the question. Does anyone have a preference of a connection pooler for postgres over:
• pgbouncer vs. anything else?
• pgbouncer vs. anything else?
HSalmost 6 years ago
Hi Everyone! I would love experts advice on scaling issues I'm having with a Monolith Application
The stack is Ruby, Mysql, Redis, RabbitMQ, and the webserver being used is Passenger, Scaling every other components (mysql, redis, rabbitmq) is not a problem, all of them are scaling awesomely, However, the application server is performing woefully, and the problem is with Passenger, the ruby app is a Legacy codebase in which every component of the application is bundle together in one place, including the mobile app graphql endpoints, and 20+ daemons that needs to run for the app to work.
I have tried scaling horizontally a couple of times which was not effective because I realized that the daemons works in a way that they can't run concurrently on multiple servers, so that means I had to resolve to scaling vertically, now during peak periods, the server resources are not really used up, but the server keeps crashing, and the major reason is because Passenger(Ruby webserver) couldn't handle some certain number of connections at once, so after thoroughly going through the passenger doc, I realized that there is a way to increase the maximum number of connection per process, this has been the temporary tweak that is keeping the server up and running for now.
However, We will be starting a campaign Next week, and we are expecting to have x5 the traffic requests we get during peak times, and I need advice on other approaches I can take to Autoscaling my Infrastructure
PS: 1. We are looking to re-architect the entire system by decoupling some services
2. We still run on Ruby 2.2, upgrading it will break our codebase (which is why we need to do 1 above)
I know this community consists of professionals that have handle way larger systems at scale, and I will appreciate everyone's input on this
Thank you
The stack is Ruby, Mysql, Redis, RabbitMQ, and the webserver being used is Passenger, Scaling every other components (mysql, redis, rabbitmq) is not a problem, all of them are scaling awesomely, However, the application server is performing woefully, and the problem is with Passenger, the ruby app is a Legacy codebase in which every component of the application is bundle together in one place, including the mobile app graphql endpoints, and 20+ daemons that needs to run for the app to work.
I have tried scaling horizontally a couple of times which was not effective because I realized that the daemons works in a way that they can't run concurrently on multiple servers, so that means I had to resolve to scaling vertically, now during peak periods, the server resources are not really used up, but the server keeps crashing, and the major reason is because Passenger(Ruby webserver) couldn't handle some certain number of connections at once, so after thoroughly going through the passenger doc, I realized that there is a way to increase the maximum number of connection per process, this has been the temporary tweak that is keeping the server up and running for now.
However, We will be starting a campaign Next week, and we are expecting to have x5 the traffic requests we get during peak times, and I need advice on other approaches I can take to Autoscaling my Infrastructure
PS: 1. We are looking to re-architect the entire system by decoupling some services
2. We still run on Ruby 2.2, upgrading it will break our codebase (which is why we need to do 1 above)
I know this community consists of professionals that have handle way larger systems at scale, and I will appreciate everyone's input on this
Thank you
S
SweetOpsalmost 6 years ago
Hey everyone, give a warm welcome to our newest members!
- @Marc Boudreau
- @Gemini
- @Peteris
Good to have you here =)
- @Marc Boudreau
- @Gemini
- @Peteris
Good to have you here =)
mfridhalmost 6 years ago(edited)
I'm looking to find some library (even shellscript...) to help write wrapper binaries/scripts for download-on-demand-and-re-execute binaries en masse... anyone seem something existing?
I've written a couple of these things standalone.. shellscripts which curl from a "repo" and re-execs itself... trying to see if there's anything out there already before I reinvent the wheel a 5th time.
whalebrew - wrapper for executing in docker for reference: https://github.com/whalebrew/whalebrew
tfenv comes to mind as an inspiration as well. although that's for one single tool and many versions.
I've written a couple of these things standalone.. shellscripts which curl from a "repo" and re-execs itself... trying to see if there's anything out there already before I reinvent the wheel a 5th time.
whalebrew - wrapper for executing in docker for reference: https://github.com/whalebrew/whalebrew
tfenv comes to mind as an inspiration as well. although that's for one single tool and many versions.
S
SweetOpsalmost 6 years ago
Hey everyone, give a warm welcome to our newest members!
- @Lucas Badico
- @RD
- @Phil Whelan
- @x80486
Good to have you here =)
- @Lucas Badico
- @RD
- @Phil Whelan
- @x80486
Good to have you here =)
S
SweetOpsalmost 6 years ago
Hey everyone, give a warm welcome to our newest members!
- @Chintan Parmar
- @highway900
- @Steffan
Good to have you here =)
- @Chintan Parmar
- @highway900
- @Steffan
Good to have you here =)
S
SweetOpsalmost 6 years ago
Hey everyone, give a warm welcome to our newest members!
- @Hutr
- @mehdi
- @chris
- @kiran
Good to have you here =)
- @Hutr
- @mehdi
- @chris
- @kiran
Good to have you here =)
S
SweetOpsalmost 6 years ago
Hey everyone, give a warm welcome to our newest members!
- @Libert
- @Mashail Almuzaini
- @Haroon Rasheed
- @Conor Maher
- @Shekhar
- @Andy
Good to have you here =)
- @Libert
- @Mashail Almuzaini
- @Haroon Rasheed
- @Conor Maher
- @Shekhar
- @Andy
Good to have you here =)
Zachary Loeberalmost 6 years ago
anyone happen to know offhand if there is a syntax for a gomplate file (like json or yaml) datasource to check for the existence of a key before attempting to use it?
HSalmost 6 years ago
Please can anyone recommend a good data masking tool to obfuscate production database in a non production environment?
Conor Maheralmost 6 years ago
I dont know of an off the shelf tool to this because its not an easy problem. e.g will your application still work if you just randomly obfuscate every field? What I have done in the past on a ruby/rails project is use https://github.com/faker-ruby/faker
Conor Maheralmost 6 years ago
And then you can have ActiveRecord models that obfuscate fields with sensitive data
Conor Maheralmost 6 years ago
e.g name, dob, and other sensitive details
Conor Maheralmost 6 years ago
You can generate sensible fake names, addresses, social security numbers etc
Conor Maheralmost 6 years ago
for most applications the obfuscated data needs to be plausible. e.g you cant just have a paragraph of lorem ipsum for date of birth or name
Conor Maheralmost 6 years ago
So context and generating the correct kind of obfuscated data matters
Conor Maheralmost 6 years ago
TLDR I dont think you will find an off the shelf tool that handles this. Build some custom-ish tooling to do this and run it in some automated process nightly / hourly etc
S
SweetOpsalmost 6 years ago
Hey everyone, give a warm welcome to our newest members!
- @Anand Maurya
Good to have you here =)
- @Anand Maurya
Good to have you here =)
S
SweetOpsalmost 6 years ago
Hey everyone, give a warm welcome to our newest members!
- @Joey
Good to have you here =)
- @Joey
Good to have you here =)
Joeyalmost 6 years ago
thanks for all of the warm welcomes guys 🦗🦗
S
SweetOpsalmost 6 years ago
Hey everyone, give a warm welcome to our newest members!
- @Mike F.
- @Erick
- @Avinash dubey
- @Valenterry
Good to have you here =)
- @Mike F.
- @Erick
- @Avinash dubey
- @Valenterry
Good to have you here =)
RBalmost 6 years ago
anyone here worked with celery ? currently on celery 2.x and looking to upgrade to the latest one but looking at the following strategies
• celery 2.x to celery 3.x to celery 4.x - ideal but seems like there are issues
• celery 4.x in tandum with celery 2.x and migration piece by piece - safer
• replacement for celery and migration piece by piece - possibly safer and better if the celery alternative has additional benefits
All celery jobs are in python... most are running like a daily cron. Questions for you folks...
• have you done this ?
• any of the above strategies seem better ?
• any alternatives to celery you might recommend ?
• celery 2.x to celery 3.x to celery 4.x - ideal but seems like there are issues
• celery 4.x in tandum with celery 2.x and migration piece by piece - safer
• replacement for celery and migration piece by piece - possibly safer and better if the celery alternative has additional benefits
All celery jobs are in python... most are running like a daily cron. Questions for you folks...
• have you done this ?
• any of the above strategies seem better ?
• any alternatives to celery you might recommend ?
joshmyersalmost 6 years ago
Not used Celery in a long time. What do the docs say about upgrade path?
RBalmost 6 years ago
there isn't one 😂
RBalmost 6 years ago(edited)
im considering dramatiq because thats the only thing that has an upgrade path (kind of)
S
SweetOpsalmost 6 years ago
Hey everyone, give a warm welcome to our newest members!
- @Bartosz
- @Rich K
- @Rameez Iqbal
Good to have you here =)
- @Bartosz
- @Rich K
- @Rameez Iqbal
Good to have you here =)
Shawn Petersenalmost 6 years ago
Hi guys,
I'm working with a small tech startup (less than 10 employees) and we're trying to come up with a good IT policy for employee laptops. With the vast amount of experience in this Slack channel, I'm wondering what you guys think the pros/cons are for a BYOD policy, versus the company-paid-for laptop that the company owns, locks down and manages (this is what most IT departments I've been with in the past do, and still do).
What do other cutting-edge tech startups do to manage their IT equipment now? What is your experience with BYOD? Does it work well? How is it implemented? or is the "tried-and-true" company-owned laptops still the best policy? … or, are there any other options?
All opinions and/or examples greatly welcome...
I'm working with a small tech startup (less than 10 employees) and we're trying to come up with a good IT policy for employee laptops. With the vast amount of experience in this Slack channel, I'm wondering what you guys think the pros/cons are for a BYOD policy, versus the company-paid-for laptop that the company owns, locks down and manages (this is what most IT departments I've been with in the past do, and still do).
What do other cutting-edge tech startups do to manage their IT equipment now? What is your experience with BYOD? Does it work well? How is it implemented? or is the "tried-and-true" company-owned laptops still the best policy? … or, are there any other options?
All opinions and/or examples greatly welcome...
S
SweetOpsalmost 6 years ago
Hey everyone, give a warm welcome to our newest members!
- @Marc Tanne
- @Stratos
- @Aabhusan
Good to have you here =)
- @Marc Tanne
- @Stratos
- @Aabhusan
Good to have you here =)
Rameez Iqbalalmost 6 years ago
Hi guys I have an issue with helm templates, wondering if someone can help. I am trying to pass private in values.yaml i.e.
And then I am using it in my template like
But it keeps replacing newlines with spaces. Has anybody come across this issue?
ssh_key: |
-----BEGIN RSA PRIVATE KEY-----
....
-----END RSA PRIVATE KEY-----And then I am using it in my template like
templates:
- |
apiVersion: v1
type: Opaque
kind: Secret
metadata:
name: bitbucket
stringData:
username: git
privateKey: {{ .Values.ssh_key | indent 4 }}But it keeps replacing newlines with spaces. Has anybody come across this issue?
Rameez Iqbalalmost 6 years ago
There seems to be something wrong with
stringData encryption. I will just stick with data for the time being.S
SweetOpsalmost 6 years ago
Hey everyone, give a warm welcome to our newest members!
- @baris
Good to have you here =)
- @baris
Good to have you here =)
S
SweetOpsalmost 6 years ago
Hey everyone, give a warm welcome to our newest members!
- @Adrian Todorov
- @Kevin Doveton
- @Mo_Nazib
- @Andrea Maruccia
- @jason witkowski
- @miles
Good to have you here =)
- @Adrian Todorov
- @Kevin Doveton
- @Mo_Nazib
- @Andrea Maruccia
- @jason witkowski
- @miles
Good to have you here =)
lorenalmost 6 years ago
joined another very active slack recently, didn't realize just how much i appreciate the way this one uses threads appropriately 😄 🤗
S
SweetOpsalmost 6 years ago
Hey everyone, give a warm welcome to our newest members!
- @Michał Czeraszkiewicz
- @Julien C
- @Kevin Chan
- @Ronald Konjer
Good to have you here =)
- @Michał Czeraszkiewicz
- @Julien C
- @Kevin Chan
- @Ronald Konjer
Good to have you here =)
BitsnBitesalmost 6 years ago
Anyone really good at MongoDB here?
Kevin Chanalmost 6 years ago
Hi
S
SweetOpsalmost 6 years ago
Hey everyone, give a warm welcome to our newest members!
- @Andrew
- @Mark R
Good to have you here =)
- @Andrew
- @Mark R
Good to have you here =)
S
SweetOpsalmost 6 years ago
Hey everyone, give a warm welcome to our newest members!
- @Satish
- @Fernando Torresan
- @Flaudísio Tolentino
- @Tim Birkett
Good to have you here =)
- @Satish
- @Fernando Torresan
- @Flaudísio Tolentino
- @Tim Birkett
Good to have you here =)
S
SweetOpsalmost 6 years ago
Hey everyone, give a warm welcome to our newest members!
- @Kalyan Sunkara
- @Anirudh Srinivasan
- @davidholsgrove
- @Hari Krishna Ganji
- @Michael Seiwald
- @Cassidy Nelemans
Good to have you here =)
- @Kalyan Sunkara
- @Anirudh Srinivasan
- @davidholsgrove
- @Hari Krishna Ganji
- @Michael Seiwald
- @Cassidy Nelemans
Good to have you here =)
S
SweetOpsalmost 6 years ago
Hey everyone, give a warm welcome to our newest members!
- @Jan Nonnen
- @ksaritek
- @Siriusx
- @Victor Aparicio
Good to have you here =)
- @Jan Nonnen
- @ksaritek
- @Siriusx
- @Victor Aparicio
Good to have you here =)
madoalmost 6 years ago
Hello I am Ken! Our next online
@WebHackMeetup
is about building your own #Golang linter. And it is our first event on
@airmeet_com
: https://webhack.connpass.com/event/175795/
@WebHackMeetup
is about building your own #Golang linter. And it is our first event on
@airmeet_com
: https://webhack.connpass.com/event/175795/
Harshal Vaidyaover 5 years ago
Hello - Where can I get help for cloudposse terraform modules? I am facing some errors and need some help.
Andrew Nazarovover 5 years ago
What do you think about “Managing stateful systems via container orchestration” blipping as Assess on the latest Technology Radar?
https://www.thoughtworks.com/radar/techniques/managing-stateful-systems-via-container-orchestration
https://www.thoughtworks.com/radar/techniques/managing-stateful-systems-via-container-orchestration
S
SweetOpsover 5 years ago
Hey everyone, give a warm welcome to our newest members!
- @Brandon Hintz
- @Frank Scalzo
- @Harshal Vaidya
- @Bill Rooney
- @Adam Stortz
- @Chad Ostler
Good to have you here =)
- @Brandon Hintz
- @Frank Scalzo
- @Harshal Vaidya
- @Bill Rooney
- @Adam Stortz
- @Chad Ostler
Good to have you here =)
S
SweetOpsover 5 years ago
Hey everyone, give a warm welcome to our newest members!
- @Prasanna Pawar
- @Adam Crown
- @Andreas P
- @Sumeet Shukla
- @sunil Kumar
- @adamdrozdz
- @Guillermo Diaz
Good to have you here =)
- @Prasanna Pawar
- @Adam Crown
- @Andreas P
- @Sumeet Shukla
- @sunil Kumar
- @adamdrozdz
- @Guillermo Diaz
Good to have you here =)
RBover 5 years ago
requesting more slack emojis
G
gyozaover 5 years ago
gyozaover 5 years ago
⏰️ 

S
SweetOpsover 5 years ago
Hey everyone, give a warm welcome to our newest members!
- @Brian Hudson
- @Shajin
- @raphael
- @Thom
Good to have you here =)
- @Brian Hudson
- @Shajin
- @raphael
- @Thom
Good to have you here =)
RBover 5 years ago
anyone ever use celery with sqs ? for some reason my tasks are sent to sqs but not taken from sqs
S
SweetOpsover 5 years ago
Hey everyone, give a warm welcome to our newest members!
- @CWills
Good to have you here =)
- @CWills
Good to have you here =)
S
SweetOpsover 5 years ago
Hey everyone, give a warm welcome to our newest members!
- @Midhun
Good to have you here =)
- @Midhun
Good to have you here =)
Andyover 5 years ago
Hi, does anyone have experience with setting up fail2ban across multiple nginx boxes with a shared database for banned IPs?
niekover 5 years ago
Last week I have create a new Terraform module to host GitHub self hosted action runners on AWS spot instances. For those who are interested the setup is explained in this post https://040code.github.io/2020/05/25/scaling-selfhosted-action-runners
S
SweetOpsover 5 years ago
Hey everyone, give a warm welcome to our newest members!
- @Rob Sung
- @Jonathan Marcus
- @Joel Mataway
Good to have you here =)
- @Rob Sung
- @Jonathan Marcus
- @Joel Mataway
Good to have you here =)
S
SweetOpsover 5 years ago
Hey everyone, give a warm welcome to our newest members!
- @Stephanie
- @Daniel Spence
- @Jonathan Djabbarpour
- @Jacob Harter
- @Zach Zimmerman
Good to have you here =)
- @Stephanie
- @Daniel Spence
- @Jonathan Djabbarpour
- @Jacob Harter
- @Zach Zimmerman
Good to have you here =)
S
SweetOpsover 5 years ago
Hey everyone, give a warm welcome to our newest members!
- @Benjamin Hudgens
- @voron
- @Piotr Maksymiuk
- @Virender Khatri
- @Aaron Brewbaker
- @Yage Hu
- @Bill B
Good to have you here =)
- @Benjamin Hudgens
- @voron
- @Piotr Maksymiuk
- @Virender Khatri
- @Aaron Brewbaker
- @Yage Hu
- @Bill B
Good to have you here =)
muhahaover 5 years ago(edited)
Guys? Is there any tool similar to https://github.com/korfuri/awesome-monorepo#build-systems--dependency-management-tools ?
I have monorepo with separate Terragrunt projects, problem is that I need to:
1. be able to select project to run in pipeline ( not so hard, I can pass $path variable to that project in gitlab )
2. be able to determine which project was changed/impacted in PR ( this is hard, it can be probably done with git diff and bash scripting )
But something that can manage projects in my monorepo and also determine blast radius would be really nice !
Thanks
I have monorepo with separate Terragrunt projects, problem is that I need to:
1. be able to select project to run in pipeline ( not so hard, I can pass $path variable to that project in gitlab )
2. be able to determine which project was changed/impacted in PR ( this is hard, it can be probably done with git diff and bash scripting )
But something that can manage projects in my monorepo and also determine blast radius would be really nice !
Thanks
Shikhar Goelover 5 years ago
Is there a way in helmfile where i can stop it to upgrade job and stateful sets.Actually currently what is happening is that i have labels in helm charts but when i use helmfile to upgrade the deployed helm charts it is failing because job and statefull sets cannot be updated(i.e. cannot add labels in my case).
S
SweetOpsover 5 years ago
Hey everyone, give a warm welcome to our newest members!
- @Rob Faraj
- @Craig Dunford
- @Barak Schoster
- @RAMAN
Good to have you here =)
- @Rob Faraj
- @Craig Dunford
- @Barak Schoster
- @RAMAN
Good to have you here =)
S
SweetOpsover 5 years ago
Hey everyone, give a warm welcome to our newest members!
- @Pyunix
- @Kenny Younger
- @phil
- @atwlam
- @Miroslav Spousta
Good to have you here =)
- @Pyunix
- @Kenny Younger
- @phil
- @atwlam
- @Miroslav Spousta
Good to have you here =)
S
SweetOpsover 5 years ago
Hey everyone, give a warm welcome to our newest members!
- @Tennis Smith
Good to have you here =)
- @Tennis Smith
Good to have you here =)