#aws - October 2020 | Slack Archive

88 messages

Discussion related to Amazon Web Services (AWS)
Archive: https://archive.sweetops.com/aws/

Sep 2020

October 2020

Nov 2020

RBover 5 years ago

How do people get all instances using amzn linux 1? I can get a list using ssm from command line but id prefer seeing it as a tag. Any recommends for tagging all ssm instances with their platform version?

Maarten van der Hoefover 5 years ago

3 free dev courses for aws cloud native development, they look quite nice on the surface:
https://www.edx.org/course/building-modern-nodejs-applications-on-aws
https://www.edx.org/course/building-modern-python-applications-on-aws
https://www.edx.org/course/building-modern-java-applications-on-aws

Erik Osterman (Cloud Posse)over 5 years ago

Our very own @Adam Blackwell works there :-)

Milosbover 5 years ago

Guys, how do you organize lambdas code? Do you prefer single git repo for all/multiple functions, or you like to keep it separate? We have 50+ functions. I like to separate everything, but developers like to keep everything in once place 🙂

Zachover 5 years ago

are they related?

Zachover 5 years ago

I could see a single repo if they were all part of an API or workflow

Milosbover 5 years ago

No, all functions are independent. Thing is that for developers obviously cloning and maintaining 50+ repos wouldn’t be OK. On the other side there is no clean way to maintain CI/CD from my side if we have single repo for all functions.

sheldonhover 5 years ago

Is using pager duty for non actionable alerts and antipattern? Let's say iam policy changes. Mostly is information and should just be acknowledged unless in rare case it is a problem. Would using pager duty vs just sending a notification to slack/teams be good to you and then open incident IF warranted, or would you have it flag in pagerduty regardless?

Personally I lean towards only actionable priority issues going in pagerduty, but wondering how others handle that. I've been playing with marbot and it made me think about the difference between something simple and notifying and something like pager duty that tends to be a lot more complicated to get right.

sheldonhover 5 years ago

Is cloud custodian better for custom rules notifications and config rule creation over doing with RDK or manually?

Mikhail Naletovover 5 years ago

Hey! Is anyone using AWS CodeDeploy? I'm trying to understand how to clarify which alarm was triggered while the version was deploying.

Jonathan Marcusover 5 years ago

I want to do per-user rate limiting. AWS WAF does per-IP rate limiting (which is important as well) but users authenticate with Cognito JWTs and it would be great to have a user-aware limit. Any ideas?

RBover 5 years ago

https://stackoverflow.com/questions/64229642/what-is-the-best-approach-to-track-descendents-of-amis

Matt Gowieover 5 years ago

Hey folks, what AWS service(s) should I be looking to utilize for ensuring notifications / alerting around AWS account changes surrounding the following:

1. CloudTrail configuration changes
2. Security Group Changes
3. AccessKey Creations
Some background: A client of mine is currently PCI compliant and they have CloudWatch Alarms / SNS Email Topics for alerting around the above changes, but they’re not in Terraform and we’re migrating all their ClickOps, poorly named resources over to Terraform. Now I could have one of my junior engineers create these same alerts through terraform, but I feel like there is a better way. Control Tower? AWS Config?

Joe Nilandover 5 years ago

https://www.sdxcentral.com/articles/news/palo-alto-networks-hacks-aws-exposes-multi-million-dollar-misconfigurations/2020/10/

What do you guys think the exploit is exactly?

Maarten van der Hoefover 5 years ago

I was recently discussing if there is room for a (OSS) tool for developers which builds the infrastructure up, multi account etc, configs, guardduty, transitgw. The argument against a tool like this was control tower which I haven't checked myself yet. I'm curious to know other's opinion regarding Control Tower & Landing zone.

RBover 5 years ago

anyone here do any integrity checking on binaries on golden amis ?

Marcin Brańskiover 5 years ago

Amazon Timestream is now Generally Available
Anyone was using it during beta? Any insight?

Igorover 5 years ago(edited)

Has anyone had aws-vault be tagged as a virus by Microsoft Defender?

Igorover 5 years ago

(I know running it in Windows is a bit weird)

Alex Jurkiewiczover 5 years ago

We have a centralised auth AWS account which has IAM users. These users get allocated permissions to sts:AssumeRole into our other AWS accounts where real infrastructure is kept.
We have a 1:1 mapping of roles to an IAM group. So to allow a user to assume a particular role, we add them to the corresponding group.
The problem with this design is users can only be part of 10 groups.
Anyone have a similar central auth AWS account? How to do you manage who can assume what in a scalable way?

lorenover 5 years ago

Slick, https://www.hashicorp.com/blog/aws-lambda-extensions-for-hashicorp-vault

reiover 5 years ago(edited)

Hi, I have a question regarding AWS Loadbalancers (ELB, classic):
It is normal, that the connection/response rate for a newly spanned LB is very slow?
After creating an EKS cluster and attaching an ELB to it I was only able to query a website (HTTP connection to a container) at about 1 request every 5 seconds
Now after some hours I have a normal response time of about 0.08s per request

Used this simple script: while true; do time curl <http://app.eks.example.com|app.eks.example.com>; done

RBover 5 years ago

anyone do integrity checks on their packed amis ?

RBover 5 years ago

for sharing amis across accounts, do people simply set the ami_users argument in packer or is there a better way to share AMIs across accounts ?

Emmanuel Gelatiover 5 years ago

is there any option to change metricsListen in eks without using kubectl edit cm?

lorenover 5 years ago

Useful for practicing queries, developing response plans? https://summitroute.com/blog/2020/10/09/public_dataset_of_cloudtrail_logs_from_flaws_cloud/

Valter Silvaover 5 years ago

Hello team, I am pleased to announce the official release of CLENCLI. A command-line interface that enables you to quickly and predictably create, change, and improve your cloud projects. It is an open source tool that simplifies common tasks that many Cloud engineers have to perform on a daily basis by creating and maintaining the code structure and its documentation always up-to-date. For more details please check the project on Github: https://github.com/awslabs/clencli. I would love to hear your feedback and which features would you like to see implemented in order to make your life in the cloud easier.

sheldonhover 5 years ago

If you didn’t have full control of infrastructure as code for your environment and all the places a common security group might be used would you:

• Gitops workflow for the shared security groups so that submissions for changes go through pull request
• Runbook with approval step using Azure Pipeline, AWS SSM Automation doc or equivalent that runs the update against target security group with basic checks for duplicates etc.
I want to move to more pull request driven workflow, but before I do it this way, I’m sanity checking to see if I’m going to set myself up for failure and be better off with a runbook approach if I can’t guarantee the gitops workflow is the single source of truth period.

U010W9VSBTLover 5 years ago(edited)

Team with zero experience of kubernetes is told to implement a gitops workflow with EKS + Airflow. Most experience is some windows ECS cluster build through terraform for a few folks. Level of difficulty to correctly setup infra-as-code

Alex Jurkiewiczover 5 years ago

I need this as a keyboard macro whenever interacting with AWS support is on the menu:

Hello, I'm Alex. How are you going? I'm going well

They don't answer my questions until we exchange these messages. And I'm only half joking 😅

Ennioover 5 years ago

Hi all,
I've an ASG on eu-west-1 based on spot instances c5.xlarge and I'm experiencing a termination of 2 instances roughly every 30 mins(Status: instance-terminated-no-capacity). Now I added another instance type and seems a bit better.
Are you aware of a way to check spot instances capacity for a given AZ and instance type?

Matt Gowieover 5 years ago

Anybody use a Yubikey w/ AWS Vault? Is there a better prompt on MacOSX than osaprompt? It’s bugging me that my machine won’t let me paste into that text box.

sheldonhover 5 years ago

Need windows + linux administration, inventory, configuration management etc. AWS SSM has lots of painpoints.
Would I be best served looking at Opsworks Puppet enterprise in AWS for managing this mixed environment?

For instance, someone asked me the state of windows defender across all instances.
With SSM it’s pretty clunky to build out a report on this, inventory data syncing, athena queries, maybe Insights or other product just to get to useful summary.
Will puppet solve a lot of that? The tear down and up rate is not quick enough and the environment mixed enough that managing through SSM Compliance and other aws tools is painful.

sheldonhover 5 years ago

Is there any easy button to getting Grafana + Influx up in AWS (single cluster) for fargate or something similar? I’ve found Gruntworks module but at first look it’s all Enterprise.
I’m working through a Telia module but nothing quick, as I need to update the grafana base image and other steps. Kinda been blocked on making progress due to it being side project.

RBover 5 years ago

how do you all use exported creds in local development?

Laurynasover 5 years ago

Hi, does anyone has experience in buying reserved instances/ saving plans? I find it confusing because there are so many different options! 😄

Tim Birkettover 5 years ago

Hey 👋 - I'm looking at getting EFS up and running cross account but need to avoid the hacking of /etc/hosts as we're planning on using the EFS CSI Driver to make use of EFS in Kubernetes. The AWS documentation is pretty lacking in this area and mentions using private hosted zones, but doesn't really go into any further detail. Does anyone have experience of cross account (or VPC) EFS and route 53 private hosted zones that could offer a bit of insightful wisdom?

Steve Wade (swade1987)over 5 years ago

does anyone have at their disposal a completely readonly IAM role?

Stan Mover 5 years ago

hi, is this the latest and greatest way to setup aws <-> g suite sso for console and cli or are people using something else? https://aws.amazon.com/blogs/security/how-to-use-g-suite-as-external-identity-provider-aws-sso/

RBover 5 years ago

anyone use aws batch ?

sheldonhover 5 years ago

If anyone has some Cloud custodian custom rules for AWS SSM inventory I could use it. I want to deploy an AWS configurable that checks SSM inventory on all windows instances for a specific role. I found the cloud formation schema but we'll have I'm guessing an hour to figure all that out.

I would also like to know if anybody uses AWS config to define desired state inside a server to be validated. Let's say I want to have a service running inside server. To test compliance of this would AWS config be the incorrect approach and instead it would just be a cloud watch alarm/event/opscenter? The boundary for what is the appropriate solution is a bit muddy as usual with aws 😁

sheldonhover 5 years ago

Is anyone using aws opscenter regularly? I've been exploring and love the level of information you get on an issue and the list of associated automation runbooks. Turn off all the defaults and setup a few opscenter items on key operations issues to handle and it seems much promising.

Pager duty would require an ton of work to get there. Thinking of trying to trigger opscenter item for say a disk space issue would be so much more effective than a pagerduty alert for equivalent.

I want more runbook automation steps associated with issues and don't see easy way to promote in pagerduty in comparison to opscenter. I know you can add custom actions to run but it's not the same thing.

Nitin Prabhuover 5 years ago

👋 We have a scenario where we have 3 services (elasticsearch. Kibana and APM server deployed on EKS cluster) and we want to expose it to public using AWS ALB.
Would you prefer 3 ALBs one for each service or one ALB handling all 3 services ? what would be your take on this ? We have tried both way it works and we see one ALB for all 3 services is less code + less cost but not sure of any downsides of this

Nitin Prabhuover 5 years ago

thanks for your help

sheldonhover 5 years ago

AWS orgs....Not sure yet how to use properly though to run commands across all environments. I have need to run the New-SSMInventoryDataSync so all my regions and accounts in org sync to common bucket.

While I enabled ssm across all accounts it didnt give me anything in the orgs screen to setup sync. Am I going to have to run a loop through every region and account now and deploy via CF or cli the bucket datasync or is there an easy button with orgs for this? Seems wierd to have to do so much per region work while the org setup was a couple clicks

Yonatan Korenover 5 years ago

Are there any compelling reasons to use a NAT gateway over a NAT instance when outbound traffic is not significant enough to justify the increased cost for switching over to a NAT gateway?

Yonatan Korenover 5 years ago(edited)

I personally prefer simplicity, support, and best practice, and as such if I’m not paying for it myself, would go for a NAT gateway. However I have met co-workers and clients who are more cost-sensitive.

PePe Amengualover 5 years ago

A Nat instance is like 20 bucks a month and you do not have to manage it

PePe Amengualover 5 years ago

4 lattes

Yonatan Korenover 5 years ago

Yeah since both are hands-free this discussion will always surround cost until multi-AZ redundancy enters the equation

RBover 5 years ago

always want to be careful with cost savings. i run into these issues too regarding costs. what's a more political/nicer way to say

penny wise pound foolish

RBover 5 years ago

maybe simply "best not to penny-pinch"

PePe Amengualover 5 years ago

A Nat instance will require maintenance, patches, access management etc

Yonatan Korenover 5 years ago

The penny pinching that really hurts is that which introduces differences between DEV, STAGE, PROD - unless the differences are properly abstracted by Terraform modules, for example. But even that adds additional complexity.

Yonatan Korenover 5 years ago(edited)

So when a client says “My Staging env has the following (list of significant differences) than Prod because I save $200 a month”… it’s rather concerning. Of course not every organization is large or VC funded, but usually these differences make it a headache to iterate on infrastructure changes across both environments.

RBover 5 years ago

interesting so your environments are setup into different VPCs. you have 1 for prod, 1 for staging, etc. so if you need a NAT, you'll need one for each VPC which is for each environment

RBover 5 years ago

i'd still say it's worth the cost. what would be an alternative ? and what would the cost of maintenance be to use the alternative ?

Chris Wahlover 5 years ago

You could also deploy a transit gateway with one VPC hosting the IGW / NAT GW for multiple other VPCs.

kalyan Mover 5 years ago

Dockerizing nodejs Code in Production with or without PM2?

Davidover 5 years ago

I added Cognito auth to some of our dev sites via an ALB listener rule.

How do I have e2e tests (using cypress) authenticate through the Cognito redirect?

Matt Gowieover 5 years ago

Hey @Andriy Knysh (Cloud Posse) — did you folks at CP need to do anything special to use the terraform-aws-datadog-integration module with 0.13? I’m continuing to get the below error when trying to use the latest (0.5.0) even though I’m specifying what I believe to be the correct required_providers configuration (below as well).

  required_providers {
    datadog = {
      source  = "datadog/datadog"
      version = "~> 2.13"
    }
   ...
  }

Error: Failed to install providers

Could not find required providers, but found possible alternatives:

  hashicorp/datadog -> datadog/datadog

If these suggestions look correct, upgrade your configuration with the
following command:

The following remote modules must also be upgraded for Terraform 0.13
compatibility:
- module.datadog_integration at
git::<https://github.com/cloudposse/terraform-aws-datadog-integration.git?ref=tags/0.5.0>

sheldonhover 5 years ago

How do I pass for_each region on this provider run the resource/module. I tried with module and it’s not workign

sheldonhover 5 years ago

Basically I need an example of creating a resource for each region AND giving it an explicit provider (as this plan handles multiple accounts, each with it’s own file)

jonjitsuover 5 years ago

Are there any metric collection agents out there that can send postgres metrics to CloudWatch as custom metrics? I'm trying to avoid having to write this myself...

Matt Gowieover 5 years ago

Anyone know of any open source datadog monitor configurations for AWS resources like ALBs, NLBs, WAF, etc?

Zachover 5 years ago(edited)

https://aws.amazon.com/blogs/aws/public-preview-aws-distro-open-telemetry/
https://aws-otel.github.io/

AWS Distro for OpenTelemetry is a secure, production-ready, AWS-supported distribution of the OpenTelemetry project. Part of the Cloud Native Computing Foundation (CNCF), OpenTelemetry provides open source APIs, libraries, and agents to collect distributed traces and metrics for application monitoring. With AWS Distro for OpenTelemetry, you can instrument your applications just once to send correlated metrics and traces to multiple monitoring solutions and use auto- instrumentation agents to collect traces without changing your code.

Chris Fowlesover 5 years ago

... yet another aws github org?

Zachover 5 years ago

gotta HA those githubs

Zachover 5 years ago

never know when one will go down

PePe Amengualover 5 years ago(edited)

OMG this is stupid

https://docs.aws.amazon.com/cli/latest/reference/ecs/update-service.html

Warning

Updating the task placement strategies and constraints on an Amazon ECS service remains in preview and is a Beta Service as defined by and subject to the Beta Service Participation Service Terms located at <https://aws.amazon.com/service-terms> ("Beta Terms"). These Beta Terms apply to your participation in this preview.

every single tool for ecs deployment that I know uses update-service I guess now if something does not work I could say “ahhhhhh is a beta feature”

PePe Amengualover 5 years ago

I’m so glad amazon does not build cars or hospital equipment

Christopherover 5 years ago

Anyone know of any courses/tutorials that will walk you through building various different AWS architectures. Ideally with some kind of IaC (Cloudformation/CDK etc), and CI/CD to deploy/test this before it’s released?

I’m trying to build something at the moment, but I don’t know enough about AWS, IaC, CI/CD to do it, and could do with following some guides first I think as it’s taken me 1.5 days to create 4 resources in AWS 😭

kalyan Mover 5 years ago

what are some of the must have polices in a initial AWS account

Matt Gowieover 5 years ago

Has anyone requested a monthly SMS spending quota increase for Amazon SNS via Terraform?

aws service-quotas list-service-quotas --service-code sns

Tells me there is no service quotas I can request… and all the docs I’ve found haven’t mentioned an API / CLI option. I’m really hoping this isn’t one of AWS’ annoying manual steps.

diogofover 5 years ago

Hey there, thank you for this community

diogofover 5 years ago

Regarding terraform-aws-ecs-alb-service-task I would like to know if I can create the NAT Gateway only on one subnet?
This would allow for a cheaper environment, as the normal is to create 2 NAT gateway

Yoni Leitersdorf (Indeni Cloudrail)over 5 years ago

Did anyone else start getting a flood of emails about AWS retiring some ECS-related infrastructure and asking for tasks to be restarted?

U010W9VSBTLover 5 years ago(edited)

How do you update AWS ECS Fargate?

Troy Tailleferover 5 years ago(edited)

Not sure ecs deploy is the same as bleu green codedeploy. To be fair the codepipeline is itself deployed with either cloudformation or terraform

kalyan Mover 5 years ago

How do we setup aws-vault with role and mfa to generate temporary credentials. with out generating any accesskeys and secret keys from user.

Milosbover 5 years ago

Did anyone have issues with iam roles for service accounts in kubernetes? From some reason it fails over to instance profile instead. I suspect there is some bug in aws-sdk for javascript, but can't confirm yet

Daniel Pilchover 5 years ago

Does anyone know of a robust solution for automatically mounting and formatting new EBS volumes when a new instance is first started?

kalyan Mover 5 years ago(edited)

How to evaluate a right monitoring tool for your environment? I am working on Microservices Kubernetes environment. how do we evaluating a right Monitoring tool for prod environments. They are many sass companies, opensource tools out there providing same set of features. Any suggestions on how to select a sass product?

RBover 5 years ago(edited)

does anyone use any tools to rotate ecs ec2 instances ?

I found this today and wondering how other devs do this

https://github.com/chair6/ecsroll

Steve Neuschotzover 5 years ago

Reaching out to the group with a very edge case question - Has anyone deployed a hardened image, such as the CIS Amazon Linux 2 image, inside a Managed Node group for EKS? I ask the question because I want to use this image but I am not sure of how to install the required Kubernetes components (kubelet and kube-proxy) , which of course come preinstalled with the Amazon Linux 2 AMI. I also am not sure I can Terraform the cluster and node group using the CIS image. I would appreciate any help anyone has to offer!

kalyan Mover 5 years ago

AWS VPC creation with IPv4 CIDR range with 10.0.0.0/16 vs creating VPC 172.31.0.0/16 which is recommended to use with aws ecs clusters.

Maciek Strömichover 5 years ago

https://aws.amazon.com/blogs/aws/new-application-load-balancer-support-for-end-to-end-http-2-and-grpc/ woohoo

sheldonhover 5 years ago

I have a lambda that is running data collection. I want to run it 1 of the 20 combinations at a time on each server at time so that I minimize traffic. Aws, events, etc any tip on what would let me do this?

September 2020 Browse by date November 2020