#aws - June 2020 | Slack Archive

88 messages

Discussion related to Amazon Web Services (AWS)
Archive: https://archive.sweetops.com/aws/

May 2020

June 2020

Jul 2020

msharma24over 5 years ago

Do we have any recommendations on how to write into dynamo table using EMR - we are using com.audienceproject:spark-dynamodb_2.11:1.0.1 lib and this takes ages to write into dyanamodb , even though I used provisioned WCU to 100, the consumed write capacity stays < 8 🤔

Prasadover 5 years ago

Hi Folks..the certificate things are pretty confusing if we are new ..are there any difference in usage of the below 2 cli commands? aws iam upload-server-certificate ..... and aws acm import-certificate....?

RBover 5 years ago

Regarding the "Amazon ECS ARN and resource ID settings". Has anyone had any issues turning on the new arn format for ECS container instance, service, or task ?

PePe Amengualover 5 years ago

yes

Zachover 5 years ago

Is there some way to convince AWS console that yes, I will use their new interfaces, please stop reverting to the old one?

madoover 5 years ago

My client just using AWS like RDS to save their customer data (financial service), how to improve the security on it? Any best practice?

RBover 5 years ago

any suggestions on tagging policies ? we use

Name - for whatever
env - for production, staging, etc
role - function
team - team name which corresponds to slack channel

Santiago Campuzanoover 5 years ago

My recommendation on tagging policy…. not too much, not a couple

Santiago Campuzanoover 5 years ago

We have a Billing tag

Santiago Campuzanoover 5 years ago

For costs/billing purposes

Davidover 5 years ago

AWS seems to recommend putting IAM Policies onto Groups, then adding IAM Users to Groups, as opposed to directly adding Policies to the Users.

But this seems in conflict with having a non-increasable service limit where IAM Users can only be in 10 Groups.

How do you all manage situations where you feel a User should be in >10 Groups?

sahil kambojover 5 years ago

Quick question
Do we need security group (ports open) to talk internally in vpc services

RBover 5 years ago(edited)

anyone get logging working between aws and datadog ? looking at datadog's ecs_fargate#log_collection and considering fluentbit over the lambda log forwarder

Joseph Ashwin Kottapurathover 5 years ago

hey everyone,
I am having an issue with an AWS deployment.
I am very new to both AWS and to terraform.
I get this error when I apply the terraform configuration.
I am using the default VPC with a security group with the following configuration:

ingress {
    from_port       = 443
    to_port         = 443
    protocol        = "tcp"
    cidr_blocks     = ["0.0.0.0/0"]
  }

  ingress {
    from_port       = 80
    to_port         = 80
    protocol        = "tcp"
    cidr_blocks     = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

Joseph Ashwin Kottapurathover 5 years ago

https://stackoverflow.com/a/30140563/5356465
according to this, the issue could be with the VPC, but this is the default VPC, isn’t it supposed to have internet access by default? I haven’t associated this with any VPC at all

Joseph Ashwin Kottapurathover 5 years ago

is this because of the VPC or is it related to something else? do you guys need any other information?
if it’s related to VPC, is it because I am using the default VPC and aren’t associating the Elastic Beanstalk environment with any VPC?

Maarten van der Hoefover 5 years ago

I'm looking for experiences with the Amazon Partner Network ? Are there freebies involved with certification ? Other benefits ?

David Medinetsover 5 years ago

Hello. Just found you. Would you have any idea about using AWS CloudWatch Agent on Fedoura CoreOS? I have the SSM Agent working but I want to copy logs file to CloudWatch and SSM is deprecated. Wish I knew that last night.

PePe Amengualover 5 years ago

SSM is deprecated?

PePe Amengualover 5 years ago

the log agent part, yes

David Medinetsover 5 years ago

I figured out how to get both ssm and cloudwatch agents running on FCOS.'

RBover 5 years ago

Any objections with storing secrets using kms encryption in s3?

Mike Schuelerover 5 years ago

hello. migrating from my self hosted k8s cluster to EKS. considering switching my ingress controller. I guess the popular approach is to use a hybrid ALB + nginx setup, best of both worlds.

got 2 questions

1. it seems there are two different nginx ingress.
https://github.com/kubernetes/ingress-nginx/

https://github.com/nginxinc/kubernetes-ingress

the one released by nginx team looks interesting, wondering if anyone has experience with it? not sure if it’s even worth considering if i’m not getting the paid version

2. while the hybrid approach seems like the way to go, i haven’t been able to find steps online to setup. i guess the AWS way is to just use the ALB ingress controller. anyone point me in the right direction? or convince me i should just use the ALB ingress?

Mattover 5 years ago(edited)

I have an existing Beanstalk application which is running as single container applications. I just enabled CloudWatch logging and now all log data is streaming to CloudWatch (nginx, docker, etc) except for the log messages written to stderr/stdout. In other words the log messages I care most about.

I can see exactly what's wrong, the config file for the Cloudwatch agent (/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml) is configured to track a stderr/stdout log file at this path /var/log/eb-docker/containers/eb-current-app/stdouterr.log. However, my applications are logging to this path /var/log/eb-docker/containers/eb-current-app/eb-f54b7e030fd5-stdouterr.log

The CloudWatch log group uses the path with 'eb-current-app/stdouterr.log' in it (no surprise there).
The f54b7e030fd5 is the (short) ID of the running Docker container.

Does anyone know how I can configure this? I'd like to do it in a way that will of course survive beyond containers and instances as they spin up and down. I can define additional log files in .ebextensions which allows Beanstalk to become aware of non-standard log files but that would require me knowing the container ID ahead of time i.e. when bundling the Beanstalk application version. 
It seems like the best fix would be to force Beanstalk to log stderr/stdout from the container to the path which CloudWatch expects.

Mattover 5 years ago

Fun with Beanstalk. . .

mkover 5 years ago

did anyone use CW>>lambda>>firehose>>splunk integration? We have an issue where the logs have a delay around 3m. Is there a better solution for app logs from AWS to splunk without any delay. I've used kinesis streams in the past but seems like the driver isnt supported and if the log data is huge it seems to be dropping logs

Matt Gowieover 5 years ago

Any Storage Gateway File Gateway users? Wondering if I can pick somebody brain… My file gateway keeps getting created with a private IP as if I’m trying to associate it to be an internal / VPC gateway which is not what I want.

RBover 5 years ago

In the coming few months, AWS Fargate will update the LATEST flag to Platform Version (PV) 1.4.0. This means all new Amazon Elastic Container Service (ECS) Tasks or ECS Services that use the Fargate launch type and have the platformVersion field in their Task Definition set to LATEST will automatically resolve to PV 1.4.0. For customers who use Amazon VPC Endpoints along with their ECS tasks running on Fargate, the new platform version has changes that may require customer action. For more information see the FAQs below. If you do not use VPC endpoints for Amazon ECR, AWS Secrets Manager or AWS Systems Manager no action is necessary.

PePe Amengualover 5 years ago

Hi I have an interesting problem, I’m enabling IAM auth for RDS and I have users that have U2F keys so the aws cli does not support them and so they can’t get a token for the RDS host, BUT they do have console access so I was wondering if there is a way to run trough the console the the command to get the token and then they can use it to connect to the host aws rds generate-db-auth-token is the command. I was thinking that maybe an SSM doc or like a container they can fire up and get the token or something like that, I do not want to use a Bastion host for this

mkover 5 years ago

does anyone know if we can heapdump or thread dumps from fargate?

Stevenover 5 years ago(edited)

As long as you can remotely trigger it and output to STDOUT, it would be possible. But haven't done myself. We use APM and haven't needed to dig deeper yet

Robert Horroxover 5 years ago

what are peoples thoughts on using https://eksctl.io/ vs terraform modules to setup an EKS cluster? Pros/cons

Maciek Strömichover 5 years ago

theoretically, would you replace web -> sqs -> worker -> firehose with a simple web -> firehose approach? I’m load testing the app currently with double the traffic I have normally and altho everything seems stable I’m having second thoughts about it because of increased latency (not much just 100-150ms on avg).

Chris Fowlesover 5 years ago

firehose is designed for handling large amounts of data ingest - so if you're putting things in front of it that's probably defeating the purpose somewhat

Chris Fowlesover 5 years ago

where's it going after firehose?

Maciek Strömichover 5 years ago

previously the sqs -> worker -> firehose/elasticsearch made sense becauyse of how the elasticsearch indexes were created

Maciek Strömichover 5 years ago

and firehose wasn’t able to deliver messages the way we needed into the index but since we have dropped elasticsearch I started to question the architecture

Chris Fowlesover 5 years ago

i'd say you're probably running more than you need to, and bottle-necking firehose. this is without knowing your specific use case though

Maciek Strömichover 5 years ago

also spending several k more per month for maintaining worker instances

Chris Fowlesover 5 years ago

yup

Bircan Biliciover 5 years ago

AWS, Create IAM Role API Collapsed
https://stackoverflow.com/questions/62341775/aws-create-role-rate-exceeded?answertab=votes#tab-top

Raymond Butcherover 5 years ago

How do people use and switch between different AWS accounts (hundreds) in the AWS console?
We currently have a CLI command that opens the browser for the specified AWS profile but it's not great.

niekover 5 years ago

Anyone any advise what to use for encrypting environment variables for AWS lambda, so any opnion about using SSM with KMS or AWS secret manager?

Maciek Strömichover 5 years ago(edited)

anyone using aws ecr get-login in a Makefile here? I’ve this annoying issue that whenever I execute any aws cli command that assumes a role and the source_profile is MFA protected I get following error:

aws ecr get-login --no-include-email
Enter MFA code for arn:aws:iam::AWS_ACCOUNT_ID:mfa/USER:  docker login -u AWS -p [...]
make: Enter: No such file or directory
make: *** [ecr_login] Error 1

here’s my makefile target

ecr_login: AWS_REGION ?= us-east-1
ecr_login:
	$(shell docker-compose run --rm -e AWS_PROFILE=$(AWS_PROFILE) $(DEV_IMAGE) aws ecr get-login --region us-east-1 --no-include-email)

as far as I can see it fails because I’m trying execute the output of the command which is returned to STDOUT but the Enter MFA code is also provided via STDOUT and the execution going nuts. 2nd execution succeeds because boto3 stores a session and doesn’t ask for MFA for another hour. What’s also puzzling is that the Enter MFA information is not shown and you need to assume it’s there because terminal ‘hangs’ while expecting an input.

Eric Bergover 5 years ago

I'm working on tightening up our NACLs and was trying to find out whether NACLs apply to load balancer ingress. I have the same question regarding the new Transfer Servers VPC; you can configure a security group, but it's not actually in a subnet, so NACLs shouldn't apply. Am I off-track here? Thanks, as always.

Chris Fowlesover 5 years ago

Most of the time NACLs are not a great pattern to follow.

Chris Fowlesover 5 years ago

Trying to tighten nacls beyond "this subnet should only receive traffic from this other subnet" is generally going push down more on the reduced functionality side of the functionality vs security balance.

Chris Fowlesover 5 years ago

(this is as a warning from someone with burnt fingers - not a criticism)

sahil kambojover 5 years ago

hey guys
Need help regarding Application Load balancer
i have setup aws acm and attach it to the listener of alb on port 443, which is redirected to target group for 443
and a simple listener for port 80 redirected to target group on port 80
target group(instances are same but on different port 80 and 443)
instances are serving webapp on port 80
I am getting 502 with https
and working well on http(which not showing ssl)

what i am doing wrong?

sahil kambojover 5 years ago

should i forward alb 443 listener to 80 listener?

Zachover 5 years ago

omg
https://aws.amazon.com/about-aws/whats-new/2020/06/amazon-ec2-auto-scaling-now-supports-instance-refresh-within-auto-scaling-groups/

Chris Fowlesover 5 years ago

omg!

Chris Fowlesover 5 years ago

i was writing my own ASG roller

Chris Fowlesover 5 years ago

now i can do something else 😄

Chris Fowlesover 5 years ago

😛

Zachover 5 years ago

Question now is how long will it take the AWS provider to support it in terraform

lorenover 5 years ago

I'm kinda curious what it would even look like in terraform... You still have to change the launch config... So, would it be a non-replacing change to that property of the asg? Maybe send the StartInstanceRefresh command instead of force replacing the asg?

lorenover 5 years ago

Here's a better article describing how the feature works, strangely not linked from the what's new post... https://aws.amazon.com/blogs/compute/introducing-instance-refresh-for-ec2-auto-scaling/

Matt Gowieover 5 years ago

SWEET. Stoked this is around. “Instance Refresh” seems like a pretty poor name for this though…

Igorover 5 years ago(edited)

Couldn't you just scale up with the new launch template and then scale down? That's how we refreshed instances.

RBover 5 years ago

has anyone tried this with ECS EC2 instances to see if those will be adequately drained ?

Zachover 5 years ago(edited)

Is there a good pattern for managing DataPipelines for EMR between dev/staging/prod in some sort of sane manner? Its simultaneously ‘infrastructure’ and software all rolled up in one. We tried doing it all in terraform but its a mess because the Data team has no idea how to maintain that, and its got like 100 more options than they need … and then we’re stuck trying to use terraform as a config mgmt tool

Igorover 5 years ago

Woah. I just noticed https://aws.amazon.com/about-aws/whats-new/2020/06/introducing-aws-codeartifact-a-fully-managed-software-artifact-repository-service/ Per usage pricing

RBover 5 years ago

So we have to maintain a list of iam roles that access our s3 bucket within kms or those roles will have s3 access but not decryption access

aws iam list-entities-for-policy --policy-arn arn:aws:iam::snip:policy/s3-access --query 'PolicyRoles[].RoleName' | egrep -v '^\]|^\[' | cut -d'"' -f2 | sort | uniq

we have to manually update the kms policy based on this list. is there a way of automating this to keep our kms policy in line with any iam role containing the policy ?

Harryover 5 years ago

Does anyone here know how to add instances to Systems Manager? I’ve got a few showing up somehow but the majority of my instances aren’t in the list and I’m not sure if it’s something I did in the web interface that added them or something I changed on the box somehow.

RBover 5 years ago

Regarding fluent-bit pushing logs from ECS to Datadog, we noticed there is a log key being used for every log entry. Is there a way to rename this to datadog's expected msg key ?

Brandon Wilsonover 5 years ago

Anyone here use Client VPN with Transit Gateway? It doesn’t look like Client VPN is supported as a transit gw attachment. So I’m trying to think of creative ways to make it work.

Pierre Humberdrozover 5 years ago

Did someone here already run the a1 instances with eks ? How does it full to run k8s on arm?

Santiago Campuzanoover 5 years ago

Hello there ! Does anyone know how to enable S3 access logs to a bunch of S3 buckets (100+) I don’t want to go one by one through the AWS UI ?

Santiago Campuzanoover 5 years ago

Seems like the aws cli does not have an option for doing that

Haroon Rasheedover 5 years ago

Hi folks - I have general question on AWS networking. Is there any limits on how many packets per seconds allowed for a AWS ENI?

RBover 5 years ago

anyone have thoughts on cost cutting of fargate services ?

Ryan Smithover 5 years ago

Are you able to use API Gateway to point directly to Target Groups (specifically, I'm curious about getting to ECS services WITHOUT using a load balancer in front of ECS)?

sahil kambojover 5 years ago

Hey Guys
aws rds (mariadb10.4) root user by default cant give GRANT ALL permission
how can we make it do that
I need grant all permission

Maarten van der Hoefover 5 years ago

Does anyone have SAP-C01 dumps to share ?

Matt Gowieover 5 years ago

For anybody using ECS — I released a side-project over the weekend that may help you avoid writing an ugly bash script in the future: ecsrun. It’s a small golang CLI tool that provides a wrapper around the ECS RunTask API and it aims to be much more easily configurable than invoking the AWS CLI yourself. It enables invoking admin tasks like database migrations, one-off background jobs, and anything similar that you can cram into a Docker CMD. I’m eager to get some feedback if anybody ends up using it!

https://github.com/masterpointio/ecsrun

RBover 5 years ago

One tool I've been looking for is one to update a task definition's single container definition's container image. Currently were using ugly fabfiles that do this that are copied and pasted everywhere and they typically recreate the task definition instead of reusing the one in terraform.

Joe Nilandover 5 years ago

We have a node.js worker that runs jobs that can take hours and don’t tolerate interruption. It has a graceful shutdown function which is called when catching sigterm and sigkill.

We’re currently running the worker on ECS Fargate.

We want to make sure the containers aren’t killed mid job. With the standard ECS deployment methods this isn’t possible.

Is anyone aware of a good way to achieve this? I’m looking into the external ECS deployment controllers. Perhaps Fargate is not a great choice for this however the client doesn’t have an Ops team, so if there’s any way to use it that’d be ideal.

JMCover 5 years ago

Hey is this what I think it is ?

https://aws.amazon.com/about-aws/whats-new/2020/06/amazon-ec2-auto-scaling-now-supports-instance-refresh-within-auto-scaling-groups/?sc_channel=em&sc_campaign=GLOBAL_CT_NL_global-snapshot-newsletter_20200624_&sc_medium=em_264309&sc_content=PA_nl_la&sc_geo=mult&sc_country=global&sc_outcome=pa&trk=em_264309_()_Velocity_WhatsNewForYou_Compute_1&mkt_tok=eyJpIjoiTTJVek5EQTFOV0ZsWWpJMSIsInQiOiJiV2hHR1wvNENYUTd6NVkrN25OQ0lKcURMdEtzQjhwVGxDbVFxYlhKTzV2UU5BSm5PZUxnWWNDZTZOaWR0N2lTYiswb1daS2w4YmJtOHdRTm1WVTMrWVRyODI5d2lvYVwvWXNOS1VXMTZHRDRSc2FyU2VZQkg3N0dzVnVYUzArRStlTmFzSndnNEErbkpvTndQZ2ttSVg4UT09In0%3D

Update AutoScalingGroup & launch template in place without the hassle or swapping them, or tweak around with terraform prefix / create before destroy ?
😯

JMCover 5 years ago

Anyone knows if Terraform will support this anytime soon in the V13 ?

David Medinetsover 5 years ago

Does anyone know the password for the 'centos' account in the Centos AMI? I want to remove NOPASSWD from the sudo function.

RBover 5 years ago

idk if this has been mentioned but this really makes a difference when all i get is an instance id from aws in an email

https://github.com/bash-my-aws/bash-my-aws

$ AWS_REGION=us-east-1 instance-tags i-snip
i-snip  env=snip snip=snip snip=snip snip=true snip=0.56 team=snip Version=snip service=snip CreatorName=snip aws:ec2spot:fleet-request-id=snip CreatorId=snip snip=snip

Michał Czeraszkiewiczover 5 years ago

Hi, does anyone know how "real time" the AWS Cost Manager is? For example when I launch an ECS task, should I see the cost already after 1h? Or hast it some delay - for example I need to wait 24h until I see the cost appear in AWS Cost Manager?

jedineeperover 5 years ago

Kind of worried about task definitions in ECS, is there a maximum number they can go up to? Do I need to worry about this?

Nover 5 years ago(edited)

@channel what would be your choice for building a devops pipeline , use gitlab and go with a single vendor approach or use a combination of tools to keep it flexible? Thanks in advance

RBover 5 years ago

Nover 5 years ago

@RB how's your experience with it ?

May 2020 Browse by date July 2020