#aws - February 2022

@Maxim Mironenko (Cloud Posse) has joined the channel

Anyone have advice on setting up a windows EC2 where the server name is changed and then the real bootstrap script is run. Windows needs a restart after changing the server name which is a pain.

i checked online on how i can updated an os on my own when it is marked as required on the maintenance tab tried doing this but it fails to start the update.
I dont know if anyone can help if they have done a New Operating System update for RDS manually before the maintenance_window ?

this OS upgrade

hi folks, anyone knows how i can continue troubleshooting an error i get with sts decode

An error occurred (InvalidAuthorizationMessageException) when calling the DecodeAuthorizationMessage operation: Message is not valid
given by aws sts decode-authorization-message --encoded-message  ?

I'm only trying to copy an AMI from one region to another (it works except when copy to Bahrain). Open a ticket with AWS support but no progress, "it works for me" And sadly i don't see it in Cloudtrail either hence ... stuck ... note i'm Admin so i shouldn't have any issues, no SCP either

hi there. please has anyone seen this error after adding datadog as a sidecar to eks fargate

Your pod's cpu/memory requirements exceed the max Fargate configuration

Hi there, do you know what's OCB Cloudfront in Cost Explorer? The usage type is UE-OCB and the Service is OCBCloudfront.

Anybody using Leapp and in-browser? I started SSO using Brave, switched my default browser to Firefox and Leapp keeps sending me thru Brave to re-authenticate

I’m on AWS and using a bunch of cloudposse modules. I have gotten CodePipeline working with ecs-web-app and Fargate, but it only has 3 stages. I want a fourth stage where I can use the new code before approving and sending it live.

I see several threads here from the past pointing users toward Codefresh instead of CodePipeline for anything beyond the standard 3-stage pipeline, but nothing for the past few years. It being 2022 now, I wanted to check and see if that’s still the recommended approach. It looks like this is an example app using this setup.

Hi there. Does anyone have any experience yet with deploying AWS KMS "multi-region kms key" with EKS clusters? In other words could a multi-region kms key be used for a cluster in us-east-1 and also for another cluster in us-west-2? I am tasked with some research into whether or not this a) even works and b) is an attractive solution. We are currently running a "live" cluster in us-east-1 and a "backup" cluster in us-west-2. Could we run both using the multi-region kms key as the cluster encryption key? Has CP covered this yet with a module, by chance?

Hi colleagues, does anyone know how we can configure our s3 buckets so we can avoid the rate exceeded error?

ThrottlingException: Rate exceeded
        status code: 400, request id

S3 throttles based on prefix, so you can restructure your bucket storage to distribute the objects across smaller prefixes.

Hi colleagues, we got an issue with two REST APIs services can not create connection. Our REST API has deployed in AWS gateway, it was created as HTTPS page with API key. When I try to connect via postman software (only https address and x-api-key are set) from company's intranet (but also tested by tethering with our mobile's 4g internet), it works. But, when NB-IoT supplier try to connect they receive "403 Forbidden". Cloud watch logs shows only our connection, whether that are correct or not correct (e.g. if we try to connect without key).
We dont have any waf configured to protect the api aws gateway, can someone point me to the proper troubleshooting steps please? 🙏

if the logs of api gateway show only your connection try to see if your your supplier is in the same area of you or try using a vpn to see if it's not something else.

Is it possible that these metrics should be different for ECS Fargate tasks, ie. ECS/ContainerInsights vs AWS/ECS namespace ?
https://github.com/cloudposse/terraform-aws-ecs-cloudwatch-sns-alarms/blob/master/main.tf#L57-L58

  ~ resource "aws_cloudwatch_metric_alarm" "cpu_utilization_high" {
      - datapoints_to_alarm       = 2 -> null
        id                        = "linius-services-qa-assembly-cpu-utilization-high"
      ~ metric_name               = "CpuUtilized" -> "CPUUtilization"
      ~ namespace                 = "ECS/ContainerInsights" -> "AWS/ECS"
        tags                      = {}
        # (15 unchanged attributes hidden)
    } 

I’m having a hard time finding / making sense of the difference but alarms aren’t firing as I might have expected with AWS/ECS?

hello all.
we have a blackbox exporter to monitor our apps behind aws elb (classic currently). Sometimes I see i/o timeout on remote app side and no logs on our ELB side. Any idea?

I am seeing quite a bit of InsufficientInstanceCapacity when I try to launch EC2s into a randomly chosen subnet from a VPC. Is there a way I can tell AWS during a RunInstancesCommand or similar command that I want to allow AWS to use any subnet within a VPC? It seems like my only option to put the EC2 into a VPC is to specify the SubnetId param, but ideally I'd like to pass in a list of acceptable subnets, not just one

Has anyone found a tool that can facilitate mass migration of data from one tier of Glacier to the other?

Geez, it is really difficult to completely block public ingress! Any instance with a public IP is reachable, even with no IGW, but if that instance does have a route out a IGW/natgw, the return traffic will succeed also... https://twitter.com/__steele/status/1490566494517825547?t=WAlMvmEq8zOXSdRL8HCiAQ&s=19

Hi All, I’m trying to use https://github.com/cloudposse/terraform-aws-s3-bucket/releases/tag/0.47.0 but when I run terraform with s3_replication_enabled = false , I get the following error: 168: for_each = local.s3_replication_rules == null ? [] : local.s3_replication_rules , I don’t understand what the module is expecting

anyone here has switched from the default 1:1 lambda <> cw log group mapping to N lambdas -> 1 single CW log group -> 1 subscription filter ?


the use case is v simple: consolidate all the logs in one place and then ship them out to 3rd party aggregation instead of having 1 filter per CW group (which goes messy if you have loads of lambdas, not to mention $$)

Hi All, what is the recommended way for EKS pods to CRUD on S3 buckets ? Is this https://github.com/cloudposse/terraform-aws-iam-chamber-s3-role/tree/0.7.5 still being supported ?

[Blog Post] https://www.cloudyali.io/blogs/how-to-get-free-aws-credits

Hi colleagues, has anyone applied the Fail2ban plugin for traefik on EKS? any issues with it? is it production ready? something that we need to have in mind during its implementation? thanks!

Hi all, we have a ElasticBean Stack instance
platform is
tomcat 8.5 with java8 running on 64 bit Amazon Linux.

everything was working fine, but recently.
if we try to access the endpoint rand only we get
Service Unavailable.

when i downloaded the logs by clicking on requist logs
under elasticbeanstack_error_logs
i cloud see following logs-

[Mon Feb 14 10:00:58.338035 2022] [proxy:error] [pid 14882:tid 139757313533696] (13)Permission denied: AH02454: HTTP: attempt to connect to Unix domain socket /var/run/httpd/ (localhost) failed
[Mon Feb 14 10:00:58.338078 2022] [proxy_http:error] [pid 14882:tid 139757313533696] [client <private-ip-here>:12566] AH01114: HTTP: failed to make connection to backend: httpd-UDS, referer: http://<custom-end-point>/1/<name.jsp>?s=sec$$4P!&refresh=300
[Mon Feb 14 10:43:40.663468 2022] [proxy:error] [pid 14882:tid 139757120071424] (13)Permission denied: AH02454: HTTP: attempt to connect to Unix domain socket /var/run/httpd/ (localhost) failed
[Mon Feb 14 10:43:40.663518 2022] [proxy_http:error] [pid 14882:tid 139757120071424] [client <private-ip-here>:21136] AH01114: HTTP: failed to make connection to backend: httpd-UDS

what more I can check to understand what is the issue to solve it.
any help is appreciated thank you.

│ Error: Unsupported attribute
│
│ on main.tf line 16, in locals:
│ 16: info.stage
│
│ This object does not have an attribute named "stage".

atmos terraform apply account --stack gbl-root

atmos terraform deploy account-map --stack gbl-root

at terraform-aws-components, it looks like we should declare 'stage' in account, which is needed by account-map.

maybe, when we didn't declare 'stage' in account, it should be equal to account name.

Hi, im trying to use the cloudposse modules to deploy cloudtrail (with org enabled) but getting errors, having issues.

when i use a custom kms (also deployed via the cloudposse module). It looks to be an issue with IAM needing extra permissions. am i missing something in the module to resolve this or do i need to create a custom IAM and attach it?

Error: Error creating CloudTrail: InsufficientEncryptionPolicyException: Insufficient permissions to access S3 bucket organizationalcloudtrail or KMS key

Id like to log to cloudwatch as well, but again not sure if i should be creating my own IAM and attach for the cloud_watch_logs_role_arn

Thanks for any help

Does anyone know whether it is possible to use a central OIDC provider within member accounts? The aws sts assume-role-with-web-identity call supports a --provider-id flag seems only to be for OAuth2 .. Or should each member account have their own OIDC provider configured?

Hi all,
We are using a webserver with ElasticBeanstalk from 2019.,
the platform is
tomcat 8.5 with java8 running on 64 bit Amazon Linux. Apache as proxy
recently (from Jan 30th) we started getting Service Unavailable issues if go to the endpoint from time to time. and if we refresh 2-3 times it will get resolved on its own.

then I download full logs. under elasticbeanstalk-error_log I can see

[Mon Feb 21 10:00:58.338035 2022] [proxy:error] [pid 14882:tid 139757313533696] (13)Permission denied: AH02454: HTTP: attempt to connect to Unix domain socket /var/run/httpd/ (localhost) failed

[Mon Feb 21 10:00:58.338078 2022] [proxy_http:error] [pid 14882:tid 139757313533696] [client <private-ip-here>:12566] AH01114: HTTP: failed to make connection to backend: httpd-UDS, referer: http://<custom-end-point>/1/<name.jsp>?s=sec$$4P!&refresh=300

[Mon Feb 21 10:43:40.663468 2022] [proxy:error] [pid 14882:tid 139757120071424] (13)Permission denied: AH02454: HTTP: attempt to connect to Unix domain socket /var/run/httpd/ (localhost) failed

[Mon Feb 21 10:43:40.663518 2022] [proxy_http:error] [pid 14882:tid 139757120071424] [client <private-ip-here>:21136] AH01114: HTTP: failed to make connection to backend: httpd-UDS

repeated multiple times from Jan30th.
and when I look at access.log
I can see 503 error log exactly at the same time when permission denied error logs in elasticbeanstalk-error_log
And I looked at the running process using ps -aux | grep httpd and ps -aux | grep tomcat
both are running from 2019 and have no restarts.
what more I can do to troubleshoot these issues

thanks

We all know AWS regions have disparity in service hosting. As of today, us-east-1 (N. Virginia) hosts a max of 306 services while ap-northeast-3 (Osaka) has only 127 services hosted. I needed to answer, for any given AWS service which regions host the service, what services are hosted in a given region. Found a few ways.
https://www.cloudyali.io/blogs/how-to-find-all-regions-an-aws-service-is-available

Is anyone using AWS SSM Session manager to enable devs to connect to a staging RDS instance, and NOT using ssh keys/connections managed through SSM?

Hi, I have a question about VPC peering using multiple state files. We have two VPCs (vpc1 and vpc2) that we want to create peering. The peering was created successfully from vpc2 and I can resolve R53, connect to instances on both VPCs, etc.
The problem is that when I run terraform plan from vpc1 directory, it doesn’t recognize the route tables added by vpc2 and wants to remove them.
How can I get this to work? thanks!

Hello all! At risk of sounding obvious, why is it a best-practice from a compliance/ops standpoint to put all s3 buckets into their own AWS project? I ask because it sort of breaks the terraform mold of working with app-specific buckets (where normally you’d put them under the “dev” or “prod” account, as opposed to in “artifacts”)

Hi Everyone! We just released the brand new UX/UI of Leapp, the local Desktop App that helps you in managing Aws credentials:

https://twitter.com/a_cava94/status/1496846237722632196

check it out and let me know what you think!

does any know about when aurora serverless v2 will go live ?

This may not the place to bring up the issue on https://registry.terraform.io/modules/cloudposse/kms-key/aws/latest where we see a yellow tag in:

Who should see the above?

better to ask this in #kubernetes

Hi all i'm attempting to use eksctl within gitlab pipelines. Curerntly i'm using the alpine/k8s image but im running into the follow error:

Error: checking AWS STS access – cannot get role ARN for current session: RequestError: send request failed
caused by: Post "<https://sts.None.amazonaws.com/>": dial tcp: lookup <http://sts.None.amazonaws.com|sts.None.amazonaws.com> on 172.20.0.10:53: no such host 

Im just using regular IAM user keys as env vars in the project. I dont understand why its looking for a role ARN here?
This post here details the problem, however the solutions all seem like one offs. https://github.com/weaveworks/eksctl/issues/1408

Hi, does anyone know how to list the values of aws:PrincipalTag/* associated with an identity? I’m trying to debug a POC, non-load-bearing AWS SSO connection with JumpCloud and would love some more visibility into what’s going on.