#aws - April 2021

Does anybody know if there is an AWS provided SSM parameter for the elb-account-id like how they provide SSM parameters for AMI IDs?

Just curious if people have purchased AWS RDS Reserved Instances before, any best practices / pitfalls to share.
• I understand that we cannot use AWS savings plans, only RI's.
• do you guys standardize on a specific instance size (ie. m4.2xlarge) so you don't waste reservations?
• confirm my understanding that if we purchase an RDS RI, it'll automatically apply to existing instances? We don't need to re-spin a new RDS from snapshot, right?

has anyone seen this before ....

cloud-nuke defaults-aws
INFO[2021-04-01T13:40:37+01:00] Identifying enabled regions
ERRO[2021-04-01T13:40:37+01:00] session.AssumeRoleTokenProviderNotSetError AssumeRoleTokenProviderNotSetError: assume role with MFA enabled, but AssumeRoleTokenProvider session option not set.
<http://github.com/gruntwork-io/gruntwork-cli@v0.1.2/errors/errors.go:81|github.com/gruntwork-io/gruntwork-cli@v0.1.2/errors/errors.go:81> (0x16a1565)
runtime/panic.go:969 (0x1036699)
<http://github.com/aws/aws-sdk-go@v1.22.3/aws/session/session.go:318|github.com/aws/aws-sdk-go@v1.22.3/aws/session/session.go:318> (0x1974a25)
<http://github.com/gruntwork-io/cloud-nuke/aws/aws.go:50|github.com/gruntwork-io/cloud-nuke/aws/aws.go:50> (0x19749ca)
<http://github.com/gruntwork-io/cloud-nuke/aws/aws.go:66|github.com/gruntwork-io/cloud-nuke/aws/aws.go:66> (0x1974b36)
<http://github.com/gruntwork-io/cloud-nuke/aws/aws.go:86|github.com/gruntwork-io/cloud-nuke/aws/aws.go:86> (0x1974ce6)
<http://github.com/gruntwork-io/cloud-nuke/commands/cli.go:281|github.com/gruntwork-io/cloud-nuke/commands/cli.go:281> (0x199506c)
<http://github.com/gruntwork-io/gruntwork-cli@v0.1.2/errors/errors.go:93|github.com/gruntwork-io/gruntwork-cli@v0.1.2/errors/errors.go:93> (0x16a175e)
<http://github.com/urfave/cli@v1.20.0/app.go:490|github.com/urfave/cli@v1.20.0/app.go:490> (0x1691402)
<http://github.com/urfave/cli@v1.20.0/command.go:210|github.com/urfave/cli@v1.20.0/command.go:210> (0x169269b)
<http://github.com/urfave/cli@v1.20.0/app.go:255|github.com/urfave/cli@v1.20.0/app.go:255> (0x168f5e8)
<http://github.com/gruntwork-io/gruntwork-cli@v0.1.2/entrypoint/entrypoint.go:21|github.com/gruntwork-io/gruntwork-cli@v0.1.2/entrypoint/entrypoint.go:21> (0x1996478)
<http://github.com/gruntwork-io/cloud-nuke/main.go:13|github.com/gruntwork-io/cloud-nuke/main.go:13> (0x19966a7)
runtime/proc.go:204 (0x10395e9)
runtime/asm_amd64.s:1374 (0x106b901)
  error="AssumeRoleTokenProviderNotSetError: assume role with MFA enabled, but AssumeRoleTokenProvider session option not set."

Does anyone know of tools that can evaluate function code (e.g. lambda), identify API actions in the code, and compare those actions against a role or set of policy documents to determine whether all the permissions are accounted for?

awesome read https://aws.amazon.com/blogs/architecture/disaster-recovery-dr-architecture-on-aws-part-i-strategies-for-recovery-in-the-cloud/

We're looking at sending events to kinesis from our frontend app. The examples in AWS docs for this all tell you to use cognito for this, but it's not clear to me how/if that makes it any more secure or if it's just obfuscation? Any thoughts/experiences here?

I'm thinking about using a sidecar on my Fargate Service to proxy the database connection.

I'm thinking this helps address two issues:
1. Simplify configuration - the applications can always use localhost:<port> to connect (though of course connection details could still be set through an env far just in case) to the database
2. Security - nothing (besides code reviews) is stopping an application developer for printing the database connection details in their application. But, if the auth is associated to a sidecar that they don't have access to 🙌
I tried searching but I don't see people talking about this. So I'm thinking either this is a bad idea or it's just so obvious that I should done it sooner...

oh cool - autoscaling windows instances is finally worthwhile

does anybody have a lookup table (or script) for all the RDS instance types and their core-processor counts?  This is for calculating licenses.
an AWS cli option like aws rds describe-instance-types would have been great. (edited)

hmm idk. your best bet might be scraping https://aws.amazon.com/rds/instance-types/

Folks, I’ve just release a small tool to run AWS Access Analyzer Policy Validation against all your IAM Policies at account level. https://github.com/z0ph/aa-policy-validator Let me know if it helps!

Hi, I've removed a subnet from Beanstalk environment, rebuildid, instances are not being launched in it, but load balancer from that environment is still using network interfaces belonging to removed subnet. How can I fux this? Thanks?

Q: I'm trying to optimize RDS Patches (db minor engine upgrades). Is it true if I take a snapshot BEFORE the patch operation (say, 1 hour before), it would reduce the time it takes for the initial pre-patch snapshot???

Q: I'm trying to build a simple PoC with CDK Pipelines and is there really no way to use GitLab as a source? Am I missing something obvious here?

Question: API Gateway VPC link subnets can be public or private but if the endpoint is public should I use a public or private subnet? using both subnets I can reach the endpoint and using both I can use a test endpoint outside the vpc and the docs......well....the docs are not the best explaining this part

@PePe Amengual your message is very timely for me. for the past few weeks i’ve been working on an HTTP API Gateway that is intended to be private. I say HTTP specifically because most of the documentation I have seen centers on REST APIs.

That aside, I am using VPC Link to attach the API to a private, internal NLB. In this case, the NLB and the EC2 instances are in private subnets. Also, I’m using private subnet for my VPC Link.

As in your case, I can access my API from the internet.

However, I have a custom domain for the API endpoint which is a subdomain on a public hosted zone. So if you know the endpoint, you can resolve it. Even if you don’t have a custom domain, the API GW still has a default endpoint with something like <http://amazonaws.com|amazonaws.com> on it which is also public resolvable.

So i explored using VPC endpoints. This may be the solution for you if the API is intended to be used internally only by other AWS resources like EC2, Lambda, etc. But note that which ever subnet you create the endpoint in for API GW, all services in that subnet will use if they need to access API GW. It might be OK but it also might not be what you want. Also, VPC endpoints are great for keeping traffic inside your VPC. But in my case, I’m trying to expose a service for applications outside of my VPC but only on the private network (peered VPCs).

My next iteration on this is to try a custom domain in a Private Hosted Zone. The intent being the endpoint will not be resolvable outside of my VPC and any peered VPCs/networks.

If you come up with a solution that doesn’t involve a private hosted zone, I would be happy to hear it! 😅

anyone seen this ? the ssm is blowing up itself

in /var/log/amazon/ssm

572M download

Something i hardly ever need, but was confused today about some changes with Peer'd VPC's across regions. $.20 later i had my test and proof case on what i thought was wrong. Its a nice feature. https://www.reddit.com/r/aws/comments/mr6v4w/have_to_give_a_nod_to_reachability_analyzer/

Can I get upvotes on this? (just 👍️ on the PR comment to help prioritize it for review and merge.) Lol, it's not my PR, but I want the feature 😭
https://github.com/hashicorp/terraform-provider-aws/pull/18644

Terraform AWS Provider to include trusted_key_groups in cloudfront distributions 🙏

Hello guys, sorry to bother you, but I have one question regards to connection between two AWS accounts, I want to have connection from one ec2 instance on Account A to RDS on account B. I have set up AWS Private Link between them. I have created everything for working like endpoint service on Account B (service provider), set up some security groups etc. and I have setup on the Account A (as consumer) endpoint which make a connection request in order to access to Account B everything looks great because I can telnet to the specific MySQL port but I am getting

k╝Host '172.xxxx' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts'
on the beginning I thought that problem is with the specific mysql instance but when I started to spin up new instance on Account B in order to check if connection works directly, between instance and rds, it works. do you know what could be a problem?

aws-nuke disables console account password, any idea how to prevent it?

Hi, After you initially store objects in S3 Bucket. Lets say 20GB of data in March. Do you pay any additional in April if you don't retrieve any data from that bucket?
I cant see that is clearly stated anywhere, but I would say no...

Anyone has seen AWS Cloudsearch in use in production and know do/don’t around it vs AWS Elasticsearch?

Anyone use a fargate container for remote ssh with visual studio remote ssh plugin? Seems promising to offer a quicker remote dev environment than instances if all i want is containers anyway.

Hi channel.

AWS added support for using Docker images for Lambdas not so long ago, but unlike other uses of Docker (ECS, Batch), Lambdas can't access cross account ECR repos. This is a pain if you, like me, like to build your artifacts in a tool account and then pull them in the customer facing accounts.

If you have time, please consider upvoting this proposal to add the feature: https://github.com/aws/containers-roadmap/issues/1281 :)

Does anyone know of an ECS terraform module or a lambda function that handles events like deployment failure/task stopping and sends an SNS/Slack notification?

I have VPC -> IGW -> PublicSubnet(Application Load Balancer) -> PrivateSubnet(ECS Task)

The thing is I've got several components in the this various ECS tasks that want to talk to other Tasks in the private subnet.

Does anyone recommend public load balancer + internal load balancer combination, or another approach?

@sheldonh all you should need to do is update your security groups to allow communication in the private subnets. I usually do that by including the ECS SG as a source in the other resource.

Can you give more detail on this part: “several components in the this various ECS tasks that want to talk to other Tasks in the private subnet”?

Please, I need help urgently. I've ran ScoutSuite to scan my AWS account but it hit API rate limit. Can this somehow break my AWS account services and communication between them? Thanks

I have a single container that will need to do some pass through traffic in AWS. All my current architecture is ECS Fargate.
The communication will be one a 1000ish range of ports randomly chosen by the caller and passing traffic.
This container would need to take in traffic on this range of ports being managed by another service and pass it through to do it's magic.

I'm not sure if I can do that with containers. Is that possible to do with ECS Fargate or such?
If not, it seems I'll have to have a single EC2 server in the entire setup which I was hoping to avoid 😉

is it currently not possible to query S3 objects by their tags? I see mention of using the resource explorer for tags on the bucket level, but I don’t think I’m seeing anything on the object level.

Hi all! Just posted a new article on AWS cost reduction: https://corey.tech/aws-cost/

Another Corey working on AWS cost control

now there are two of them

Anyone know if an EC2 instance refresh is in anyway network aware? I need to refresh my ECS clusters EC2 instances to use new AMIs and don't want to cause an outage for any in progress users

Do custom eventbridge event buses not take aws.* (e.g. aws.ec2) events? I had a bug in my code where I was putting my event rule on the default event bus. When I changed the code and added it to my custom event bus, I stopped getting events

We just updated a EKS controllers from 1.15 to 1.16 in our staging cluster. This is the second cluster to do it but this one had a problem.
All of our NLB target groups except 1 became unhealthy (all targets/ec2 instances) and we ended up recreating the Loadbalancer Service's to restore service - with accompanying DNS change via External DNS.
1. nginx-ingress health check nodePorts were responding 200's from bastion, no change to security groups etc.
2. EC2 Route Analyzer from NLB ENI to EC2 Instance ENI where nginx pod was was green.
3. Nothing interesting in status when aws elbv2 describe-target-health ... .
4. Can see routine Target Group update CloudTrail event but all correct port and instances.
Have created a support ticket but a bit anxious about prod update.

Anyone have any ideas?

I suppose this is a rancher / aws crossover event.

I’m trying to get the Rancher Quickstart https://github.com/rancher/quickstart to work, but I’m having issues getting the SSL correct. I’d also like to use my own domain name (hosted on AWS) .

I need this done and I can pay for anyone’s time if they’re looking for a (hopefully) quick gig. ;-)

Has anyone tried AWS WAF bot control? Looks like it could be quite expensive, but curious as to how well it works.

hey, has anyone got good working examples of deploying and connecting to an efs from with eks, i have all the resources deployed, the pvc is connecting to the volume, however when the pod is trying to connect to the pvc i am gettign the following error:

 Warning  FailedMount  0s (x4 over 4s)  kubelet            MountVolume.MountDevice failed for volume "pv-efsdata" : rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /var/lib/kubelet/plugins/efs.csi.aws.com/csi.sock: connect: connection refused

In the case of using a bastion to connect to instances on a load-balanced Elastic Beanstalk environment, is there a common way to dynamically name instances so that I can easily login to various instances from the bastion?

# Example: easily connecting to instances from the bastion
ssh api-1
ssh api-2

Also, I would be interested to know how this works in load-balanced environments using the EB CLI , e.g., eb ssh .

What is this magic? https://aws.amazon.com/about-aws/whats-new/2021/04/ec2-enables-replacing-root-volumes-for-quick-restoration-and-troubleshooting/

Can someone explain ngnix reverse proxy like I'm 5? Not certain how this fits with load balancers and all .... Maybe more like I'm 18 since 5 year olds probably don't know the proxy. 😁

Looking for a friendly advise 🙂
Customer has 2 AWS Orgs
• First Org has Legacy Landing Zone Setup
• Second Org has the Shiny Control Tower Setup
I would like to make them one AWS CT Org
I was thinking if it would just be easier to move the accounts from the Legacy Landing Zone Org to the CT Org ? or If I should convert the Landing Zone Org to CT and then move the accounts from other CT Org to the "new" CT Org ?