#aws - November 2021

Hi everyone, I was wondering if I can categorize my logs in cloud watch based on a filter. I.e having somewhere to look at all logs that has a tag “failed-error”. If there isn't a way to do this with cloud watch kindly suggest any third-party solution you know. Thanks

Great technical blog by Jonathan Rau, our CISO, about securing your AWS EC2 Instances with Microsoft Defender

https://blog.lightspin.io/microsoft-defender-for-endpoint-on-aws

Hi Everyone! I'm using a bastion host on AWS and am trying to add a new user. The user generates a new RSA key pair. I add the public key to the authorized_keys file on the host. But they still get Permission denied (publickey,gssapi-keyex,gssapi-with-mic). I've tried myself and it still fails. Its not a file/directory permissions issue because I can currently connect with the key/pair I generated a couple of months ago. How can I troubleshoot this further?

Hello,

Sadly I'm working in an environment where most of our resources are not controlled with IaC; or else this would be relatively simple. I'm needing to keep a subset of tags (~5) from an EC2 instance in sync with its associated volumes and snapshots. An easy example of what I would be looking for is if a billing tag changed in 6 months, I would need to replicate out to its volumes and snapshots.

We already have solution with some custom coded automation, but wanted to hear if there are any projects or AWS tools that would provide this type of functionality.

Does anyone have experience adding splunk universal forwarder to elastic beanstalk? Running into an "issue" that I'm stumped on and not sure where to ask anymore

Any ideas how to turn off aws cloudwatch alarms at specific hours?

If I will use math expression on aws cloudwatch alarms, does it mean that I will increase number of alarm querying per minute by doing that?

Has anyone ever had an issue where you can access an object in a public bucket, but when an ecs container tries to access that same object its access denied?

From my Twitter, a thing that may be of interest to y’all: “As of Late 2021, which AWS service should I use to run my new containerized app in production?”

is it possible that If I have cert arn with a.com and *.a.com cert and I would like to use it with nlb then nlb will use just the 1st cert (a.com) ?

awesome GraphQL for AWS https://github.com/cloudgraphdev/cli

General PSA when working with Fargate (via AWS Support):

Below is the summary of our conversation:

You had contacted us since you wanted to know whether STOPSIGNAL or SIGTERM sent to the running container is sent when you stop a task. While on chat, I informed you that unfortunately this feature is not available through ECS yet. The STOPSIGNAL is not supported by ECS and there is no way to specify any other custom signal for StopTask.

We are aware of this limitation and there is currently a feature request to have this available with ECS. However, there is no ETA provided as this requires extensive testing and several approvals before making any changes into production environment. You can keep an eye on the GitHub issue on the containers-roadmap repository [1] for future updates regarding this [2] - https://github.com/aws/containers-roadmap/issues/359

Later, you asked on how common is it for AWS to not have spare capacity and not replace a terminated spot task. (SERVICE_TASK_PLACEMENT_FAILURE event). I informed you that this is a known issue and there is a Feature Request open mentioning the use of “on demand” instances when spot is not available. https://github.com/aws/containers-roadmap/issues/773

Hi everyone! is there someone here who partecipate to the AWS reInvent?

It could be a great idea to have the opportunity to meet each other! 🙂

Personally I’m happy to partecipate, and also, I will present with @Eric Villa Leapp at the Open-Source lounge on Monday at 5 PM 😄

Hope to see you all 🙂

Hi all, I'm using ECS's Docker Compose integration and everything is working well EXCEPT the output from the docker compose command is so verbose and noisy that even CircleCI wants me to write it to a file and inspect it after because it goes over the 100MB limit that there log output supports.

I've also noted that docker compose --loglevel doesn't work? Has anyone else experienced this?

I guess my general question isn't entirely AWS related but I do wonder how other infra/ops people quiet logs in this sort of situation? Are y'all just piping to grep or awk/sed?

It specifically the part where it's creating resources in ECS, so the LogGroup, ECS Service, CloudMap etc. takes some time and each update writes another 20-30 lines to the screen - I don't want to lose the ability to track progress of my CI pipe, but I also can't find any flags to shut up ECS/Docker Compose

Hi all, this is an EKS related question, can someone tell me how to trigger an instance refresh? is it EKS, AMI version or something on the ASG has to change to trigger it? we are looking for the minimum impact so i guess eks update is out of the question, thanks!

Hi all, which one would you use between Cloudcraft vs. Lucidchart to have automatically updated charts as we edit the infra via console or terraform? thanks!

A bit of a networking question, but would be glad if someone could give their take on it. It seems that AWS recently has a way to use NLB with ALB so that you can take advantage of things like SSL termination on the ALB while still using the NLB to do non HTTP traffic (https://aws.amazon.com/blogs/networking-and-content-delivery/application-load-balancer-type-target-group-for-network-load-balancer/). If I didn’t have any non HTTP needs, is there a purpose to use an NLB? The only reason we used an NLB is that our company wanted to use API Gateway and wanted to call an ECS Service behind an ALB, but API Gateway only allows NLBs to use AWS Private Link to create a connection. Otherwise, the ALB has to be public if we want to use API Gateway to reroute it but it ruins the point, since the ALB is still exposed to the public.

Anyone know if there's a way to set up bucket replication on the s3 bucket created by the AWS-cloudfront-s3-cdn module? A bit confused over how I can ensure the failover bucket stays up to date.

I'm evaluating migrating my CI pipelines away from CodePipeline and CodeBuild. Any recommendations? Are there hosted CI that is cheaper or comparable to CodePipeline?

Hi there,
Does anyone know how to set AWS Managed Apache Kafka data retention to forever?

Hi everyone,

Hoping to get some advice from some of you. I’m currently migrating our infrastructure off of Heroku into AWS. Our dev shop has one monolith and a handful of micro-services that are build with Django. The idea here is to move from a monolith to micro-services. Each application also has a celery worker and beat. I’m planning on migrating over to AWS using Fargate and initially I will be supporting the celery worker and beat with a spot instance. Has anyone had experience doing a similar migration? The two pieces that i’m trying to iron out is how we will be implementing continuous deployment and how service to service communication will work for applications that will only interact within the VPC. From what I’m reading. Codedeploy is used to help with the blue/green deployments via traffic shifting from one image to the other and AppMesh is a service mesh that will enable me to do do service to service communication. Does Codedeploy work with AppMesh?

hello all, We are facing a really strange issue. We have k8s cluster and have spot fleet for nodes. we have a java app that can start on intel all time and cannot start few times on amd epyc... Any idea ?

aurora serverless? I read it somewhere the volume cluster behind this service is way slower than gp2... is that right ? If yes, then can we somehow move to gp2 or just w/o serverless?

Anyone using Cloud Custodian? As we’re moving to more accounts, we’re finding it desirable to put all the policy lambdas in one account. However, we then run into an EventBridge rule limit which we’re requesting be raised. It also seems that we could change the way the policies are packaged to not require so many lambdas. Seeing if anyone has run into this, or suggests an alternative to Cloud Custodian such as AWS Config.

Marc uses it heavily
[e] oh lol I was in wrong slack. Marc isn’t in this one

cloud custodian uses aws config for inventory information. I've never run into the event bridge limitation tho. have you checked out the cloud custodian gitter?

kapilt and his posse are super helpful and if it's a limitation they will know immediately

does anybody has good config for aurora mysql serverless? Or the default innodb and mysql conf is fine ?

Interesting read on AWS Access keys - https://www.hunters.ai/blog/hunters-research-is-aws-recycling-your-access-keys

Hi colleagues, we are using aws iot service and we have been wondering whitch is the best place to store securely online some certificates without the need to maintain servers (like hashicorp vault etc.) Can we do it securely with some s3 buckets for example?
thanks!

hey people, I was asked if I could create regular AWS EKS control plane and make local, on premises bare-metals join that EKS -- any thoughts, experience, ideas ?

(last time I did EKS it was ~1.15, I've just used terraform with eks module, run terraform apply - it created control plane, worker group within my vpc and that was it)

Hi all, is it possible to have gui access to an ec2 linux server, but use only aws authentication? something like using the web terminal of the SSM, but doing it with gui instead?
Thanks!

perhaps you can play with pam module, because gui or cli it's the same

In my view, it’s still early as heck for Proton, but they added Terraform support as a preview now: https://aws.amazon.com/about-aws/whats-new/2021/11/aws-proton-terraform-infrastructure/

TL;DR: Proton wants to be a pretty UI over Terraform (modules). It will help answer questions like “what’s the latest stable version of module X?” and allow non-tech users to use Terraform and Terraform modules by filling variables in a nice UI. Kind of like Service Broker, or a service platform in which you fill in details about what you want and next, next, next, finish (and then Proton commits it to your IaC repo maybe?)

^^^ I could be wrong, this is just my current understanding

So there is:
• CloudFormation (AWS)
• Terraform (Hashicorp)
• CloudFormation CDK (AWS)
• Terraform CDK (Hashicorp)
• Cloud Control (AWS)
And now Proton is thrown in the mix.

It seems like AWS thinks neither CF or TF as-is are ideal, and is throwing ideas at the wall to see what sticks. So we devs are going to be stuck in a churning market for the next few years, either continuing to use CF/TF and miss out on innovation, or risk betting on a losing technology

I don't know about anyone else, but I never adopt anything new from AWS for at least a year if I can avoid it. Their initial GA releases seem too rough around the edges to be worth touching until they've been proven and had some time to mature.

Eh, not really. I should do a flowchart for this too 😅

IaC has options:
• Declarative
◦ CloudFormation
◦ Terraform
◦ Bonus declarative using k8s’ continuous reconciliation loop: ACK, Crossplane
• Imperative
◦ CDK which “compiles” to CloudFormation
◦ CDKTF which “compiles” to Terraform
◦ Pulumi which does direct changes (I think/kinda?)
• Tooling
◦ Managed applies and things around that: Atlantis, Terraform Cloud, Spacelift, Env0, Scalr
◦ Nice UIs with “Installation wizard” instead of editing HCL/YAML/Code: Proton
They all solve different things. We don’t have one programing language or one IDE, so having one way to do infra as code won’t be a thing. And options are good!

I mean, cloudformation was a hackathon project that was built in 24 hours to help provision internal aws services. It wasnt originally designed for general availability

Reading through the way Proton "compiles" things, I'd liken it to something like Terragrunt + TFC

Hmmm, it relies on CodeStar for repository connections, which somehow doesn't support CodeCommimt? Nor GitLab? But it does support BitBucket?

Blog post with more info on the new features and terraform support... https://aws.amazon.com/blogs/aws/new-aws-proton-supports-terraform-and-git-repositories-to-manage-templates/

Ahh, not like TFC then...

... AWS Proton is not the one managing the provision of infrastructure. Therefore it is important that in the process of provisioning the infrastructure, there is a step that notifies AWS Proton of the status of the deployment.

in case folks missed https://aws.amazon.com/blogs/aws/aws-free-tier-data-transfer-expansion-100-gb-from-regions-and-1-tb-from-amazon-cloudfront-per-month/ ( after some behind the doors info been mentioned https://blog.cloudflare.com/aws-egregious-egress/. Is good to se this sort of actions from folks like AWS and co.

hell all. we are using m5 and m5a instances (intel and amd cpu). We see significant performance differences between them. is that possible ? Our java code times out on AMD cpu and we see strange issues in our app if we run on m5a types...

hi folks, i'm trying to find a sweet spot to improve the experience of developing / test lambda or ecs containers + rds (in private subnets, of course 😉 ) using least privileged access.

Context

Have a bunch of various solutions using lambda deployed inside VPC to interact with RDS deployed on private subnet. Equally same apply when having a container running as part of ECS talking with RDS.

Challenges

• developing/ debugging the lambda locally while setting breakpoints connected to RDS is very painful as you need to first overcome the network access: using a Bastion and a ssh tunnel over it is okay-ish although rough ...
• because the initial RDS and the additional services were deployed using TF and the least privileges, running the code locally requires extensive effort to go over the IAM
Tried out and ... failed 😨

• thought of moving the local dev env to Cloud9 where i could associated an existing/ working IAM locked policy as extended instance profile but ... Issue #1 : by default Cloud9 gets deployed in public subnet. Issue #2: created an env using private subnet and SSM but sadly i couldn't modify the associated instance profile from Console

Has anyone faced similar situations and if so able to share their stories ?

Hey, any good info on multi region active-active Architectures? I'd like to route users to their assigned aws region based on authentication cookie. e.g if user has eu cookie they are redirected to us-west-1 alb

This’ll make a bunch of people very happy: https://aws.amazon.com/blogs/aws/announcing-pull-through-cache-repositories-for-amazon-elastic-container-registry/

Anyone else at reinvent?

Hi guys, is there a better way to track request count with aws application load balancer? I have connected cloudwatch metrics to aws managed grafana to display daily requests on the dashboard but the data provided isn't useful. We needed something to give us a figure of daily traffic. Any other recommendation to achieve this is appreciated too.

Thanks

anyone @ aws re:Invent? 🙂 would love to meet community people