#aws - February 2021

How do you configure auditing / access logging in your aws account? do you have a separate organization just for storing the audit logs across environments (dev,staging..)

Hi all, does anyone know of a way to compare a CloudWatch metric between two different dates? e.g. comparing RequestCount from an ALB so we can say “Oh we’re 10% busier compared to the previous Tuesday”. I know we can open two separate windows and tile them side by side, but would be nicer to have a more precise method. 🙏

Hey all! I'm trying to geo-replicate my RDS backups, but I'm not too sure which would be the best way to do that. I managed to automatically geo-replicate the RDS Snapshots taken every day by exporting them to S3 and then setting up S3 replication to another region, but it does not seem possible to restore a database from those exported snapshots since they are in the parquet format. Anyone implemented such a solution in here?

Has anyone hit the issue of “Note that you must create the Lambda function from the same account as the container registry in Amazon ECR.” when using lambda container images? https://docs.aws.amazon.com/lambda/latest/dg/images-create.html. This seems like a rough limitation when your ECR is in a separate account. Has anyone heard of any progress of this changing or a possible workaround?

The new AWS ECS UI appears to be missing service logs. Why does AWS keep on doing this with their UI improvements?

does anyone use aws-vault here? We’re using it, and it seems that actions that were performed by an aws-vault authenticated user, are logged with a session id instead of it’s username. Anyone familiar?

Any suggestions for debugging Route53 DNS latency issues? I have a DataDog TCP synthetics test for an NLB TCP service in a client’s production account. It tests that the TCP connection takes less than 500ms. This works 95% of the time, but every other day or so the client has the monitor alerting because the DNS lookup takes 450ms+ which combined with the latency from the NLB / Service triggers the alert. This seems a bit off out there to me and I’m wondering what tools I can use to debug that.

The root domain is delegated from another account:
1. The TTL of the NS record in the delegated account is 172800 seconds.
2. The TTL of the NS record in the primary account that points $delegated.$base_host to the delegated hosted zone NS record has a TTL of 60 seconds.
Any tools / thoughts / suggestions on this problem?

We just shared a self-case study of our own journey to securing our cloud environment. The conclusion of it can be summed up as:
• Shift left is far far better than just CSPM on live cloud environments. That is, if you’re actually interested in fixing things and not just visibility.
• The relationship between developers and security doesn’t have to be a standoff. With the right process, personnel approach and tools you can actually have your developers fix issues (and not chase them for months on a single ticket…).
Would love to hear other people’s experience.

Case study: https://indeni.com/blog/indeni-cloudrail-case-study-eating-dogfood-and-enjoying-it/

Hi all - I am running an app on Fargate and have logconfig as logDriver: Splunk to ingest logs to splunk. I did set the docker container log path which consists of multiple log files. but as of now its only ingesting docker default logs stdout or stderr to Splunk. Can someone help me on how to ingest all the log files in the logs directory to splunk ?

Thank you very much in advance.

How are you folks handling Lambda & ECS (Fargate) CI/CD alongside long lived infrastructure? We're trying to standardize on tools like aws sam, openfaas, serverless, etc... rather than rolling our own for handling Lambda / APIGW. I noticed AWS Copilot and ECS CLI but not sure what else is out there unless you go to EKS unless I'm misunderstanding the services. Pretty much just preparing for a lot of on-prem -> cloud migration.

Anyone has some experience with AWS Control Tower? would love to hear how is it in practice

Hi everyone! I have created an EKS cluster with the terraform_aws_eks module and the cluster was created with a particular access key and secret key. On a client machine, I cannot use that access key but have to use another set of accesskeys and then assume a role using the aws  sts command. After assuming the role, I have "admin access". When I then call kubectl get pods, I do not have access. I thought I could solve this by including this bit in the cluster creation:

map_roles = [
    {
      rolearn  = "arniam::844857508710:role/my-role"
      username = "my-role"
      groups   = ["system:masters"]
    }
  ]

where rolearn is the role that I assumed... but when executing kubectl get pods, I still have no access. Could someone point me to a solution 🙂 ?

anyone else having issues with their ECS/Fargate deployments today? I know AWS was forcing deployment updates today

ohhhh hey

that may explain why we saw a 1x daily task suddenly kick off mid afternoon

This could be handy, for generating minimal iam policies... https://github.com/iann0036/iamlive

Also this one! Generates IAM from your application code
https://github.com/iann0036/iamfast

has anyone figured out how to run aws-vault --server in the background here ?

been doing it in a detached screen session so far

Hi guys! any recommendation of VPN server? we want SSO with google, OpenVpn only accepts Gsuite with LDAP plan, which we dont want to pay. We want as well to have groups of users, and depending on the group to can allow the users go to one IP range tdestination or not.

Anyone get AWS Network Firewall to Prod yet? We're thinking about trying it out for East<->West traffic filtering. Wondering if anyone has an impression of how reliable it is during spikey/scale out traffic events. Also, since it's a newish service, if you find it pretty solid or still kinda buggy.

Anyone has experience with iotcore? I'm trying to understand why some arquitectures use kinesis firehose to send data to a s3 bucket.. Im able to the same with iotcore rules. Thanks

Hi everyone! I have a strange issue and wonder whether any of you have encountered it or managed to solve it.. I deploy an EKS cluster with fargate profiles using terraform, and this works perfectly the first time round. Then I issue a TF destroy and all resources are gone, so far so good.
Now, when again applying the TF scripts, with the same cluster name, the creation gets stuck on creating fargate profiles.. as if something is hindering AWS from recreating the same fargate profile names (which have been correctly deleted by TF):
module.eks.module.fargate.aws_eks_fargate_profile.this["default"]: Still creating... [44m50s elapsed]
Is this is a bug or is there a workaround for this? Often I can see that the Profile got created for the cluster, yet TF is somehow not "seeing" that the creation is complete...

Nice. If you attempt to create a CloudWatch dashboard with a hex colour specified with fewer than six hex characters (eg #ddd ), you get an error like

The dashboard body is invalid, there are 1 validation errors: [ { "dataPath": "/widgets/0/properties/metrics/0", "message": "Should NOT have more than 4 items" } ]

clear as mud.

Question on Kinesis: if data ordering is critical, can we only have one lambda consuming the data - or is there an elegant way of writing the data in the correct order to Dynamo for example..?

Hi, I have an issue - when I'm doing 'rolling deployment' - changing instance AMI in my ECS cluster, which consists of 1 instance. Old instance is set to draining, but the new instance doesn't run the task until the old instance dies. Any tips?

Has anyone here tried to create a nested cloudformation using the stackset resource? all works well until I try to retrieve the output values for the nested cloudformation that "lives" on the target account

I was following the docs and also found a blog explaining it. But at the end of the blog he says the following:

"outputs of the stacks created using StackSets are not easily accessible. If you want to reference something from the sub-template, the only way is to synthesize the resource names / ARNs. This is not always possible, e.g., B. with generated ARNs such as ACM certificates*

Does anyone know what he means? In my case is just an ec2
All help is appreciated it. thank you

it means you can predict the ARN in some cases if it depends on data you have provided. For example if you create an IAM Role with a hardcoded name, the ARN is predictable. But if you are looking at a resource name where the ARN is not predictable, you are SoL

However, it is possible to read the stack set directly, as if it were a normal stack. You just need access to the target account and knowledge of the stack set name. Then just inspect the stack's outputs as per normal. The passage above refers to the fact the outputs aren't consolidated up at the master stackset level automatically

anyone here have experience wiring up SES with IAM? Looks like it’s a bit different to everything else

edit: never mind, it’s normal, but there’s an extra thing that I was conflating with IAM

Hi, does anyone knows if it is possible to use a bastion server to access containers in AWS Fargate or the best approach to establish secure connections to Fargate containers? I have found some examples of people building SSH containers and exposing it through public IP however, I don't like the idea of having developers using a private key to ssh into those containers. I would appreciate any hint.

Guys how do I install AWS CLI v2 if I am using the base image that is derived from alpine linux? (inside Gitlab CI, but the CI tool doesn’t matter that much)

Hi anyone has experience achieving https://docs.aws.amazon.com/apigateway/latest/developerguide/getting-started-client-side-ssl-authentication.html but instead of the REST api with the HTTP api ?
https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api.html
thanks !

I'm using ALB - if I don't specify target groups, but set healthchecks to EC2 & ELB, then do ELB healthchecks work at all? Thanks!

Hi all i just going through this link :-
https://docs.aws.amazon.com/general/latest/gr/signing_aws_api_requests.html

my question is how to check what version of Signature am using? is there any clues in aws console ?

Putting this here since I’m using RDS (postgres) and there’s no database channel. I upgraded my production database from postgres 9.5.22 to 12.4 over the weekend as well as upgrading the instance type from r4.2xl to r5.2xl. In this upgrade, my read iops went from 12,000 -> 2,000 and I cannot figure out why. I understand that both the instance type upgrade and the postgres version upgrade introduce performance gains just by merely performing the upgrade, but I wasn’t expecting this big of an improvement. No changes application-wise and I haven’t seen any issues w/ our applications in regards to query data. Obviously I’d be happy with this much of an improvement if it’s expected, but I’m a bit weary as well.

thats a huge upgrade to do in one shot o.O

we went from 11 -> 12 in prod a week ago and all hell broke loose. I’d highly recommend running an ANALYZE on all your indexes, we had a lot of problems until we did that

Are you saying performance is the same, but read iops reduced by 80%+? Or your DB is 80%+ slower?

our performance looks roughly the same, the applications are functioning the same but my read iops have dropped by 80%

two variables that have changed (pg 9.5.22 -> 12.4 and r4 -> r5)

that's very cool. I was about to deploy some new clusters, will make sure devs use 12 instead of 11

are you using more memory? is more data being cached?

I went from r4.2xlarge -> r5.2xlarge so roughly 3gb more memory.

I am a bit worried by what @Zach mentioned, I am very weary that those 2 things could introduce that big of a performance boost. All the sanity checks I’ve done look fine though.

you could artificially limit the amount of memory postgres uses to see if that increases iops again 😄

Do any folks here install any open source or paid product for server hardening / security monitoring software on their EC2 instances? I have a client going through PCI / SOC2 and there have been requests from the auditing team to validate that we have security monitoring tooling installed on all servers. The client primarily runs applications on EKS Fargate so we hand-wave around this for the most part, but there are a couple EC2 instances in each environment account to support a bastion and system resources running on EKS Node Groups. All are using the base, up-to-date Amazon Linux 2 AMIs.

Looking for any recommendations around something simple to satisfy this requirement. Also, I’d be completely happy to hear that the community consensus is “Hey you should just hand-wave around that”.

How are everyone handling EC2 ASG Lifecycle Hooks with SQS or SNS where you want to run a script on the instance before termination?
Eg: ASG Lifecycle Hook -> SNS/SQS -> consumer in instance
- github.com/buildkite/lifecycled
- https://github.com/scopely/shudder
- other?

We were going to avoid SQS and cron AWS CLI calls but Spinnaker requires notification and role arn when it creates ASG's.
It creates a unique ASG per deploy (version number suffix) and destroys old ASG. Or some variance in multiple stages.

Hey, anyone using Eventbridge with ECS tasks here? I’m trying to figure out a way to pass the event details to ecs task but can’t find anything particular in the documentation.

https://docs.aws.amazon.com/eventbridge/latest/userguide/eventbridge-tutorial-ecs.html
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-events-rule-ecsparameters.html
https://docs.aws.amazon.com/eventbridge/latest/userguide/eventbridge-tutorial-ecs.html

Hi all,

am using AWS ses service, generated my SMTP credentials using the SES Console. sending email is working well am using boto3 to that, but after sometime i reviced a mail saying this....

Hello,
If you have already migrated your credentials from Signature Version 2 to Signature Version 4, you can ignore this communication.
We have observed Signature Version 2 requests (on an Amazon SES SMTP endpoint) originating from your account over the last week. Please note that Amazon Simple Email Service (SES) is working on an infrastructure upgrade with improved security controls. As a result, Signature Version 2 is being deprecated in favor of Signature Version 4 which offers enhanced security for authentication and authorization of Amazon SES customers by using a signing key instead of your secret access key........

now i generated the new keys and replaced the old keys but am still getting the error saying this...

The Canonical String for this request should have been
’POST
/

content-type:application/x-www-form-urlencoded; charset=utf-8
host:email.us-east-1.amazonaws.com
x-amz-date:20210219T081335Z

content-type;host;x-amz-date

The String-to-Sign should have been
’AWS4-HMAC-SHA256
20210219T081335Z
20210219/us-east-1/ses/aws4_request


any one having any idea ??? what extra steps i have to to

AWS SSO Q: Has anybody found a way to 'mass-sign-in' to multiple AWS SSO accounts? Our org has dozens of AWS accounts, and I wanted to login or have profiles for many of them. Right now, I have to pre-setup a profile for EACH account:
~/.aws/config

[profile account1.readonly]
sso_start_url = <https://mycompanysso.awsapps.com/start/#/>
sso_region = us-west-2
sso_account_id = 1111111111
sso_role_name = AWSReadOnly
region = us-east-1

[profile account2.readonly]
sso_start_url = <https://mycompanysso.awsapps.com/start/#/>
sso_region = us-west-2
sso_account_id = 2222222222
sso_role_name = AWSReadOnly
region = us-east-1
...

..then sign on to EVERY profile manually:

aws sso login --profile account1.readonly
(opens by browser, I have to enter the 8-character code)
aws sso login --profile account2.readonly
  # repeat for DOZENS of accounts!!!

There has to be a better way.

does anyone know how to obtain the correct account ID for the AWS CNI docker image when switching regions?

I'm just curious to see what others are using to manage/track the lifecycle of their AWS resources/assets? Any CMDB or other type of single pane of glass inventory solution?

How can I setup SSM so it doesn't start overriding the current prompt with new characters after certain number of characters?

Hi all, sorry for asking again,

I followed this code example (https://docs.aws.amazon.com/ses/latest/DeveloperGuide/examples-send-raw-using-sdk.html) to send SES sendRawEmail.

Which worked perfectly fine for the SMTP credentials which I created like 6 months ago,
But when I use the same code and replaced the newly created smtp credentials, I got the following error
.
.
The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.
The Canonical String for this request should have been
'POST
/
content-type:application/x-www-form-urlencoded; charset=utf-8
host:email.us-east-1.amazonaws.com...........................................

I believe old the credentials which I used was using SigV2 and with the new creds which uses SigV4 am getting the above error.
please let me know if anyone know how to solve this or any link or blog which shows how to use sigv4 creds to send email using python..

Thanks in advance (edited)

People in Multi-Account organizations (really, all of us at this point): how have you managed access to SSM Explorer and AWS Trusted Advisor reports for security/best-practice flagging?  We find that running an org-wide trusted advisor report requires us to be in the Management/master account, and that's extremely locked down.  Can you delegate this stuff down to a sub-account?

Does anyone have a recommended way of monitoring SAML cert expiry in AWS?

Can someone confirm that this will be possible?

    VPC A     <---peering--->     VPC B
(10.10.0.0/16)                (172.31.254.0/24)

Third party private CIDR: 10.115.0.0/24

VPC B routing and IPSec connection to third party private CIDR network
VPC A route traffic to this third party private CIDR thru VPC B

Hi All,

we have a requirement where i have to use an S3 bucket as a debian repository(client access server for the repository),

options we found is

1. Use s3 bucket as a static website hosting, but the problem with this option is, it can be only used as http

2. Other option is to use Cloud Front, which we can be used for https
our problem is with both options the s3 bucket will be public, we dont want everyone to access our repository.


we tried with api gateway mutual TLS(https://aws.amazon.com/blogs/compute/introducing-mutual-tls-authentication-for-amazon-api-gateway/) but it is not working for us,
also we followed this link (https://www.rapyder.com/blogs/static-website-authentication-using-lambda/#:~:text=Configuring%20Cloudfront%20for%20S3%20website,click%20on%20Create%20new%20Identity) here authentication is heppening only through browser, we need something like CLI auth.


Is there any way or any method to add authentication for the debian repository or s3 bucket. so that only authorized system can download the package.

Thanks in advance

Does anyone know if you can put a Cloudfront in front of Cognito yourself to raise the minimum tls to 1.2 ? AWS Support says to use an ALB but that doesn't really go with me.. Thoughts ?

Trying to run trusted advisor recommendations from CLI. is there a way to get trusted-advisor-checks against a single resource (ie. an RDS instance)? I seem to be able to pull all trust-advisor-recommendations for the account but theres no filter for resource-id (or ARN).

  aws support describe-trusted-advisor-checks  \
  --profile ${account_profile}  \
  --language en  \
  | jq -r  ".checks[] | \"$account_profile,\(.id),\(.name),\(.category),\(.metadata)\" "

## sample output:
account123456,nNauJivDiT,Amazon RDS Security Group Access Risk,security,["Region","RDS Security Group Name","Ingress Rule","Status","Reason"]

^^^ there's a check-id but nothing else tying the check to a resource.