#aws - January 2020 | Slack Archive

aws

136 messages

Discussion related to Amazon Web Services (AWS)
Archive: https://archive.sweetops.com/aws/

Dec 2019

January 2020

Feb 2020

rohitover 6 years ago

Does anyone know how to make an internal ALB with private hosted zone use DNS cert validation ?

Eamon Keaneover 6 years ago(edited)

When using nginx-ingress (installed via helmfile provider), is there a good way to get the load balancer CNAME into terraform for use with populating dns (e.g.

<http://a92da7dd42d9c11ea9d19028a775bee0-865849920.us-east-2.elb.amazonaws.com|a92da7dd42d9c11ea9d19028a775bee0-865849920.us-east-2.elb.amazonaws.com>

)

Erik Osterman (Cloud Posse)over 6 years ago

Do not use terraform for that. Instead use the external-dns controller.

Erik Osterman (Cloud Posse)over 6 years ago

This is where the power of Kubernetes controllers really shines

Erik Osterman (Cloud Posse)over 6 years ago

E.g. use the cert-manager controller to automatically generate certificates

Eamon Keaneover 6 years ago(edited)

oh yea good point, I was getting confused (I do use external-dns and cert-manager). What I actually was driving at was gcp lets you assign a static ip to a load-balancer which can be passed in as a variable to helmfile and remain constant if for some reason the cluster had to be recreated (useful if passing this ip to a third party domain you don't control). On AWS, I don't think there's an ability to control the CNAME so it will change between cluster creations, is that right?

Andriy Knysh (Cloud Posse)over 6 years ago

On AWS to get a static IP and attach it to load balancer, you can use https://aws.amazon.com/global-accelerator/

Eamon Keaneover 6 years ago

thanks @Andriy Knysh (Cloud Posse) I'll give that a shot.

Eamon Keaneover 6 years ago(edited)

hmm, so global accelerator gives multiple static ip addresses, but looks like nginx-ingress is only set up for one load-balancer ip address.... though I guess just choosing one might work with some loss of HA.

https://github.com/helm/charts/blob/master/stable/nginx-ingress/values.yaml#L243

rohitover 6 years ago(edited)

Does anyone here use RDS IAM authentication feature ?

rohitover 6 years ago

I am trying to figure out if we would still have to generate token and use it in place of password when RDS IAM authentication is enabled

rohitover 6 years ago

If my app is running on ec2, can i just attach iam policy and can get rid of using password as a whole ?

rohitover 6 years ago

The documentation also says that the “The authentication token has a limited lifetime of 15 mins”, does that mean once the token is generated we have to use it with in 15 mins to make a connection

PePe Amengualover 6 years ago

yes you app will have to use the mariadb driver to deal with the token expiry

PePe Amengualover 6 years ago

so it will need to request a token every 15 min

PePe Amengualover 6 years ago

the doc detail how to do it with some code examples

PePe Amengualover 6 years ago

and once you are using mariadb driver you can just delete all the other user not using the driver

PePe Amengualover 6 years ago

is a specific mysql driver that get installed in the mysql aurora server

PePe Amengualover 6 years ago

and you create the user in a specific way

Brij Sover 6 years ago

I have an ACM question, are the following certs the same? (I'm trying to understand what 'additional names' are for)
cert a:

domain name: <http://test.domain.com|test.domain.com>
additional name: *.<http://test.domain.com|test.domain.com>

cert b:

domain name: *.<http://test.domain.com|test.domain.com>
additional name: -

Andriy Knysh (Cloud Posse)over 6 years ago

cert b will not cover <http://test.domain.com|test.domain.com>

Andriy Knysh (Cloud Posse)over 6 years ago

just all subdomains

Brij Sover 6 years ago

so for example, cert b will not cover test.domain.com but aaa.test.domain.com ?

Andriy Knysh (Cloud Posse)over 6 years ago

Yes

Brij Sover 6 years ago

so its more ideal to go with cert a?

Brij Sover 6 years ago

essentially cert a will cover <http://test.domain.com|test.domain.com> and all subdomains?

Andriy Knysh (Cloud Posse)over 6 years ago

yes, create it exactly as cert a

Andriy Knysh (Cloud Posse)over 6 years ago

since you use star for subdomains (all of them) and if you use DNS validation, ACM will generate two exactly similar records to put into DNS zone

Andriy Knysh (Cloud Posse)over 6 years ago

and since the records are the same, you can add just one of them and all will work

Andriy Knysh (Cloud Posse)over 6 years ago

but, if you request a cert for test.domain.com and just for www.test.domain.com, then ACM will generate two different records, and you will need to put both of them into DNS

Eamon Keaneover 6 years ago(edited)

has anyone used Iam Roles for Service-Accounts with EKS and a cluster of just Managed Node Groups with the cluster-autoscaler (v1.14.7)?

The role has the required permissions and it is getting injected into autoscaler pod.

Erik Osterman (Cloud Posse)over 6 years ago

PSA: https://aws.amazon.com/blogs/aws/urgent-important-rotate-your-amazon-rds-aurora-and-documentdb-certificates/

Nick Vover 6 years ago(edited)

Anyone know how I can skip text when using parse on Cloudwatch Insights? I have

parse message '* src="*" dst="*" msg="*" note="*" user="*" devID="*" cat="*"*' as _, src, dst, msg, note, user, devID, cat, other

but I'd like to discard everything before src=

Igorover 6 years ago

I don't see the Opt-In option for new ARN format in ECS

Igorover 6 years ago

Has anyone come across that before?

Igorover 6 years ago

I am getting "The new ARN and resource ID format must be enabled to add tags to the service. Opt in to the new format and try again." from TF, but not sure how to resolve

Bernhard Lenzover 6 years ago

I'm getting below error. Is this the right forum to ask for help?

terraform init
Initializing modules...
Downloading cloudposse/ecs-container-definition/aws 0.21.0 for ecs-container-definition...

Error: Failed to download module

Could not download module "ecs-container-definition" (<http://ecs.tf:106|ecs.tf:106>) source code
from
"<https://api.github.com/repos/cloudposse/terraform-aws-ecs-container-definition/tarball/0.21.0//*?archive=tar.gz>":
Error opening a gzip reader for

Erik Osterman (Cloud Posse)over 6 years ago

@Bernhard Lenz better to use #terraform

Bernhard Lenzover 6 years ago

Nick Vover 6 years ago

Anyone have experience with how much overhead RDS postgres multi-az adds? I was doing some testing on tiny instances and the added replication seemed to add a decent amount (20-30% CPU) but that was a tiny t3.small instance

joshmyersover 6 years ago

don’t use tiny instances.

Erik Osterman (Cloud Posse)over 6 years ago

I don’t have an answer, but it seems a common complaint, explained in more detail: https://stackoverflow.com/questions/47162231/rds-multi-az-bottlenecking-write-performance/50441734#50441734

Erik Osterman (Cloud Posse)over 6 years ago

I suppose the amount of overhead would depend on many many factors: amount of data written, size of data written, randomness, io characteristics of the instance. The t3 class will probably be worst to use as a reference instance since they are burstable.

Andriy Knysh (Cloud Posse)over 6 years ago

multi-AZ will always have (some) performance impact since Aurora does replication synchronously

Davidover 6 years ago

Say I upload to files to an S3 bucket for a website multiple times per day: index.html and some_file.some_cache_busting_string.js, and that the new version of index.html that is uploaded will always reference the most recent js file.

index.html is a much smaller file than the js file, so when I upload these files the html file will typically complete its upload first.

If someone visits my site during the time after the HTML is uploaded, but before the js finishes uploading, the page will fail to load.

I'm guessing I'm not the first person to experience this, so how do you all handle this?

Andriy Knysh (Cloud Posse)over 6 years ago

sounds like a good puzzle to solve 🙂 could be done by uploading files to S3 sequentially or using this in index.html (in onerror, try to load the old file) https://javascript.info/onload-onerror

kskewesover 6 years ago

Hey team, am running a Hugo static site with the cloudposse CDN module with great results.
Site is converted from a lamp stack minus the deprecated CMS.
Looking to replace the phpmailer email form and came across this module. https://github.com/cloudposse/terraform-aws-ses-lambda-forwarder
Wonder if I could stitch this and AWS API Gateway together to handle the form POST per a few blogs?
Preferably in terraform but haven't used these services before.
Will need to find a way to avoid spam usage.

Sergeyover 6 years ago(edited)

Hi guys!
I faced issue with ALB' hard limit that is 1000 Targets per ALB. I tried to get targets that are belong to ALB using aws cli, but I did not found such option there. Did someone face this issue or make work around for it?

Maxim Tishchenkoover 6 years ago(edited)

Hi guys, I have a question about EKS:
You pay 0,20 USD per hour for each Amazon EKS cluster that you create (from https://aws.amazon.com/eks/pricing/)
Is it apply for any type of eks confirmation (ec2, fargate ) ?
Do you know why is it so expensive ? it is almost 144$/month.
is it price for k8s service containers ?

Igor Rodionovover 6 years ago

Hi guys
Have anyone experience with EMR auto scaling related to Presto app?
We have errors in Presto when downscaling EMR, it looks like EMR does not support graceful shutdown for Presto because it is not managed by YARN.
Is there any workaround?
Are we doing something wrong?

Michael Coffeyover 6 years ago

Does anyone know where I can find an example of using terraform to add a trigger to a lambda function? I found aws_lambda_event_source_mapping but it seems to only support event streaming from Kinesis, DynamoDB, and SQS. I want to add a trigger for changes in an S3 object.

Joe Hostenyover 6 years ago

Has anyone tried using the reference architecture with a .ai domain? You cannot register .ai domains in AWS. I am wondering if this would work if, after provisioning, the SOA record in the apex domain's hosted zone were deleted, and name servers at the registrar pointed to the name servers in that same zone?

Erik Osterman (Cloud Posse)over 6 years ago(edited)

@Joe Hosteny haven't tried... but don't see why not? ref arch does not register TLD. it just creates the zones.

Erik Osterman (Cloud Posse)over 6 years ago

so if you can create the zones, then it's fine.

Erik Osterman (Cloud Posse)over 6 years ago

after provisioning, the SOA record in the apex domain's hosted zone were deleted, and name servers at the registrar pointed to the name servers in that same zone?

Erik Osterman (Cloud Posse)over 6 years ago

please share context (also lets use #geodesic)

Erik Osterman (Cloud Posse)over 6 years ago

office hours starting in 10m: https://cloudposse.com/office-hours

Saichovskyover 6 years ago

Hey peeps

Saichovskyover 6 years ago

I am trying to add a config rule to Landing Zone so it can be created across accounts

Saichovskyover 6 years ago

There’s very few (if any) tutorials on LZ - all I see are videos and articles on why LZ is good and what it does and its advantages, but barely any meaningful examples

Saichovskyover 6 years ago

so my CodePipieline pipelines are passing after adding the custom config rule to some template

Saichovskyover 6 years ago

I have also added the lambda that should get invoked (both source code and definition in the template) as well as the permissions required to execute the lambda

Saichovskyover 6 years ago

After the pipeline executes successfully (which takes forever to complete), the resources are still not created and at this point, I am not sure what it is that I need to understand between cloudformation and LZ

Saichovskyover 6 years ago

It’s hard to troubleshoot when everything is passing

Saichovskyover 6 years ago

Any pointers to some resource which can help me understand LZ like a 5 year old?

Michael Warkentinover 6 years ago

Nice little extension from a coworker: https://addons.mozilla.org/en-US/firefox/addon/amazon-web-search/

Lets you create a shortcut to open up the AWS console service list / search bar - and most importantly lets you hit esc to close it

Mike Croweover 6 years ago

@Mike Crowe has joined the channel

erikover 6 years ago

Erik Osterman (Cloud Posse)over 6 years ago

@Mike Crowe sound familiar? ☝️

Mike Croweover 6 years ago

Ha ha btdt

Erik Osterman (Cloud Posse)over 6 years ago

AWS NAT gateways are stupidly expensive for what they do

Mike Croweover 6 years ago

And not documented in control tower

Chris Fowlesover 6 years ago

i have to think that they're just spinning up multiple ec2 instances to do NAT or something like that - for the price that you pay it would have to be something supper inefficient

mfridhover 6 years ago

Was thinking about this the other day... at least modifying my terraform so that the "spoke" VPCs just get a solo NAT gateway instead of 3x... but then I guess you start paying cross-AZ for 2 thirds of egress traffic anyway so I dunno 🤷

lorenover 6 years ago

can also use transit gateway to route egress traffic through one vpc, and just pay for one set of nat gateways

Abel Luckover 6 years ago

hey folks, a client of mine is using the AWS SSO to manage all access creds. I'm having some trouble finding a nice solution for using this with terraform.

Abel Luckover 6 years ago

Currently I am logging into the aws sso page, choosing the aws account, clicking "Command line or programmatic access" and copy pasting the provided env vars into my shell.

Abel Luckover 6 years ago

I have to do this every hour 😐️

Abel Luckover 6 years ago

Both aws-vault and terraform have tickets open about AWS SSO, but has anyone come up with a decent workaround?

Dzhuneytover 6 years ago(edited)

AWS recommends that we give access/secret keys only to physical users. Machines or systems should instead assume IAM roles (e.g. an EC2 instance should never use access keys passed as environment variables).

However, how does this best practice work in real life when the system that communicates with AWS is a third party one (e.g. GitHub actions). Can I make a GitHub Actions CI pipeline communicate with AWS APIs without creating an IAM user for it? After all, GitHub is a machine, not a human. It should use IAM roles, right?

aaratnover 6 years ago

AWS means to do that in AWS eco-system. If you have a custom github runner in aws you can use IAM role

Dzhuneytover 6 years ago

So if I use a real third party solution like Jenkins, GitHub actions, CircleCI - I have no other option but to resort to IAM users and secret keys per provider?

aaratnover 6 years ago

Yes

vFondevillaover 6 years ago

Yup

lorenover 6 years ago

did you try aws-cli v2, and its new integration with aws sso? login with that to get the creds for the profile, then run terraform? https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-sso.html

Joe Hostenyover 6 years ago

Depending on your integration, one thing you can also consider is having some privileged task inside your infra deliver STS credentials to the third party service.

Joe Hostenyover 6 years ago(edited)

That means your infra code has to have creds to the service, but that may be more palatable

lorenover 6 years ago

may be able to use the credential_process feature of the aws shared config file to retrieve temporary creds, if the integration is running in something that gives you that kind of access, https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-sourcing-external.html

rohitover 6 years ago

what is the best way to implement API authentication for REST API built in nodejs (not AWS Gateway) ?

rohitover 6 years ago

any ideas and thoughts are appreciated

Maciek Strömichover 6 years ago(edited)

https://awsapichanges.info/ - not sure if anyone else posted it earlier

Maciek Strömichover 6 years ago

https://aws.amazon.com/blogs/containers/results-of-the-2019-aws-container-security-survey/

Chris Fowlesover 6 years ago

am i nuts or is cloudwatch synthetics really really expensive

Chris Fowlesover 6 years ago(edited)

i'm calculating $73.44‬/month for a single per 1 minute check

Chris Fowlesover 6 years ago

@ $0.0017 per canary run

Davidover 6 years ago

Just found an SES rule that was sending to an s3 bucket that used to exist, but no longer does. Is there any way to retrieve the emails it missed in the meantime?

Igorover 6 years ago

Has anybody run to issues with Internal DNS not resolving correctly for ECS Service Discovery? I can see the correct IP in Route53 with TTL of 60, but it's not being picked up

Erik Osterman (Cloud Posse)over 6 years ago

You application doesn’t pick it up?

Erik Osterman (Cloud Posse)over 6 years ago(edited)

What is your zones SOA TTL?

Igorover 6 years ago

My application doesn't, but I also tried pinging it elsewhere, and it's also giving me the wrong ip

Igorover 6 years ago

SOA TTL is 900

Erik Osterman (Cloud Posse)over 6 years ago

So I would cut SOA to 30 just in general

Erik Osterman (Cloud Posse)over 6 years ago

Because that acts like a negative cache

Erik Osterman (Cloud Posse)over 6 years ago

So if the record is queried before it exists and the response is not found, that negative response will be cached for for SOA TTL

Erik Osterman (Cloud Posse)over 6 years ago

But what you are describing sounds like it could be something else.

Igorover 6 years ago

AWS Cloud Map service is magically supposed to manage these mappings

Erik Osterman (Cloud Posse)over 6 years ago

Since you are getting an IP back but the wrong one, it doesn’t have to do with the SOA

Igorover 6 years ago

First time I ran into an issue with it

Erik Osterman (Cloud Posse)over 6 years ago

I don’t have experience with that service so someone else here might know more like @Maarten van der Hoef

Igorover 6 years ago

Bah, just noticed Route53 operational issue alert in the AWS console

Igorover 6 years ago

Been going on for over 2 hours too. SMH. Please tell me this almost never happens.

caretak3rover 6 years ago

#aws anyone have experience dealing with aws state machines? specifically, a missing state machine after a cloudformation (landing-zone) update, and my avm (vending machine) plays fail.

Michelover 6 years ago

hello, anyone using private aws gateway api with custom host header? Thanks

Maciek Strömichover 6 years ago

https://aws.amazon.com/about-aws/whats-new/2020/01/amazon-eks-announces-price-reduction/

Blaise Pabonover 6 years ago

Since VPCs are limited in my environment, I naively thought I could just kops cluster create a new cluster inside my existing VPC. (with this snippet).
Then it blew up because of subnet conflicts and I could see where this was going.
Most of the TF vpc modules I see expect to create a new one....
Should I:
• terraform import the VPC resource,
• manually replace vpc.vpc_id with something in var.vpc_id
• stop overthinking this and use some kops flag I don't yet know about
?

grvover 6 years ago(edited)

Usual approach I follow with KOPS is to create VPC and other aws reosurces using terraform and then use kops to create the actual cluster (just like you are doing). I however split the subnets cidr's accordingly when creating the VPC from tf. In your case, I guess if you are planning to use the vpc, terraform import might help. Not sure if this helps you though 🙂

Alejandro Riveraover 6 years ago

Hi, has anyone deployed EKS on me-south-1 and successfully interacted with the cluster via kubectl getting credentials using aws sts assume-role ?

joshmyersover 6 years ago

Yes, same as other regions…

GFox)(AWSDevSecOpsover 6 years ago

Looking for simple condition to set in EC2 instance template with ALB, that will hold the 2nd (HA instance being deployed) so I can get the first up with some manual config running then remove condition to lauch the instanceB

Dan Gover 6 years ago

Hello, I've used the AWS landing zone pattern for a while but just wondering how others use the shared account for things like Vault? I.e put Vault in here or separate account? Also usage pattern - 1 Vault for dev/prod or 2 separate installations...Just wondering what others have done...cheers

joshmyersover 6 years ago

I’ve always gone with a Vault cluster per environment

joshmyersover 6 years ago

Deployed into the same account unless it is a break-glass type scenario

Dan Gover 6 years ago

That's an option, I was thinking we will have multiple prod accounts so a shared prod Vault cluster would be another way

Saichovskyover 6 years ago

Hey people

Saichovskyover 6 years ago

I have a question for AWS gurus

Saichovskyover 6 years ago

How do I use the PutOrganizationConfigRule API in Landing Zone to create an organization config rule?

https://docs.aws.amazon.com/config/latest/APIReference/API_PutOrganizationConfigRule.html

Igorover 6 years ago

Going to try this channel as well. Does anyone know if ECS provides an SNS topic to subscribe to events like updating service, tasks starting/stopping, autoscaling events?

joshmyersover 6 years ago

I don’t believe so

joshmyersover 6 years ago

Does https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_cwet.html get you what you need?

Igorover 6 years ago

I guess, if that's the only option. Thanks

joshmyersover 6 years ago

Looks like it only gives you “ECS Task State Change”, “ECS Container Instance State Change” triggers

Stevenover 6 years ago

Unless they've adding something since I last looked. You subscribe a lambda to the event. The lambda needs to parse to figure out what the change was, then run your logic for that change type.

Stevenover 6 years ago

At that point whether you send to SNS or do anything else is up to you

Igorover 6 years ago

Thanks @joshmyers @Steven

Stevenover 6 years ago

Not great. It needs finer events

timduhenchanterover 6 years ago

How are people dealing with upgrading minor versions of Kubernetes in EKS in large clusters?

December 2019 Browse by date February 2020