#aws - May 2019 | Slack Archive

157 messages

Discussion related to Amazon Web Services (AWS)
Archive: https://archive.sweetops.com/aws/

Apr 2019

May 2019

Jun 2019

Andriy Knysh (Cloud Posse)almost 7 years ago

Apple spends more than $30m a month on AWS, though this number is falling.
Slack's AWS spend is a contractual $50M/year minimum.

Andriy Knysh (Cloud Posse)almost 7 years ago

https://news.ycombinator.com/item?id=19757269

Lee Skillenalmost 7 years ago

... And I thought our bill of ~$20k/year was terrifying 🙂

Erik Osterman (Cloud Posse)almost 7 years ago

https://www.cnbc.com/2019/03/01/lyft-plans-to-spend-300-million-on-aws-through-2021.html

Lee Skillenalmost 7 years ago(edited)

Better to sell shovels than mine gold during a gold rush, eh. 😆

rohitalmost 7 years ago

Does anyone know if it is possible to add tags to S3 object (node sdk) when using presigned url ?

rohitalmost 7 years ago

maybe this is not a right place to ask this question

rohitalmost 7 years ago

but i just did

Erik Osterman (Cloud Posse)almost 7 years ago

Hrmmmmm good question. Don't think it's possible natively, but anything is possible with lambdas 😃

rohitalmost 7 years ago

let me explain my scenario so that you can better understand

rohitalmost 7 years ago

when i want to upload something from my app, i am making a request to my backend service(nodejs) which returns presigned url then i use that in my frontend to directly upload the object from the browser

rohitalmost 7 years ago

i was following the documentation https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#getSignedUrl-property

rohitalmost 7 years ago

it says

Note: Not all operation parameters are supported when using pre-signed URLs. Certain parameters, such as SSECustomerKey, ACL, Expires, ContentLength, or Tagging must be provided as headers when sending a request.

rohitalmost 7 years ago(edited)

so i tried sending the tags in the ajax request with presigned url and i get invalid tag error

Issifalmost 7 years ago

SDK Python but could help : https://stackoverflow.com/questions/52593556/any-way-to-use-presigned-url-uploads-and-enforce-tagging

Issifalmost 7 years ago

To clarify, I need to use {'tagging': '...'} with the verbose XML tag set syntax for both fields and conditions, and that seems to work as required.

Issifalmost 7 years ago

Golang SDK (I do know it much better) says :

Issifalmost 7 years ago(edited)

// The tag-set for the object. The tag-set must be encoded as URL Query parameters.
// (For example, "Key1=Value1")
Tagging *string `location:"header" locationName:"x-amz-tagging" type:"string"

rohitalmost 7 years ago

@Issif thanks. Unfortunately, i did not any luck

Alex Siegmanalmost 7 years ago(edited)

I think just recently I saw a note about a better tool/ui to look at your AWS Parameter Store Secrets, but for the life of me I can't find it. The searching is terrible in the AWS console. All I really want is fuzzy search and proper pagination on searches >.< Anyone have something they know about?

rohitalmost 7 years ago

anyone here used lambda as a compiler ?

rohitalmost 7 years ago

I want to send the code enter in a code editor inside a web application to lambda and compile it and get the results back

rohitalmost 7 years ago

let me know if i am crazy

Tim Malonealmost 7 years ago

i’m sure someone’s done it! ppl have done all sorts of awesomely crazy things in lambda

Tim Malonealmost 7 years ago

sounds like a fun project. you can shell out to anything so i suspect if you packaged a compiler with your code (or downloaded it from s3 during function invocation) then you could run it 🙂

rohitalmost 7 years ago

I want to build some sort of endpoint which accepts code sent from my web app and compiles

rohitalmost 7 years ago

i am trying to figure out how companies like hackerrank,repl.it do it

rohitalmost 7 years ago

i did not find similar usecases online

btaialmost 7 years ago

anyone ever get this error DNS_PROBE_FINISHED_NXDOMAIN intermittently with route 53

btaialmost 7 years ago

accessing a web app

Erik Osterman (Cloud Posse)almost 7 years ago

Is this under k8s?

Erik Osterman (Cloud Posse)almost 7 years ago

Check the logs for kube-dns. Had lots of problems like this in the past, but that was way back on Kubernetes 1.7 or earlier

btaialmost 7 years ago

@Erik Osterman (Cloud Posse) it can’t be the cluster if I do an nslookup on the hostname and it doesn’t resolve the ip addresses. At that point is it a route 53 issue? (Happens intermittently for a few minutes)

btaialmost 7 years ago

Also only started happening since Saturday but 3 times now

btaialmost 7 years ago

But yes on k8s and I don’t see anything out of the ordinary in the kube-dns logs

vishnu.shuklaalmost 7 years ago

Hi, can anyone help, why the deployment is failing

vishnu.shuklaalmost 7 years ago

here is the details what i can see "Invalid action configuration
The action failed because either the artifact or the Amazon S3 bucket could not be found. Name of artifact bucket: codepipeline-eu-central-1-516857284380. Verify that this bucket exists. If it exists, check the life cycle policy, then try releasing a change."

vishnu.shuklaalmost 7 years ago

but the bucket exist and there is no policy as well

Andriy Knysh (Cloud Posse)almost 7 years ago

@vishnu.shukla without seeing the code, it’s difficult to say anything. Make sure you configured the input and output artifacts for all stages correctly (the output from one stage should go to the input to the next stage), for example https://github.com/cloudposse/terraform-aws-cicd/blob/master/main.tf#L233

Andriy Knysh (Cloud Posse)almost 7 years ago

also make sure you setup all the required permissions to access the bucket https://github.com/cloudposse/terraform-aws-cicd/blob/master/main.tf#L101

vishnu.shuklaalmost 7 years ago

Sure Aknysh, thanks a lot

btaialmost 7 years ago

anyone else have issues with route53 not resolving dns for you lately?

Maciek Strömichalmost 7 years ago

If anyone started to experience issues with MySQL RDS it’s because of today’s route53 outage

Maciek Strömichalmost 7 years ago

because how connection is being established with mysql you can see a lot of


| 23653344 | unauthenticated user | ip:port  | NULL    | Connect | NULL | login            | NULL                  |

in
show full processlist;

Maciek Strömichalmost 7 years ago

you can set skip-name-resolve to 1 in your parameters group fix the issue

Maciek Strömichalmost 7 years ago

sadly it’s a partial fix ;-/

Maciek Strömichalmost 7 years ago

how do you set --dns-opt in AWS ECS? it’s not avaialble via dns settings in ecs agent. i know that I can update resolv.conf via entrypoint.sh custom script but I wonder if there’s a better/easier way

Bogdanalmost 7 years ago

Anyone here used https://aws.amazon.com/solutions/centralized-logging/? I'm considering but at the same hesitating due to costs (their cheapest cluster starts from 35 USD/day - https://docs.aws.amazon.com/solutions/latest/centralized-logging/considerations.html#custom-sizing as well as complexity (logs are first collected in CW Logs then get to ES)

Bogdanalmost 7 years ago

I'd much rather prefer them being sent directly to an ES cluster via an Interface VPC Endpoint of course

Tim Malonealmost 7 years ago

Haven’t used that solution, but most AWS-vended logs end up in either CloudWatch or S3 (there’s no native ability to send to ES) so unfortunately there’s not much way around the complexity. For logs on instance, though, I would recommend something like Filebeat rather than going via CW

Maciek Strömichalmost 7 years ago

We use kinesis firehose to send logs directly to es

Maciek Strömichalmost 7 years ago

And simple lambda to send the ones that end up in cloud watch logs

Maciek Strömichalmost 7 years ago

Much simpler than what AWS proposed in this doc

joshmyersalmost 7 years ago

I wrote a blog post about this https://joshmyers.io/blog/aws-logging-pipeline.html

vishnu.shuklaalmost 7 years ago

IAM user has AmazonS3FullAccess then also she fails to upload and download the file

vishnu.shuklaalmost 7 years ago

any clue why?

Maciek Strömichalmost 7 years ago

@joshmyers


A CloudWatch Log Group can subscribe directly to ElasticSearch, so why bother with Kinesis and Lambda? Flexibility and ElasticSearch lock-in

Subscribing to ES from CW logs requires a lambda function that will translate gzipped format of CW Logs into ES. If someone did not automate it and just clicked subscribe then the lambda will be created automatically but automation requires maintaining this lambda function.

joshmyersalmost 7 years ago

Ah, good to know. This was going back some years now, not sure what may have changed too

joshmyersalmost 7 years ago

Am working on something similar now, but significantly more complex for a global client

joshmyersalmost 7 years ago

~15TB a day of log data

Maciek Strömichalmost 7 years ago(edited)

only app logs or os logs as well?

joshmyersalmost 7 years ago

+ flowlogs + elb + s3 + ALL THE LOGS

Maciek Strömichalmost 7 years ago

yeah that can be complex to maintain

joshmyersalmost 7 years ago

Across multiple (read: hundreds) of AWS accounts and back to a Splunk cluster backed by tin on-prem

joshmyersalmost 7 years ago

You can guess where the bottleneck is 😂

Maciek Strömichalmost 7 years ago

depends on the on-prem part

Maciek Strömichalmost 7 years ago

😉

joshmyersalmost 7 years ago

spoiler: It is terrible

Stevenalmost 7 years ago

If you use fargate, it can log directly to splunk now (small good news for you :)

Maciek Strömichalmost 7 years ago

I guess that even our office connectivity which is ~1Gbps would be a problem in such a setup

Maciek Strömichalmost 7 years ago

also awslogs can stream to splunk directly afair

joshmyersalmost 7 years ago

the awslogs agent?

joshmyersalmost 7 years ago

That is news to me

joshmyersalmost 7 years ago

This particular engagement is more complex because Splunk is on-prem

Stevenalmost 7 years ago

Splunk was added to fargate logger about 2 weeks ago. Not sure when it was added to awslogs agent.

joshmyersalmost 7 years ago

I can’t find any info on awslogs agent and Splunk

Stevenalmost 7 years ago

awslogs is specific for logging to CloudWatch. Was surprised when @Maciek Strömich said it could log to splunk. I can't find anything either. Just the standard stream from CloudWatch to splunk via lambda, etc

Maciek Strömichalmost 7 years ago(edited)

ah. it’s not awslogs but docker logging driver that logs to splunk

Maciek Strömichalmost 7 years ago

my bad

Maciek Strömichalmost 7 years ago

i thought that defining logging in Dockerrun.aws.json configures awslogs to ship logs to any of the available loggers

Maciek Strömichalmost 7 years ago

https://github.com/localstack/localstack

Bogdanover 6 years ago

trying my luck here as well

Maciek Strömichover 6 years ago

ssm parameter store has few purposes which one of them is not exposing e.g. secrets. the correct way to do it is to integrate other tools with ssm parameter store not expose them via terraform

Maciek Strömichover 6 years ago

(and yes I can understand that it’s not always possible)

Bogdanover 6 years ago

thanks @Maciek Strömich - I’m not using Terraform to expose them, but to provision infrastructure. Once provisioned successfully the ARNs, IDs and names of those resources are stored in the JSON-like state file. If i’d like to reference them from another framework like Serverless or CDK I need to use HCL and the remote state datasource.
The reason for using SSM (Parameter Store) which also has String and StringList types is to allow others to get the IDs/ARNs/etc of the resources built with Terraform

Maciek Strömichover 6 years ago

ah, that makes more sense

Maciek Strömichover 6 years ago

I’m not even a terraform noob so can’t help with that.

Maciek Strömichover 6 years ago

I misread your message

Maciek Strömichover 6 years ago

sorry for making noise

Andriy Knysh (Cloud Posse)over 6 years ago

@Bogdan we do it all the time - store TF outputs in SSM for later consumption from other modules or apps

Andriy Knysh (Cloud Posse)over 6 years ago

for example, https://github.com/cloudposse/terraform-aws-ecs-atlantis/blob/master/main.tf#L190 - here we save a bunch of fields into SSM

Andriy Knysh (Cloud Posse)over 6 years ago

Then we use chamber to consume them

Andriy Knysh (Cloud Posse)over 6 years ago

https://github.com/cloudposse/geodesic/blob/master/rootfs/etc/init.d/atlantis.sh#L44

Andriy Knysh (Cloud Posse)over 6 years ago

You can use any other means to read them, terraform or SDK

Bogdanover 6 years ago

@Andriy Knysh (Cloud Posse) I saw that, but unfortunately not all modules are built like yours: meaning that your solution is module dependent, so if I haven’t written the module myself just using one from the registry I can’t go and create N+1 PRs to add SSM params to all the modules that are open-source

Bogdanover 6 years ago

@Andriy Knysh (Cloud Posse) I could however use the module outputs and do them outside

Andriy Knysh (Cloud Posse)over 6 years ago

You can wrap any module in your module and then add SSM write for the outputs

Bogdanover 6 years ago

which is what I’ll try to do, but it’s still a suboptimal solution as I have to do it everytime I create a resource/module I’d like in SSM

Andriy Knysh (Cloud Posse)over 6 years ago

That’s what you have to do anyway since not all modules will need SSM

Bogdanover 6 years ago

I started building something that iterates through terraform state list, then calls terraform state show on a particular type of resource - like VPCs, subnet_ids, etc

Bogdanover 6 years ago

so I don’t have to do it at module init or handle it on a per-module basis

Bogdanover 6 years ago

it’s just a pity that terraform state show doesn’t return JSON

Bogdanover 6 years ago

and I have to combine head and awk for getting the value that interests me, which then I have to aws ssm put-parameter with

Andriy Knysh (Cloud Posse)over 6 years ago

Hmm, that should work, but looks complicated. Why not just assemble low level modules into a top level module and then write just those outputs that you really need to SSM

Bogdanover 6 years ago

I’m actually considering doing that since it’s faster

Andriy Knysh (Cloud Posse)over 6 years ago

there is a separate module for writing and reading to/from SSM https://github.com/cloudposse/terraform-aws-ssm-parameter-store

Bogdanover 6 years ago

@Andriy Knysh (Cloud Posse) did you also get error creating SSM parameter: TooManyUpdates?

Bogdanover 6 years ago

it appears to be a known issue: https://github.com/terraform-providers/terraform-provider-aws/issues/1082

Abel Luckover 6 years ago

I'm looking for a solution to manage ssh access to internal hosts via a bastion box. Teleport way more than what we need.

Abel Luckover 6 years ago

my team all use SSH keys on physical tokens. so far we bake the ssh keys into the AMIs

Abel Luckover 6 years ago

but revoking a users access requires rebuilding the amis, which isn't ideal.

Abel Luckover 6 years ago

Hoping to find a simple system to dynamically add/remove keys

Rice Bowl Juniorover 6 years ago

Maybe via Secret Manager?
And load the Keys on startup via a common tag or something like that.
When you want to delete an access, juste delete the secret manager entry.

Haven't tried that, just suggesting things

PePe Amengualover 6 years ago

bastion is easier in the sense you revoke keys to users only on the bastion/s host/s

PePe Amengualover 6 years ago

and then you can just open ssh from that specific host

Juan Cruz Diazover 6 years ago

Hi everyone! I’d like you to invite to the next chapter of our webinar series next Thursday 04/30, where we’ll talk about how to create and administrate a productive environment reaching operational excellence, and how to incorporate this processes to your workspace.

It’s a great learning opportunity no matter what role you have, as long as your business relies on IT workloads.

See you there!

https://www.eventbrite.com.ar/e/alcanzando-la-excelencia-operacional-tickets-62208718953

Abel Luckover 6 years ago(edited)

I'm thinking of trying out the system manager sesssion manager feature of aws, and do away with a bastion entirely. we only need shell access for debugging, so accessing it through the session manager will get us auditing and remove the need for key management.

Sureshover 6 years ago

Ansible should straight away help instead of startup scripts / AMI builds.

Tim Maloneover 6 years ago

Session manager is great - love being able to use IAM to control ‘SSH’ access. Only thing it’s missing is if you need to SCP stuff - we’ve written a quick wrapper for aws s3 cp to make that feel a bit more native (basically using S3 as a proxy of sorts, so you have to run it both locally and remotely to push/pull the file you want).

Bogdanover 6 years ago

anyone encountered The AWS Access Key Id you provided does not exist in our records after aws sts assume-role and exporting all the output into ENV vars?

Bogdanover 6 years ago

and I couldn’t fix it with https://aws.amazon.com/premiumsupport/knowledge-center/access-key-does-not-exist/

Issifover 6 years ago

We use SSM too, I wrote a small Go program that let us to select instance quickly

Issifover 6 years ago

Issifover 6 years ago(edited)

you can select your instance with arrows, of filter by taping something, and an enter connect you to the instance directly

Issifover 6 years ago

IAM manages who can access

Tim Maloneover 6 years ago

that looks nice @Issif! i don’t suppose… you’ve open sourced that? 😄

Issifover 6 years ago

I want to, still discussing about

Abel Luckover 6 years ago

so testing session manager has been going well. our team likes it.

Abel Luckover 6 years ago

we don't copy files, so that hasn't been an issue

Abel Luckover 6 years ago

however it turns out we did use ssh tunnels to access RDS postgres instances for running ad hoc analytics/queries

Abel Luckover 6 years ago

thinking about how best to manage that now

Issifover 6 years ago

@Abel Luck same trouble for us, don't find out yet how to manage tunnels to access to RDS

Issifover 6 years ago

we do copy files, but we use S3 for that, even with presigned URL to PUT/GET

Issifover 6 years ago

the most important gain for my part, is that I'm able to quickly connect to any instance while an on-call triggered without waiting VPN be UP

Issifover 6 years ago

I'm thinking about how to integrate direclty in my golang app the websocket connection, no extra dependency

Issifover 6 years ago

as you seem to be several interesting, will try to release tomorrow (need to find out a better name)

Abel Luckover 6 years ago

we use metabase for doing adhoc queries to share with other folks and it works quite well, gotta get devs/sysadmins to use it too now

Abel Luckover 6 years ago

https://metabase.com/

Abel Luckover 6 years ago

though it's more for read-only querying.

Abel Luckover 6 years ago

maybe an instance with pgadmin accessed via vpn would suffice too

Issifover 6 years ago

this a kind of phpmyadmin or adminer?

Issifover 6 years ago

a tool I wrote and I would like to make FOSS is that :

Abel Luckover 6 years ago

yea

Issifover 6 years ago

tomvover 6 years ago

can you not give metabase the RDS hostname? or is it in a different VPC?

Abel Luckover 6 years ago

exactly, deploy metabase inside the vpc, route through a LB.

Issifover 6 years ago

@Tim Malone @Daniel Lin https://github.com/claranet/sshm

Issifover 6 years ago

need a binary?

Issifover 6 years ago

i'm planning to use goreleaser soon

Mebover 6 years ago

doesn't support aws-vault

Mebover 6 years ago

@atom seem you are the maintainer in Oxalide 😉

Mebover 6 years ago

would be great if we can keep the credential in safe vault instead of plain clear text

Issifover 6 years ago

we don't have secrets in our aws/config

Issifover 6 years ago

we use our adfs to connect on our main account and then switch role

Issifover 6 years ago

can you tell me more about what you want please

April 2019 Browse by date June 2019