#aws - January 2019

Anyone tried to figure out what the SSM Sessions Manager client binary is doing ?

😃 is this the one that's closed source?

yeah I guess so ..

whoaaaaaaa

now to move jenkins slaves to ecs fargate 😧

@LeoGmad has joined the channel

that’s nice

@Maciek Strömich has joined the channel

Discovered this today

didn’t realize they had an official ecs-specific cli tool

Actually, it was improved greatly lately

it used to be very limited, we are considering migrating to use that to manage the services+docker-compose files

as tbh terraform is really shitty for deploying ECS tasks

yeap, and many people like to use this - https://github.com/silinternational/ecs-deploy

Anyone work with EKS clusters and Databases being in seperate VPCs?

@Matthew you have to do VPC peering and add the EKS workers security group as ingres to the database security group

this is how you get the EKS workers SG https://github.com/cloudposse/terraform-aws-eks-workers/blob/master/outputs.tf#L61

this is how you do VPC peering b/w EKS VPC and backing services VPC https://github.com/cloudposse/terraform-root-modules/tree/master/aws/eks-backing-services-peering

in the database module, you can allow SG from EKS workers, e.g. https://github.com/cloudposse/terraform-aws-rds-cluster/blob/master/main.tf#L22

or from the CIDR block https://github.com/cloudposse/terraform-aws-rds-cluster/blob/master/main.tf#L29

You're a God sent @Andriy Knysh (Cloud Posse) I appreciate you, going to explore now

@Matthew if you need read only, it’s possible to setup replicas in a separate VPC without peering

but for rw, you’ll want the peering

Oh man this is awesome, it supports EFS 🙂

oh wow, finally a real backup solution for EFS?

that's what the docs say

freaking finally!

😞 not in our region

which region is that ?

Frankfurt

oh, yeah they are always behind 😞

does each rds instance type have a max num of connections it can possibly have?

Not sure about hardlimits

That's mostly constrained by instance type. The only chart I have seen is of default values by instance type.

im trying to find concrete documentation in aws about connection limits for instance types but i cant really find it

I'm super excited for a backup solution for EFS, assuming it's sane.

As I had to write one previously

and it works fine, but it's just annoying.

ya, we were using datapipelines to call s3 sync

hmm I never thought of doing it that way

that's actually clever

I like it

https://github.com/awslabs/efs-backup - I forked this and made it not stupidly expensive

by backing up to an EBS volume that gets snapshotted etc

as I didn't like the idea of using hardlinks or using yet another EFS file system as a backup destination

lots of lambda

and terrible bash scripts

@Erik Osterman (Cloud Posse) do you know if increasing IOPS has an impact on performance while it is being applied?

I don’t...

Hey folks, is anyone here using ses configuration-set to track open/click events? general configuration works fine but I’m trying to figure out a more fine grained solution where I can graph individual link clicks in cloudwatch.

I’m trying with different ses:tags in email links but it seems that regardless of the configuration they always are categorised as a general click event in the configuration-set name in cloudwatch metrics

Oh interesting… haven’t ever looked into doing that.

Maybe some one in #terraform has seen something

it’s easy with sns/firehose because you can either save the object in s3 or you can trigger lambda function which will then put it in whatever service there is

but there’s also direct cloudwatch destination which seems perfect

especially if you’re interested only in aggregations

and graphs

but it seems it doesn’t work as expected

or maybe I expect too much from ses -> cloudwatch integration ;D

Is there an MFA solution for Windows bastion hosts (for RDP)?

Anybody knows a tool to forward logs from a k8s kind pod/container (minicube or such) into cloudwatch?

we have this https://github.com/cloudposse/prometheus-to-cloudwatch, it forwards logs from prometheus to CloudWatch (you need to have a Prometheus endpoint to scrape)

@tweetyix if you’re using files to store logs then most probably you’re already mounting logs dirs from host to your containers. in that case start a container with awslogs and point it to the main logs directory

Hey people. I've a question. How do you tackle Datadog with ECS in awsvpc mode.
e.g. If I have all tasks running in awsvpc mode then I would need service discovery to be able to get to the ip:port of the DD daemon.

If I use bridge mode I have the same problem, I need the ip of the instance

if I stick the DD docker in the task , then I have repetition in the task definitions for every app.

wat do ?

run it as a task with replication mode DAEMON

that's all fine

but afaik the app needs to push metric to it's endpoint

which is an ip:port combo

afaik the docker host always has a bridge ip, like 172.17.0.1

using rep mode daemon, the port is the same everywhere

so if the aps that run in awsvpc mode can see this I am good to go

you'd need to test that, haven't tried myself

it doesn work 😞

and what about ping $(curl <http://169.254.169.254/latest/meta-data/local-ipv4>)

ok, but you allow the ec2 metadata from your tasks ?

well that's what I am thinking now

why this is not working

give me asec

wait I am drunk 🙂

it works

yup it gives the ip of the instance

allright

that's one way to do it thanks!

@Maciek Strömich I don't have access to the host, neither persistent storage. Do you know if awslogs could be setup inside the container?

@tweetyix yeah it can but this would kind of break the concept of single purpose containers because you would need some process supervisor which would be PID 0

Maybe you can use a sidekick container and a shared volume

otherwise, why not use the docker log drivers?

Get the logs into CloudWatchLogs and then you can pump into Datadog?

thoughts on ebs optimized?

Do you need it?

when does it make sense?

when you look at the IO charts in cloudwatch and see that IO is a bottleneck (pegged)

Note that if you blow through your EBS credits, no bueno

Your instance basically becomes unusable and CPU spikes as things queue up waiting for IO

bad bad bad

if all my services use fargate, is there any reason why i would not just place them all into a single ECS cluster? what are some reasons for having a separate cluster for each service?

So I think it comes down to a few things

Mostly the security architecture

You might want to run in multiple AWS accounts (recommended)

E.g. Dev, Staging, and Prod.

You could also think of each ECS Fargate cluster as a namespace

There's no extra cost to run more than one fargate cluster, so it can be a way to logically isolate resources

thanks erik! makes sense 👍️

Finally!

Is it possible to store a CodeDeploy package in a different region? The s3:// protocol syntax does not seem to provide the ability to specify a region.

S3 bucket names are globally unique

but when you create a bucket, you specify a region where it will be hosted

so a bucket always belongs to a region

if you specify a bucket hosted in diff region in CodeDeploy, it will use it (if all the permissions are in place)

@Andriy Knysh (Cloud Posse) That's what I thought, but I got the following error message:

The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.

Or is this a permissions error?

The instance has the AmazonEC2RoleforAWSCodeDeploy policy which allows s3:GetObject on *

The revision is using the following location: s3://{bucket_name}/{key_prefix}/{key}.zip

maybe you need to use S3 regional endpoints https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region

Hi all.
An ELB/ALB with a public IP can serve instances/target groups with private IPs, correct?

Yea, provided routes and security groups are setup correctly

That won’t depend on a IGW or NATgw, right?

And the instances won’t be able to access the public internet beyond serving through the ELB/ALB

If one of the above isn’t present

(Well, if the NATgw isn’t present, since they don’t have public IPs)

So if there is no NGW they won’t be able to egress directly to the public

But the ALB can still send/receive requests to the instance

The public subnet will need an IGW

And the ALB will need to be on the public subnet

Ah of course, LB wouldn’t be able to get out without IGW.

Thanks again Erik

a useful little plugin for AWS if you have many roles https://github.com/tilfin/aws-extend-switch-roles

that's slick

if you have missed it. https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-attribute-updatereplacepolicy.html

cloudformation has now an attribute protecting you from accidental data loss upon update replacement

and also EKS became ISO and PCI compliant

Oh great!

I didn’t see that

yeah, last week

and also last week AWS introduced pull from private ecr repositories with secrets manager integration

I don’t think we handle that

Also, this is related to #terraform ?

Thanks @Erik Osterman (Cloud Posse) - I'll ask in #terraform