#geodesic - January 2019 | Slack Archive

geodesic

455 messages

Discussions related to https://github.com/cloudposse/geodesic

Archive: https://archive.sweetops.com/geodesic/

Dec 2018

January 2019

Feb 2019

Janalmost 8 years ago

wow man, trying to get back into this after a 2 week brake is hurting my mind 😛

joshmyersalmost 8 years ago

Too much 🧀 and 🍷 !

Janalmost 8 years ago

haha

Janalmost 8 years ago

in the tf root modules is there a list someplace as to what all the backing services relate to?

Erik Osterman (Cloud Posse)almost 8 years ago

can you expand on “relate to”?

Erik Osterman (Cloud Posse)almost 8 years ago

---

Erik Osterman (Cloud Posse)almost 8 years ago

I’ve jotted down some notes on #atlantis with #geodesic

Erik Osterman (Cloud Posse)almost 8 years ago

https://github.com/cloudposse/docs/issues/355

Dave Yualmost 8 years ago

Not sure if I configured something wrong, but is there a way to keep an assume-role session active indefinitely?

[2]+  Stopped                 aws-vault exec ${AWS_VAULT_ARGS[@]} $role -- bash -l
-> Run 'assume-role' to login to AWS
 ⧉  testing
 ✗   (none) ~ ⨠  assume-role
Enter passphrase to unlock /conf/.awsvault/keys/:
aws-vault: error: Failed to start credential server: listen tcp 127.0.0.1:9099: bind: address already in use

Erik Osterman (Cloud Posse)almost 8 years ago

Hrm.....

Erik Osterman (Cloud Posse)almost 8 years ago

so yes, it should be possible. We do run the aws-vault server in the background

Erik Osterman (Cloud Posse)almost 8 years ago

I am using a VM (AWS WorkSpace), so it doesn't work for me

Erik Osterman (Cloud Posse)almost 8 years ago

aws-vault: error: Failed to start credential server: listen tcp 127.0.0.1:9099: bind: address already in use

Erik Osterman (Cloud Posse)almost 8 years ago

This doesn't look right

Erik Osterman (Cloud Posse)almost 8 years ago

Are you using a recent version of geodesic?

Dave Yualmost 8 years ago

It’s only annoying because I find I have to exit out of geodesic completely before I can successfully assume-role again.

Erik Osterman (Cloud Posse)almost 8 years ago

hrmmm

Erik Osterman (Cloud Posse)almost 8 years ago

If that's the case, that's a bug 😃

Dave Yualmost 8 years ago

yeah i just rebuilt FROM cloudposse/geodesic:0.57.0

Erik Osterman (Cloud Posse)almost 8 years ago

ok, that's recent

Erik Osterman (Cloud Posse)almost 8 years ago

sec

Erik Osterman (Cloud Posse)almost 8 years ago

to help me triage this...

Erik Osterman (Cloud Posse)almost 8 years ago

are you running multiple geodesic containers at the same time?

Erik Osterman (Cloud Posse)almost 8 years ago

are you using aws-vault server mode natively on yourmac?

Erik Osterman (Cloud Posse)almost 8 years ago

if you run your geodesic shell and then run ps uxaww do you see aws-vault server running?

Erik Osterman (Cloud Posse)almost 8 years ago

when you start the shell, do you see the message:


* Started EC2 metadata service at <http://169.254.169.254/latest>

Erik Osterman (Cloud Posse)almost 8 years ago

ugh

Erik Osterman (Cloud Posse)almost 8 years ago

(was hoping for the opposite order of thumbs)

Dave Yualmost 8 years ago

aws-vault exec --assume-role-ttl=1h --server lootcrate-testing-admin -- bash -l

Dave Yualmost 8 years ago

guessing im running into problems at the 1h mark

Erik Osterman (Cloud Posse)almost 8 years ago

ohh

Erik Osterman (Cloud Posse)almost 8 years ago

ok, try this:

Erik Osterman (Cloud Posse)almost 8 years ago

AWS_VAULT_ASSUME_ROLE_TTL=24h

Erik Osterman (Cloud Posse)almost 8 years ago

set that in the Dockerfile and rebuild

Dave Yualmost 8 years ago

fyi aws-vault: error: Maximum duration for assumed roles is 12h0m0s

Erik Osterman (Cloud Posse)almost 8 years ago

I wonder why I don't see this [2]+ Stopped aws-vault exec ${AWS_VAULT_ARGS[@]} $role -- bash -l

Dave Yualmost 8 years ago

ok this is getting somewhere:

aws-vault: error: Failed to get credentials for lootcrate (source profile for lootcrate-testing-admin): ValidationError: The requested DurationSeconds exceeds the MaxSessionDuration set for this role.
	status code: 400, request id: ef5c9f5d-12c6-11e9-bd1b-b5609fd02acf

Erik Osterman (Cloud Posse)almost 8 years ago

oh interesting.

Erik Osterman (Cloud Posse)almost 8 years ago

(a) I didn't know it was possible to scope it to a role (b) I feel like it's a red herring since this used to work

Dave Yualmost 8 years ago

Andriy Knysh (Cloud Posse)almost 8 years ago

@Dave Yu what version of geodesic are you suing?

Erik Osterman (Cloud Posse)almost 8 years ago

see above ☝️

Erik Osterman (Cloud Posse)almost 8 years ago

0.57

Erik Osterman (Cloud Posse)almost 8 years ago

so the way it's supposed to work is actually unreltaed to that ttl

Erik Osterman (Cloud Posse)almost 8 years ago

basically, it could be 60 seconds

Erik Osterman (Cloud Posse)almost 8 years ago

the way it's supposed to work is when something needs AWS access, it uses the metadata api

Erik Osterman (Cloud Posse)almost 8 years ago

that is proxied by aws-vault server which mocks the API

Erik Osterman (Cloud Posse)almost 8 years ago

then it fetches some temporary credentials.

Erik Osterman (Cloud Posse)almost 8 years ago

Can you open an issue against cloudposse/geodesic with this info

Dave Yualmost 8 years ago

sure thing

Erik Osterman (Cloud Posse)almost 8 years ago

I'll have someone look into it.

Erik Osterman (Cloud Posse)almost 8 years ago

PSA: there have been a lot of awesome updates to geodesic in the past month

Erik Osterman (Cloud Posse)almost 8 years ago

if you haven't upgraded, do give it a shot.

Erik Osterman (Cloud Posse)almost 8 years ago

if you're using k8s, then the kube-ps1 prompt is really slick

Erik Osterman (Cloud Posse)almost 8 years ago

if you want to run multiple k8s clusters (kops) per account, that' now supported using the direnv pattern

Erik Osterman (Cloud Posse)almost 8 years ago

we're using geodesic regularly with #atlantis, so if that's interesting hit me up

Erik Osterman (Cloud Posse)almost 8 years ago

we've added a script called aws-config-setup which makes it easier for users to setup aws-vault (no copy pasta)

Erik Osterman (Cloud Posse)almost 8 years ago

we're adding iam-authenticator support to kops

Erik Osterman (Cloud Posse)almost 8 years ago

we've enhanced kubens and kubectx autocomplete with fzf

Erik Osterman (Cloud Posse)almost 8 years ago

we've added tfenv to automatically export envs for terraform consumption

Erik Osterman (Cloud Posse)almost 8 years ago

for terraform-root-modules, we've added support for multple new account types: identity, corp, security as well as generalized the pattern so we can add more account types easily

Erik Osterman (Cloud Posse)almost 8 years ago

we've started embracing SSM with terraform so that we automatically populate settings for consumption with chamber (no more copy pasta)

Erik Osterman (Cloud Posse)almost 8 years ago

anyways, if any of this sounds interesting and you want more info/demo, hit me up.

sarkisalmost 8 years ago

I think CloudPosse needs a newsletter 😄

sarkisalmost 8 years ago

I'd love to get updates like this weekly or monthly

Andriy Knysh (Cloud Posse)almost 8 years ago

https://cloudposse.com/newsletter/

Andriy Knysh (Cloud Posse)almost 8 years ago

https://cloudposse.com/newsletters/

sarkisalmost 8 years ago

thanks @Andriy Knysh (Cloud Posse)! signed up 😄

Erik Osterman (Cloud Posse)almost 8 years ago

@sarkis yes - we’re well overdue for the next edition

Erik Osterman (Cloud Posse)almost 8 years ago

I not had a chance to it together

sarkisalmost 8 years ago

catching up with all the new geodesic changes 😄

sarkisalmost 8 years ago

https://github.com/kubernetes-sigs/aws-iam-authenticator

Erik Osterman (Cloud Posse)almost 8 years ago(edited)

right now, we’re working on getting the slack archives published on a static site so we can have the history available

Erik Osterman (Cloud Posse)almost 8 years ago

are you guys looking at k8s now?

sarkisalmost 8 years ago

not yet 😞

sarkisalmost 8 years ago

at least a quarter out

sarkisalmost 8 years ago

im just staying in touch with everything - getting ready

Erik Osterman (Cloud Posse)almost 8 years ago

that’s cool - at least it’s on the horizon

Janalmost 8 years ago

Almost all the additions I will need

Janalmost 8 years ago

Will do an upgrade today

Erik Osterman (Cloud Posse)almost 8 years ago

@Jan we still need to work on injecting kops to existing subnets/vpcs

Janalmost 8 years ago

yes though that will fall to my backlog as I will do the vpc creation via kops for now

Janalmost 8 years ago

its a very easy though

Janalmost 8 years ago

simply populate the cluster manifest with the vpc id, the subnet ID's and make sure the cidr is the same

Janalmost 8 years ago

then just kubectl create -f

Erik Osterman (Cloud Posse)almost 8 years ago

Yes, just the devil in the details. :-)

Erik Osterman (Cloud Posse)almost 8 years ago

@Jan regarding the kops manifest, we’ve been using gomplate (go templates) because terraform templates not not enough

Erik Osterman (Cloud Posse)almost 8 years ago

(no conditionals)

Erik Osterman (Cloud Posse)almost 8 years ago

like you said, easy enough to parameterize

Erik Osterman (Cloud Posse)almost 8 years ago

Though I think maybe we would create a separate manifest

Erik Osterman (Cloud Posse)almost 8 years ago

since the changes would be significant.

Erik Osterman (Cloud Posse)almost 8 years ago

have a look at the latest changes for how we provision kops settings from terraform

Erik Osterman (Cloud Posse)almost 8 years ago

https://github.com/cloudposse/terraform-root-modules/blob/master/aws/kops/main.tf#L75-L81

Erik Osterman (Cloud Posse)almost 8 years ago

I am quite fond of it. We write them to SSM from terraform. Consume them with chamber and call build-kops-manifest or kops

Janalmost 8 years ago

Cool

Janalmost 8 years ago

Will be Re setting everything up with the cleaner reference architecture tomorrow morning

Janalmost 8 years ago

And doing kops vpcs

Janalmost 8 years ago

in the new reference arch...

the org cidr range, whats the expectation there?

Janalmost 8 years ago


# Network CIDR of Organization
org_network_cidr    = "10.0.0.0/8"
org_network_offset  = 100
org_network_newbits = 8    # /8 + /8 = /16

Janalmost 8 years ago

that all the sub accounts fit within that cidr?

Janalmost 8 years ago

So at this point I would love to have the new reference arch make root have a --dry-run option so I can still go verify and edit all the configuration before it starts bootstrapping

Janalmost 8 years ago

and then despite having set the region to eu-central-1 there is a hard expressed parameter or a variable thats only doing its default of using us-west-2 for the tfstate-backend s3 bucket

Janalmost 8 years ago


* module.tfstate_backend.aws_s3_bucket.default: 1 error(s) occurred:

• aws_s3_bucket.default: Error creating S3 bucket: IllegalLocationConstraintException: The us-west-2 location constraint is incompatible for the region specific endpoint this request was sent to.

Janalmost 8 years ago

simply adding the following to the Dockerfile.root solves it for this case


# Jan hacking
ENV TF_VAR_region="${aws_region}"

Andriy Knysh (Cloud Posse)almost 8 years ago

@Jan

Andriy Knysh (Cloud Posse)almost 8 years ago

that all the sub accounts fit within that cidr?

Andriy Knysh (Cloud Posse)almost 8 years ago

yes, the idea was that we could peer the VPCs from any of the accounts, so the VPCs should have not-overlapping ranges

Andriy Knysh (Cloud Posse)almost 8 years ago

thanks for finding the issue, we’ll fix that

Janalmost 8 years ago

totally

Janalmost 8 years ago

and I do like the setting of the cidr's per account/vpc

Janalmost 8 years ago

but setting into a pre calculated range doesnt fit, especially for me

Janalmost 8 years ago

and im trying to work out how to use it

Janalmost 8 years ago

same as I dont want to provision the aws_orgs part

Janalmost 8 years ago

as I have the accounts already

Andriy Knysh (Cloud Posse)almost 8 years ago

thanks @Jan

Andriy Knysh (Cloud Posse)almost 8 years ago

any suggestions and improvements are welcome

Andriy Knysh (Cloud Posse)almost 8 years ago

it was the first version of the ref architectures, definitely has room for improvements

Janalmost 8 years ago

I have forked the 3 projects and will work on fixing the tf state bucket var

Janalmost 8 years ago

Already validated it works

Janalmost 8 years ago

And will try run a modded version of the reference architecture that fits my needs

Erik Osterman (Cloud Posse)almost 8 years ago

we’re using tvenv instead of using TF_VAR_ in Dockerfile

Janalmost 8 years ago

I think I found the issue with the tfstate-backend vars

Janalmost 8 years ago

https://github.com/cloudposse/reference-architectures/blob/master/templates/Dockerfile.root#L33-L35

Janalmost 8 years ago

Janalmost 8 years ago(edited)

should those not rather be ENV TF_VAR_

Erik Osterman (Cloud Posse)almost 8 years ago

no more TF_VAR_ ever

Erik Osterman (Cloud Posse)almost 8 years ago

😃

Janalmost 8 years ago

mmmm really

Erik Osterman (Cloud Posse)almost 8 years ago

standardize around normal envs

Erik Osterman (Cloud Posse)almost 8 years ago

that can be used by kops, terraform, scripts etc

Janalmost 8 years ago

so im trying to track down why its reverting to the default value of us-west-2

Erik Osterman (Cloud Posse)almost 8 years ago

use tfenv to export them to terraform

Erik Osterman (Cloud Posse)almost 8 years ago

hrmm

Erik Osterman (Cloud Posse)almost 8 years ago

what do you have in root.tfvars config?

Janalmost 8 years ago


# The default region for this account
aws_region = "eu-central-1"

Erik Osterman (Cloud Posse)almost 8 years ago

hrmm

Erik Osterman (Cloud Posse)almost 8 years ago

ok, let’s address on our call 😃

Janalmost 8 years ago

I ended up with everything getting added in eu-central-1 and a error from s3 when it tried to create the bucket with region set to us-west-2

Erik Osterman (Cloud Posse)almost 8 years ago

ohhh

Erik Osterman (Cloud Posse)almost 8 years ago

that’s TF_STATE_BUCKET_REGION

Erik Osterman (Cloud Posse)almost 8 years ago

we don’t se tthat in the bootstrap (oversight)

Janalmost 8 years ago

I got around it by adding ENV TF_VAR_region="${aws_region}" to the Dockerfile.root

Janalmost 8 years ago

ah!

Janalmost 8 years ago

good catch

Erik Osterman (Cloud Posse)almost 8 years ago

(let me confirm)

Erik Osterman (Cloud Posse)almost 8 years ago

slightly wrong

Erik Osterman (Cloud Posse)almost 8 years ago

https://github.com/cloudposse/geodesic/blob/master/rootfs/usr/local/bin/init-terraform#L5

Erik Osterman (Cloud Posse)almost 8 years ago

TF_BUCKET_REGION

Erik Osterman (Cloud Posse)almost 8 years ago

the problem is when going multi-region with terraform is where the statefiles belong

Erik Osterman (Cloud Posse)almost 8 years ago

hrm

Janalmost 8 years ago

so im not trying to do multi region

Janalmost 8 years ago

lol

Janalmost 8 years ago

lets clarify it later

Erik Osterman (Cloud Posse)almost 8 years ago

https://github.com/cloudposse/terraform-root-modules/blob/master/aws/tfstate-backend/main.tf#L51

Erik Osterman (Cloud Posse)almost 8 years ago

here’s the other problem with the coldstart

Erik Osterman (Cloud Posse)almost 8 years ago

you’re right

Erik Osterman (Cloud Posse)almost 8 years ago

i’ll tell you the fix

Janalmost 8 years ago

yea I saw the default value

Erik Osterman (Cloud Posse)almost 8 years ago

add a config template here:

Erik Osterman (Cloud Posse)almost 8 years ago

https://github.com/cloudposse/reference-architectures/tree/master/templates/conf

Erik Osterman (Cloud Posse)almost 8 years ago

for tfstate-backend

Erik Osterman (Cloud Posse)almost 8 years ago

https://github.com/cloudposse/reference-architectures/blob/master/templates/conf/kops/terraform.tfvars#L4

Erik Osterman (Cloud Posse)almost 8 years ago

then add something like that

Erik Osterman (Cloud Posse)almost 8 years ago

then it will generate a terraform.tfvars for that project

Erik Osterman (Cloud Posse)almost 8 years ago

since each project might be in a different region

Erik Osterman (Cloud Posse)almost 8 years ago

(so there’s nothing wrong with setting TF_VAR_region in the Dockerfile - just I’m not advocating it anymore)

Erik Osterman (Cloud Posse)almost 8 years ago

I like the explicitness of terraform.tfvars

Janalmost 8 years ago

will happily stick to the new convention

Erik Osterman (Cloud Posse)almost 8 years ago

… anyways this is a 🐛

Erik Osterman (Cloud Posse)almost 8 years ago

if you track a list of these issues I’ll get them fixed

Janalmost 8 years ago

👍️

Janalmost 8 years ago

that worked great btw, thanks

Janalmost 8 years ago

is anyone using the new reference architecture with geodesic?

Janalmost 8 years ago

got make root working

Janalmost 8 years ago

though get
make children


make[1]: *** No rule to make target `staging/validate'.  Stop.
make: *** [staging/validate] Error 2

Janalmost 8 years ago

wow the removal and deletion of aws org accounts is a pita

Andriy Knysh (Cloud Posse)almost 8 years ago

yep, TF can’t delete them https://github.com/cloudposse/reference-architectures#known-limitations

Janalmost 8 years ago

Oh yea not even trying via tf

Janalmost 8 years ago

Doing it manually and it's a grind

Janalmost 8 years ago

What's the rational in having the reference architecture route 53 zones public?

Janalmost 8 years ago

I get that you need to have a vpc existing in order create a private zone

Janalmost 8 years ago

Mmmm actually

Janalmost 8 years ago

Re reading the tf docs I need test it

Andriy Knysh (Cloud Posse)almost 8 years ago

@Jan you need them public to be able to access the resources in the clusters, e.g. <http://us-west-1.prod.cloudposse.co|us-west-1.prod.cloudposse.co> is to access the k8s cluster in prod

Janalmost 8 years ago

Absolutely not the case

Janalmost 8 years ago

What I want is public people to access a elb via a set record that has nothing to do with the zones I would use for env cloudposse. Co

Janalmost 8 years ago(edited)

Why leak all the info about all the environments you have and the record in them

Janalmost 8 years ago

I will change our version of the ref arch to build internally

Janalmost 8 years ago

So for example.. Have cloudposse.co as a public zone with an associated private zone

Janalmost 8 years ago

All internal zones should have internal zones

Janalmost 8 years ago

It is an anti pattern to expose anything (including dns hosts information) for anything not intended to be publicly consumed

Janalmost 8 years ago

I normally do what I am describing with split horizon dns

Janalmost 8 years ago

Especially so with any direct connect or vpg/vpn peering

Janalmost 8 years ago

https://aws.amazon.com/premiumsupport/knowledge-center/internal-version-website/

Janalmost 8 years ago

Ah Yea aws call it split-view

Andriy Knysh (Cloud Posse)almost 8 years ago

@Jan you can do it with private zones as you described

Janalmost 8 years ago

Yes

Andriy Knysh (Cloud Posse)almost 8 years ago

another reason to have public zones is when you use kops

Andriy Knysh (Cloud Posse)almost 8 years ago

so I guess there are many diff use-cases here

Janalmost 8 years ago

What I was asking is, given the focus on security in geodesic, the is seems wrong in the reference architecture

Janalmost 8 years ago

Kops works perfectly fine for private

Janalmost 8 years ago

Kops has no hard reliance on public r53 zones

Janalmost 8 years ago

I will rework our fork of the reference architecture tonight and see if I can resolve this

Erik Osterman (Cloud Posse)almost 8 years ago

btw, kops works without any DNS at all

Erik Osterman (Cloud Posse)almost 8 years ago

it supports gossip mode

Erik Osterman (Cloud Posse)almost 8 years ago

we used this with our caltech vanvalen project

Janalmost 8 years ago

is it safe to assuem this is the current state to follow?

Janalmost 8 years ago

https://github.com/cloudposse/geodesic/blob/bbd796d98ab8e0c6176c56ab3ed634a486b70919/docs/kops.md

Janalmost 8 years ago

for getting k8s into geodesic

Janalmost 8 years ago

Cool so I have kops creating a cluster in a vpc with a private hosted zone

Janalmost 8 years ago

all working

Janalmost 8 years ago

need to polish a bit

Janalmost 8 years ago

and stuff

Janalmost 8 years ago

but I have my previous idea, I define my vpc and launch k8s into it

Andriy Knysh (Cloud Posse)almost 8 years ago

Nice @Jan

Janalmost 8 years ago

also using the kops-backing-services

Janalmost 8 years ago

mostly

Janalmost 8 years ago

so with the cluster running as per geodewssic aws/kops method

Janalmost 8 years ago

mostly

Janalmost 8 years ago


helmfiles ⨠  helmfile apply
Adding repo coreos-stable <https://s3-eu-west-1.amazonaws.com/coreos-charts/stable>
"coreos-stable" has been added to your repositories

Updating repo
Hang tight while we grab the latest from your chart repositories...
...Skip local chart repository
...Successfully got an update from the "incubator" chart repository
...Successfully got an update from the "coreos-stable" chart repository
...Successfully got an update from the "cloudposse-incubator" chart repository
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈ Happy Helming!⎈

Comparing prometheus-operator coreos-stable/prometheus-operator
Error: configmaps is forbidden: User "system:serviceaccount:kube-system:default" cannot list configmaps in the namespace "kube-system"
configmaps is forbidden: User "system:serviceaccount:kube-system:default" cannot list configmaps in the namespace "kube-system"
Error: plugin "diff" exited with error

err: exit status 1
failed processing /conf/helmfiles/releases/prometheus-operator.yaml: exit status 1

Janalmost 8 years ago

Is rbac being taken into account?

Janalmost 8 years ago

- rbacEnable: {{ env "RBAC_ENABLED" | default "false" }}

Janalmost 8 years ago

https://github.com/cloudposse/helmfiles/blob/master/releases/prometheus-operator.yaml#L32

Andriy Knysh (Cloud Posse)almost 8 years ago

@Jan by default our helmfiles don’t use RBAC

Janalmost 8 years ago

I am asking if in the geodesic reference arch rbac is being used

Janalmost 8 years ago

ah nm

Janalmost 8 years ago

read your response wrong

Janalmost 8 years ago

so..... I can understand its easier to not

Janalmost 8 years ago

but its much much worse

Andriy Knysh (Cloud Posse)almost 8 years ago

agree

Andriy Knysh (Cloud Posse)almost 8 years ago

do you have all the rbac resources (service account, role binding, etc.) provisioned in the cluster?

Janalmost 8 years ago

just added

Janalmost 8 years ago

more exploring how much is or is not dealt with in the current ref arch

Janalmost 8 years ago

and extending it to do what I need

Janalmost 8 years ago

hAHAHA

Janalmost 8 years ago


Error: identified at least one change, exiting with non-zero exit code (detailed-exitcode parameter enabled)
identified at least one change, exiting with non-zero exit code (detailed-exitcode parameter enabled)
Error: plugin "diff" exited with error

Janalmost 8 years ago

https://github.com/helm/helm/blob/master/docs/rbac.md#example-service-account-with-cluster-admin-role

Janalmost 8 years ago

this by itself is a better place to start

chrismalmost 8 years ago

Hey; I love the centralised management of roles/users using this setup; but how would you restrict a users access at a sub-org level with the current setup? Say for example you have a user jim in root who only has access to the data org, and you want to restrict that to just elasticache in the data org.

Erik Osterman (Cloud Posse)almost 8 years ago

there are a few options

Erik Osterman (Cloud Posse)almost 8 years ago

1) create more groups with fine grained access control like you describe; then add users to those groups.

Erik Osterman (Cloud Posse)almost 8 years ago

2) add the policies directly to the IAM user.

Erik Osterman (Cloud Posse)almost 8 years ago

I like (1) the best.

Erik Osterman (Cloud Posse)almost 8 years ago

what you describe is a highly specific requirement (not wrong!), so we haven’t tried to come up with any patterns to generalize that.

Erik Osterman (Cloud Posse)almost 8 years ago

…perhaps after seeing more patterns like this, we can whip up a module for it 😃

chrismalmost 8 years ago

I'll have a prod around, the less unicorns the better 🙂

Erik Osterman (Cloud Posse)almost 8 years ago

cool

Erik Osterman (Cloud Posse)almost 8 years ago

happy to give you a high level overview / demo of what it all looks like

chrismalmost 8 years ago

I'm guessing the simplest way would be to create a role within the sub-organisation to (as per the example) allow only access to one area
and a group. tbh I probably need to RTFM on iam + suborgs.
Previously we'd manage the users in each org (which is a headache in itself)

chrismalmost 8 years ago

Love the wrapper around aws-vault and the sts usage; its a really nice way to standardise access

Erik Osterman (Cloud Posse)almost 8 years ago

right - avoid managing users in suborgs

Erik Osterman (Cloud Posse)almost 8 years ago

yea, the geodesic shell has made it A LOT easier to distribute an advanced toolchain

Erik Osterman (Cloud Posse)almost 8 years ago

we used to try to do it all “natively” (e..g on osx) and always ran into problems

Erik Osterman (Cloud Posse)almost 8 years ago

old version of XYZ package

Erik Osterman (Cloud Posse)almost 8 years ago

“works on my machine”

Erik Osterman (Cloud Posse)almost 8 years ago

communicating upgrades etc

chrismalmost 8 years ago

We're a mix of windows / nix / mac so its good to see something that wont turn me from a developer into desktop support 😄

Erik Osterman (Cloud Posse)almost 8 years ago

do you guys run WSL?

chrismalmost 8 years ago

Docker is a god send for tooling

Erik Osterman (Cloud Posse)almost 8 years ago

we have early support for that.

chrismalmost 8 years ago

yeah all in WSL

Erik Osterman (Cloud Posse)almost 8 years ago

a few community members were using geodesic in that setting.

chrismalmost 8 years ago

docker for windows; wsl talking over tcp. Works relatively seamlessly

Janalmost 8 years ago

Make vs make :(

chrismalmost 8 years ago

We're mostly using the ref setup (setup individually) for the org management. Our actual infrastructure in the orgs will be a tad more hybrid as we have our own terraform modules etc / splits across regions

Janalmost 8 years ago

Tool chains in docker are the future

chrismalmost 8 years ago

Yeah docker's pretty sweet for simplifying tooling; we already use it as an ops tool for running scans and long custom chains of tools to analyse deployments / nmap / etc.
Kubernetes is a mixed bag though; its like creating a magic-bag for developers to throw half baked code into 😄

chrismalmost 8 years ago

Good work chaps; its nice to see an opinionated, secure by default, design 🙂

Erik Osterman (Cloud Posse)almost 8 years ago

🙇‍♂️ thanks @chrism!

Gonzalo Pecialmost 8 years ago

I was going over geodesic, im trying to understand what goofys is used for, I understand you mount some s3 buckets, but what for?

Gonzalo Pecialmost 8 years ago

as afaik you use terraform backends, so state should work/happen through that

joshmyersalmost 8 years ago

@Gonzalo Peci it was used to mount secrets from s3 to a local filesystem

joshmyersalmost 8 years ago

I think @Erik Osterman (Cloud Posse) is wanting to move away from it and instead use SSM

Gonzalo Pecialmost 8 years ago(edited)

OK, that makes sense, like sensitive tfvars and other variable/config/secrets

Gonzalo Pecialmost 8 years ago

thanks

joshmyersalmost 8 years ago

mounts private keys by the looks of it

joshmyersalmost 8 years ago

Instead favouring things like https://github.com/cloudposse/terraform-root-modules/blob/master/aws/backing-services/aurora-mysql.tf#L105-L121

erikalmost 8 years ago

Erik Osterman (Cloud Posse)almost 8 years ago

who will be lucky #200?

nutellinoitalmost 8 years ago

★ repo 😄

erikalmost 8 years ago

Erik Osterman (Cloud Posse)almost 8 years ago

WOOHOO! thanks @nutellinoit

Janalmost 8 years ago

What version of kops / k8s is supported currently?

joshmyersalmost 8 years ago

https://github.com/cloudposse/geodesic/blob/master/Dockerfile#L112

joshmyersalmost 8 years ago

https://github.com/cloudposse/geodesic/blob/master/Dockerfile#L130

Andriy Knysh (Cloud Posse)almost 8 years ago(edited)

@Jan we recently deployed 1.10.12 with kops

Andriy Knysh (Cloud Posse)almost 8 years ago(edited)

https://github.com/kubernetes/kops#kubernetes-version-support
https://github.com/kubernetes/kubernetes/releases

Janalmost 8 years ago

Awesome

Janalmost 8 years ago

Will start at that Version

Dave Yualmost 8 years ago

trying to hack my way through using the reference architecture with an existing root account. got prod and audit accounts created with make root, but now running into this:

➜ make children
make[1]: *** No rule to make target `prod/validate'.  Stop.
make: *** [prod/validate] Error 2

Dave Yualmost 8 years ago

artifacts/Makefile.env looks right:

ACCOUNTS_ENABLED = prod audit

and the account IDs are in artifacts/acounts.tfvars

Dave Yualmost 8 years ago

where else should I look for problems?

Erik Osterman (Cloud Posse)almost 8 years ago

ohhhh yes, there’s a bug.

Erik Osterman (Cloud Posse)almost 8 years ago(edited)

edit tasks/Makefile.child

Erik Osterman (Cloud Posse)almost 8 years ago

find the macro called child

Erik Osterman (Cloud Posse)almost 8 years ago

there is an = sign after it. that works in some makes and not in others

Erik Osterman (Cloud Posse)almost 8 years ago

remove the = and it will work.

Dave Yualmost 8 years ago

👍️ beautiful. thanks

Dave Yualmost 8 years ago

if it helps..

GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for i386-apple-darwin11.3.0

Erik Osterman (Cloud Posse)almost 8 years ago

@Jan ran into this

Erik Osterman (Cloud Posse)almost 8 years ago

we spent some time debugging it. Yea, I’ve been developing it on an AWS WorkSpace instance which is Amazon Linux

Erik Osterman (Cloud Posse)almost 8 years ago

on OSX this happens

Erik Osterman (Cloud Posse)almost 8 years ago

https://github.com/cloudposse/reference-architectures/issues/10

erikalmost 8 years ago

Erik Osterman (Cloud Posse)almost 8 years ago

Geodesic inside of tmate over the web 😃

Erik Osterman (Cloud Posse)almost 8 years ago

https://github.com/cloudposse/geodesic/pull/359

Erik Osterman (Cloud Posse)almost 8 years ago

🧠 💥

mumoshualmost 8 years ago

@mumoshu has joined the channel

Erik Osterman (Cloud Posse)over 7 years ago

https://www.reddit.com/r/devops/comments/ahzof3/which_technology_tool_you_use_the_most_for/eejyd3v/

Janover 7 years ago

ha that gnu make on osx thing is a pita

chrismover 7 years ago

Is there a reason the org names are filtered to a predefined list rather than just allowing anything?

Erik Osterman (Cloud Posse)over 7 years ago

it’s to reduce the number of settings

Erik Osterman (Cloud Posse)over 7 years ago

the more predictable they are, they more assumptions we can make

Erik Osterman (Cloud Posse)over 7 years ago

… if we’re talking about terraform-root-modules, the idea here is that these are hyper opinionated.

Erik Osterman (Cloud Posse)over 7 years ago

All of our other terraform-aws-* modules are designed to be building blocks

Erik Osterman (Cloud Posse)over 7 years ago

but at some point, you need to stop generalizing and implement something. That’s terraform-root-modules 😃

Erik Osterman (Cloud Posse)over 7 years ago

@chrism would be happy to give you a high level over view of everything and some of the design decisions.

Erik Osterman (Cloud Posse)over 7 years ago

Of course, we want to capture all that in documentation, but that is not caught up.

chrismover 7 years ago

Everyone loves documenting things 😄

Erik Osterman (Cloud Posse)over 7 years ago

lol

chrismover 7 years ago

I was thinking more along the lines of
root / audit / maybe data would be opinionated, if its not in the list assume its more of a blank slate like dev or prod sans the pre-included docker

Erik Osterman (Cloud Posse)over 7 years ago

soooooo what I’m leaning towards right now is adding more types of accounts

Erik Osterman (Cloud Posse)over 7 years ago

rather than renaming the current accounts

Erik Osterman (Cloud Posse)over 7 years ago

it makes it impossible to document a moving target

Erik Osterman (Cloud Posse)over 7 years ago

what are the account names you’d like to see?

Erik Osterman (Cloud Posse)over 7 years ago

---

chrismover 7 years ago

To be honest its probably a bit domain specific to us but data-prod rather than just data. and something akin to dev-{team}.
I can see myself being buried in orgs which isn't great but at the same time it reduces risk around shared areas between teams / improves the ability to audit / split billing.

Erik Osterman (Cloud Posse)over 7 years ago

another idea that @Jan had is to supply another “set” of accounts

chrismover 7 years ago

I also realise i can do similar with policies in aws like we do in our original 1 org structure

chrismover 7 years ago

I'm simply thinking of dumb accounts basically that you can setup as needed / make use of the centralised auth with

Erik Osterman (Cloud Posse)over 7 years ago

So the part that I found tricky was NS delegation.

Janover 7 years ago

The dns delegation 8s easy

Erik Osterman (Cloud Posse)over 7 years ago

It’s totally possible to do what you want to do, just with trade offs.

Janover 7 years ago

I found the getting the networking / vpc stuff more interesting

chrismover 7 years ago

Admittedly not using any of the K8 / domain stuff as we use rancher for K8 management and multiple vpcs per env (we just run stateless apps in k8)

chrismover 7 years ago

Tapping my foot waiting for AWS to match Azure + Google and make EKS free 😄

Erik Osterman (Cloud Posse)over 7 years ago

yea, seriously

Erik Osterman (Cloud Posse)over 7 years ago

it’s really far behind in as it relates to k8s

Erik Osterman (Cloud Posse)over 7 years ago

Would love to see the rancher stuff. I bet we could expand the geodesic story to support the rancher cli.

chrismover 7 years ago

Ranchers pretty neat for what it does (Api proxy / rbac auth); we run the rancher control plane (HA) from our datacenter rather than AWS so its a bit of manual wiring to get the K8 plane setup. But it does talk to EKS which is good (from a compliance front) as it takes away 6 vms we'd otherwise have to manage

chrismover 7 years ago

In an ideal world it would be more terraform friendly

chrismover 7 years ago

as would everything 😄

Erik Osterman (Cloud Posse)over 7 years ago

yea, getting everything in under terraform is nice

Erik Osterman (Cloud Posse)over 7 years ago

in the end though, it’s hard to expect one tool to do the job

chrismover 7 years ago

live the dream 😄

chrismover 7 years ago

We've used it to manage dualDNS providers for a couple of years now; I just hope the 0.12 conversion tool works well

Erik Osterman (Cloud Posse)over 7 years ago

yea, srsly

chrismover 7 years ago

I just see the words breaking change; look at all the repos of terraform and think FOOOOK

chrismover 7 years ago

its either that or tag everything to run with <.12 😄

chrismover 7 years ago(edited)

@Erik Osterman (Cloud Posse) https://github.com/cloudposse/terraform-root-modules/commit/547302316db329f492411bff44aae045bd70e430#diff-b4b339745fca6a54353bf7f392d8dc60 was the naming change intentional?
Terraform hates refactoring 😄


Terraform will perform the following actions:
  - aws_organizations_account.audit
  + module.audit.aws_organizations_account.default

chrismover 7 years ago

breaks out terraform state mv

joshmyersover 7 years ago

Refactoring Terraform is painful.

chrismover 7 years ago

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!
Not half 😄

Erik Osterman (Cloud Posse)over 7 years ago

@Nico You might run into a few bugs with the referrence-architectures

Erik Osterman (Cloud Posse)over 7 years ago(edited)

there are a few people who have recently run through it and can maybe help if you get stuck

Nicoover 7 years ago

ok cool, I'll definitely reach out

Erik Osterman (Cloud Posse)over 7 years ago

(check the open issues)

Nicoover 7 years ago

cool, thank you

Erik Osterman (Cloud Posse)over 7 years ago(edited)

I can also give you (or anyone else) an overview via Zoom

Nicoover 7 years ago

yeah I will definitely need some guidance once I actually start working on the architecture

raehikover 7 years ago

font recs for a stubborn Arch user

raehikover 7 years ago

symbola is awful and emoji fonts don't include the math symbols in geodesic prompt T_T

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/cloudposse/geodesic/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aclosed+prompt

Erik Osterman (Cloud Posse)over 7 years ago

We’ve had a few issues in the past related to the prompt

Erik Osterman (Cloud Posse)over 7 years ago

(fwiw, on OSX and Amazon Linux it’s working for me)

Erik Osterman (Cloud Posse)over 7 years ago(edited)

Can’t lookup the font right now.

Erik Osterman (Cloud Posse)over 7 years ago

If you set PROMPT_STYLE=plain it should use no funky characters

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/cloudposse/geodesic/blob/master/rootfs/etc/profile.d/prompt.sh#L32-L37

raehikover 7 years ago

Cheers. stick with Symbola for now. Surprised that DejaVu doesn't include those chars

chrismover 7 years ago

I keep seeing DEPRECATED: /usr/local/bin/init-terraform is no longer needed. Use tfenv instead.
init-terraform still works (and is still prompted as you change folders), but how is the other supposed to work?

Erik Osterman (Cloud Posse)over 7 years ago(edited)

sec (docs haven’t been updated)

Erik Osterman (Cloud Posse)over 7 years ago

In each project folder in /conf

Erik Osterman (Cloud Posse)over 7 years ago

Add a file like this one:

Erik Osterman (Cloud Posse)over 7 years ago(edited)


# Import the remote module
export TF_CLI_INIT_FROM_MODULE="git::<https://github.com/cloudposse/terraform-root-modules.git//aws/ecs?ref=tags/0.33.0>"
export TF_CLI_PLAN_PARALLELISM=2

use terraform
use tfenv

Erik Osterman (Cloud Posse)over 7 years ago

Optionally, add a Makefile like this one:

Erik Osterman (Cloud Posse)over 7 years ago



## Fetch the remote terraform module
deps:
	terraform init

## Reset this project
reset:
	rm -rf Makefile *.tf .terraform

Trent Schollover 7 years ago

Hey guys - do you generally recommend that people create a new docker image based on the cloudposse geodesic image, or fork the geodesic repo and customize as needed?

Andriy Knysh (Cloud Posse)over 7 years ago

Use geodesic image as base for your own image

Andriy Knysh (Cloud Posse)over 7 years ago

Look at cloudposse/testing.cloudposse.co as an example

Trent Schollover 7 years ago

👍️

Erik Osterman (Cloud Posse)over 7 years ago

(though <http://testing.cloudposse.co|testing.cloudposse.co> is quite out of date)

Erik Osterman (Cloud Posse)over 7 years ago(edited)

The reference-architectures are most current

Erik Osterman (Cloud Posse)over 7 years ago

@Trent Scholl got your PRs

Erik Osterman (Cloud Posse)over 7 years ago

thanks!

Erik Osterman (Cloud Posse)over 7 years ago

I’m going to get that taken carre of

Trent Schollover 7 years ago

🙂

raehikover 7 years ago

Appears that geodesic doesn't have a terminfo file for Termite

raehikover 7 years ago

upon running the geodesic image script:

raehikover 7 years ago


 ✗   (none) ~ ⨠  echo $TERM
xterm-termite
tput: unknown terminal "xterm-termite"
tput: unknown terminal "xterm-termite"
-> Run 'assume-role' to login to AWS
tput: unknown terminal "xterm-termite"
tput: unknown terminal "xterm-termite"
(~4 more times)

raehikover 7 years ago

for anyone else with this problem, run it in tmux or simply export TERM=screen-256color

Erik Osterman (Cloud Posse)over 7 years ago

geodesic is just alpine under the hood

chrismover 7 years ago

and alpine is awesome for docker images 🙂

chrismover 7 years ago

I'm trying to glue a couple of things together but I've a feeling the only way it'll work with the audit/root org setup is if I change the cloudwatch module

by default cloudtrail in root/data etc creates a log-group to match the org namespace (it doesnt set the cloud_watch_logs_role_arn/cloud_watch_logs_group_arn so the resulting log group is null as their's no cloudwatch)

chrismover 7 years ago

adding


module "cloudwatch_log" {
  source            = "git::<https://github.com/cloudposse/terraform-aws-cloudwatch-logs.git?ref=tags/0.2.2>"
  namespace         = "${var.namespace}"
  stage             = "${var.stage}"
  retention_in_days = 180
}

and the cloudwatch role+group arn to the cloudtrail
tries to create
module.cloudwatch_log.aws_iam_role.default
which fails

chrismover 7 years ago

 aws_iam_role.default: Error creating IAM Role testns-root-log-group: MalformedPolicyDocument: Invalid principal in policy: com.amazon.balsa.error.InvalidPolicyException: The passed in policy has a statement with no principals!

chrismover 7 years ago

Remind me to mount the local volume on a folder so I can sync in changes rather than typing make all every few minutes 🤣

joshmyersover 7 years ago

You homedir gets mounted to /localhost in geodesic

giphyover 7 years ago

chrismover 7 years ago

you know I've seen that everytime I open the shell and still didnt click

joshmyersover 7 years ago

For exactly that reason 😄

chrismover 7 years ago

its been a long week 😄

Erik Osterman (Cloud Posse)over 7 years ago(edited)

sorry for our contribution to that!!

Erik Osterman (Cloud Posse)over 7 years ago

haha

Erik Osterman (Cloud Posse)over 7 years ago(edited)

so our (my) dev workflow is to develop in my regular IDE

Erik Osterman (Cloud Posse)over 7 years ago

keep a geodesic shell open to the dev/sandbox account

Erik Osterman (Cloud Posse)over 7 years ago

cd /localhost/...../project

Erik Osterman (Cloud Posse)over 7 years ago

and iterate that way

chrismover 7 years ago

that makes perfect sense 😄

chrismover 7 years ago

not randomly typing vi and duplicating stuff backwards like a pleb

Erik Osterman (Cloud Posse)over 7 years ago

yea, and forgetting that the one thing that you actually made in the shell manually

chrismover 7 years ago

oh that never happens whistles innocently

Erik Osterman (Cloud Posse)over 7 years ago

here’s something else that’s cool:

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/cloudposse/geodesic/issues/204

Erik Osterman (Cloud Posse)over 7 years ago

I haven’t formalized a workflow around it, but I did a POC

Erik Osterman (Cloud Posse)over 7 years ago

we can actually use geodesic with minikube

chrismover 7 years ago

neat; we all run rancher locally / docker for desktop minikubes handy for quick pocs

chrismover 7 years ago

ahh localhost mounts to the user folder. Still need to map 🙂 all my work lives on a separate ssd.

Erik Osterman (Cloud Posse)over 7 years ago

ohhh

Erik Osterman (Cloud Posse)over 7 years ago

look at /usr/local/bin/<script>

Erik Osterman (Cloud Posse)over 7 years ago

I think you can override it

Erik Osterman (Cloud Posse)over 7 years ago

if not, we would accept PR to support overriding it

chrismover 7 years ago

cool I'll have a dig as I assume the home paths used for aws-vault

Erik Osterman (Cloud Posse)over 7 years ago

that too

chrismover 7 years ago

its easy enough to get docker to dance 🙂

Erik Osterman (Cloud Posse)over 7 years ago

and a local “Cache”

chrismover 7 years ago

windows subsystem; where home is home but not quite home 😄

Erik Osterman (Cloud Posse)over 7 years ago

are you using WSL?

chrismover 7 years ago

yep

chrismover 7 years ago

ubuntu 18

Erik Osterman (Cloud Posse)over 7 years ago

yea, that was A MAJOR pain

Erik Osterman (Cloud Posse)over 7 years ago

it took us a few PRs to get that right

chrismover 7 years ago

works REALLY well so you did a good job there 😄

chrismover 7 years ago

Think I got the cloudwatch issue down. I already had it scripted up before splitting all the orgs so I've made it sorta hybrid for the time being. https://gist.github.com/ChrisMcKee/0ca78c207fa7c3aca3b973c824aab069

chrismover 7 years ago

Now to wait for the SIEM monitoring company to start screaming... probably shouldnt have changed it all on a friday 🤣

Erik Osterman (Cloud Posse)over 7 years ago(edited)

what company do you use? (curious)

chrismover 7 years ago

dm'd 😄

joshmyersover 7 years ago

Hopefully not AlienVault 😂

joshmyersover 7 years ago

I remember doing some due diligence a while back and found private keys in their AMIs

chrismover 7 years ago

lol cough

chrismover 7 years ago

we use all the aws stuff as well as the managed service but it all feeds into the one place for monitoring as well as the aws notifications. Much as I'd love to trust a single service

chrismover 7 years ago

There's a distinct lack of silver bullets 😄

lorenover 7 years ago

any good guides on using docker in WSL? i was looking into it a while ago, but at the time it seemed either/or not both

chrismover 7 years ago

@loren https://nickjanetakis.com/blog/setting-up-docker-for-windows-and-wsl-to-work-flawlessly

chrismover 7 years ago

Works fine

lorenover 7 years ago(edited)

ahh, now i remember the problem, i need to be able to use virtualbox locally (for packer, nothing persistent), and installing docker for windows enables hyper-v, which breaks virtualbox

Erik Osterman (Cloud Posse)over 7 years ago

ohhhh yea, I've had similar issues when trying to use other VMs

lorenover 7 years ago

Maybe I can get terraform to launch i3 metal instances to run virtualbox and packer for me 🤣

chrismover 7 years ago

Packer runs fine on windows though , you can run it in docker as well. Very few reasons for vbox thesedays

lorenover 7 years ago

i understand that, it's really a vagrant box that packer is creating, with a virtualbox provider

lorenover 7 years ago

of course packer runs fine itself on windows, but packer launches virtualbox locally to create the image/ovf, and then converts it to a vagrant box

lorenover 7 years ago(edited)

so, somehow or other, i need packer to be able to run virtualbox

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/cloudposse/geodesic/releases/tag/0.68.0

Erik Osterman (Cloud Posse)over 7 years ago

We've added okta support

December 2018 Browse by date February 2019