#geodesic - April 2019

There is 1 event this week

Hello,

I'm rebuilding some packages of the cloudposse/packages container and getting this error: mktemp: failed to create directory via template ‘../../tmp/tmp.XXXXXXXXXX’: No such file or directory

I could observe that the problem is related to this file included at makefile: "/packages/tasks/Makefile.apk"

...
export APK_TMP_DIR := $(realpath $(shell mktemp -p ../../tmp -d))
...

As I wasn't interested in apk packages I've just erased this file: echo > /packages/tasks/Makefile.apk to not get errors during the build process, but I thing that It worth's be mentioned this problem,
just to check if I doing something wrong



/packages # echo "1.14.0" > /packages/vendor/kubectl/VERSION
/packages # make -C install kubectl
make: Entering directory '/packages/install'
make[1]: Entering directory '/packages/vendor/kubectl'
mktemp: failed to create directory via template ‘../../tmp/tmp.XXXXXXXXXX’: No such file or directory
curl --retry 3 --retry-delay 5 --fail -sSL -o /packages/bin/kubectl https://storage.googleapis.com/kubernetes-release/release/v1.14.0
/bin/linux/amd64/kubectl && chmod +x /packages/bin/kubectl
make[1]: Leaving directory '/packages/vendor/kubectl'
make: Leaving directory '/packages/install'

Thanks for reporting

@Pablo Costa can you open an issue for that?

I'll have @Maxim Mironenko (Cloud Posse) look into it

for now, maybe use an earlier version of cloudposse/packages?

(btw, if you're using alpine, then we suggest using apk add instead of the make based installer)

Thanks Erik, btw how is the policy for package updates ? For example helm version is at 2.11 (avaliable 2.13) terraform is 0.11.11 (avaliable 0.11.13)

oh, open policy!

open a PR and we'll approve it usually with in a couple of hours

only reason our versions lag is (we or you) didn't submit a PR to update it

since we can version pin everything, we're very open to keeping things current.

Here are the slides from the "Los Angeles Kubernetes Meetup" where we presented on Geodesic.

I'm going to hang out in this zoom for a little bit in case anyone has any questions.

(fwiw, we had daylight savings in the US on March 10)

I am not sure :-)

hey hey

danyone know why kops is still 1.10.* in geodesic?

rather than 1.11.0

There is 1 event this week

So if I wanted to get starting trying to use geodesic where is the best place to start. I see lots of information about and references to geodesic and reference architectures, just wonder if there is a certain order to follow. Any info would be helpful, thanks

i've noticed the build-harness has a lot of cloudposse references... is forking this repo also recommended when getting setup with geodesic? it seems to be a dependency and affects commands like readme generation and such.

@Josh Larsen yes/no - I think it's a great idea to fork it for your own org's needs

that said, there's no easy way to take advantage of your fork in our repos 😃

If you add sops to the geodesic
RUN wget <https://github.com/mozilla/sops/releases/download/3.2.0/sops-3.2.0.linux> -O /usr/bin/sops && chmod +x /usr/bin/sops
and set the env vars

RUN curl <https://keybase.io/user/pgp_keys.asc> | gpg --import 
ENV SOPS_KMS_ARN="arn:aws:kms:xx-xxx-x:xxxx:key/xxx-xxx-xxx-xxx-xxxx"
ENV SOPS_PGP_FP="APGPKEY,ANOTHERPGPKEY,ETC"

You can use sops encryption before pushing files into storage so they're only un-encrypted within the container during use

Question I forgot I had yesterday: I know that Geodesic opens a port (range?) for handy kubectl port-forwards. How do I use it, and what version was it released in? (in the last month, or older?)

Yes, that's supported - it's been in the container for years

there's an env inside the container that contains the port number

That said, I think we should rename the port to be more generic

so it can be repurposed/overloaded

Thanks, I was looking for $GEODESIC*

Here's how we used geodesic with Codefresh to achieve GitOps with terraform on Codefresh

Here's what it looks like https://github.com/cloudposse/testing.cloudposse.co/pull/75

how does geodesic handle issues of uid/gid when mounting dirs from the host?

could someone point me to the code where that happens?

It does nothing with uid/gid mapping

Geodesic installs a wrapper script to run the container, which launches the container with something like (but not exactly, and with other options) docker run -it --privileged --volume=$HOME=/localhost. The shell inside the Geodesic container is run as root. The permissions mapping is handled by Docker. I use docker-machine on my Mac, so everything Docker runs runs as me and the root user inside the Docker container has the same permissions as I do on the host. Files created on the host from Geodesic are owned by me and files I cannot read cannot be read from inside Geodesic.

I think the behavior is slightly different on linux, where UID/GID within the container are preserved on the host machine, but what @Jeremy G (Cloud Posse) describes is correct on OSX.

ah ok, that explains what i'm seeing

on macos it just works, but on our linux workstations the host mounted files are written with root:root

I was hoping they supported uid/gid mapping in bind-mount (by now), but it's not supported

we could technically drop to a non-privileged user in geodesic, but haven't optimized for that.

There are no events this week

Hello, everyone. I am trying to follow the cold-start procedure described here https://docs.cloudposse.com/reference-architectures/cold-start/ but it seems that it is outdated, right now I have an issues trying to create users, could anybody help me?

@Eugene Korekin sorry for the troubles!

yes, the cold-start docs are quite out of date and refer to a older implementation.

our current process is being kept up to date here: https://github.com/cloudposse/reference-architectures

Also, please see https://archive.sweetops.com/geodesic/#5097fc92-7270-4b16-960a-44a9c18a953a

thanks, Eric! I'll take a look

@Eric Johnson could you please tell me what would be the best approach if I already created the master aws account in aws orgs? I think I can't remove it

@Eugene Korekin

I used the older implementation to create it

soooo you have a couple of options.

did you create those AWS accounts by hand or using terraform?

I used the existing account as the base for the procedure described in the old cold start, the master account referring to the existing account was created in aws orgs in the process

ok, so option (1) is to continue going down the path you were. there's nothing wrong with that per say.

so the master account was created via terraform, but it already has some ec2 instances etc

so there's no one way to bring up this infrastructure. the idea is the geodeisc is just a run time environment.

what you do inside of it is entirely open-ended.

I stuck on that path, as the user creation step requires some contents present in ssm and I cannot find how to create this content with the old procedure

hrmmm so mostly likely, the user creation stuff will then need to use older versions of the modules that pre-date the SSM dependencies

i can't say what version that would be.

to start fresh though, would involve the following:

1. password reset on each account

here is the error I have

2. login to each account, accept t&c

3. terraform destroy accounts

---

then you have a clean base

is there a way to somehow import existing accounts?

they are already in use

you can import existing accounts, however @Jeremy G (Cloud Posse) recently went through this with one of our current clients and if every parameter doesn't match 100% it wants to recreate them

(jumping on a ☎️)

I see, is there a way to only create a new account (let's say 'testing') and leave the master one as it is (including all the existing users)?

yes, you can probabblby do that

use accounts_enabled flag to only select testing

e.g. get your feet wet with the system

there's a lot of moving pieces, so getting your hands dirty with one account would be a good idea

so, do I just need to skip 'make root' and start right from 'make children'?

I've just tried it, and it doesn't seem to work, the make command doesn't provide any output

You need to set up your configs/root.tfvars with all the right configuration (hopefully it's reasonably self-explanatory) including accounts_enabled limited to the accounts you want to create. Then set yourself up with AWS admin credentials in your "root" AWS account and run make root. This creates the children accounts and sets up roles and network CIDRs and so forth and creates a bootstrap user you will use for the rest of the configuration.

but in won't change anything inside the root account in the process, right?

It will definitely change things inside the root account. The root account is where your users are created and roles for the children account are created and DNS entries for children DNS subdomains are created.

I don't want to change any of the existing users, won't it be possible to proceed without that?

in other words, would it be possible to use the existing user entries in the master account and reuse then in the children ones?

so our reference architectures don't provide that level of configurability b/c the number of permutations is insurmountable

however, all of our terraform modules are compose of other modules

you can pick and choose exactly what you want to use

Office Hours
Today from 11:30 AM to 12:20 PM at https://zoom.us/j/684901853

(PST)

There are no events this week

Join us for "Office Hours" every Wednesday 11:30AM (PST, GMT-7).

This is an opportunity to ask us questions about INLINECODE_0, get live demos and learn from others using it. Next one is Mar 20, 2019 11:30AM.

👉️ Add it to your calendar

https://zoom.us/j/684901853

#office-hours (our channel)

question: let's say my terraform-root-modules is a private repo... how can i get my github keys into geodesic so the terraform init will be able to retrieve the modules?

@Josh Larsen you have a few options

are you running this under CI/CD or as a human?

here's how we do it with #codefresh https://github.com/cloudposse/testing.cloudposse.co/blob/master/codefresh/terraform/pipeline.yml#L99-L103

then we add the ssh public key as a deploy key to the root modules repo

ideally i'd like to do both

but let's start with human for now

i'm thinking something like cat /localhost/.ssh/id_rsa | ssh-add - would do it, but would just have to do that every time i started the shell. or is there a better way?

are you on a mac or linux?

on linux, we mount your ssh agent socket into the container

we can't do that on a mac

the other option is to store your ssh key in SSM

i'm on mac... but i think i get the idea. thanks.

I was experimenting today with terragrunt vs vanilla terraform init and wanted to point out a few important differences between these two similar methods:
- terragrunt { source = "<blueprint source>" } and
- terraform init -from-module=<blueprint source>

terragrunt allows the use of:
- override.tf [1] files in the CWD
- "mix-in" files in the CWD (*.tf files that do not match blueprint filenames), useful for adding one-off resources to a blueprint
- "upstage" files in the CWD (*.tf files that do match a blueprint filename), these replace the contents of a blueprint source file of the same name), useful for removing blueprint resources
This is due to the fact that terragrunt init creates a tmp dir, clones the SOURCE to the tmp dir, and then copies/overwrites (aka "upstages") all files in the CWD to the tmp dir (overwriting any duplicates).

Whereas terraform init -from-module= requires that the CWD contains zero files matching *.tf or *.tf.json. This prevents all the above techniques: overrides, mix-ins, and upstage files.

Has anyone thought about how to support/implement the override, mix-in and upstage patterns without the use of terragrunt?

[1] https://www.terraform.io/docs/configuration-0-11/override.html

Hi- if I have started the "quick start" with example.io, had an issue and ran make clean (after terraform had created some things), how can I most easily "reset"?

I want to run terraform destroy essentially

I am getting, for example-

* module.staging.aws_organizations_account.default: 1 error(s) occurred:

* aws_organizations_account.default: Error creating account: AccessDeniedException: You don't have permissions to access this resource.
	status code: 400, request id: 5b93365b-67b0-11e9-808b-3fc6f3a60880
* module.dev.aws_organizations_account.default: 1 error(s) occurred:

* aws_organizations_account.default: Error creating account: AccessDeniedException: You don't have permissions to access this resource.
	status code: 400, request id: 5b9335cd-67b0-11e9-b64d-9b693972455f
* module.prod.aws_organizations_account.default: 1 error(s) occurred:

* aws_organizations_account.default: Error creating account: AccessDeniedException: You don't have permissions to access this resource.
	status code: 400, request id: 5b92998f-67b0-11e9-b23d-dd34fe4bf6c3
* module.audit.aws_organizations_account.default: 1 error(s) occurred:

* aws_organizations_account.default: Error creating account: AccessDeniedException: You don't have permissions to access this resource.
	status code: 400, request id: 5b93362c-67b0-11e9-a70f-ffba4d75ead7

@Mike Pfaffroth recommend to start here instead: https://github.com/cloudposse/reference-architectures

@Erik Osterman (Cloud Posse) I am running that (I’m in the make root stage)

so terraform destroy will fail when trying to delete the accounts

you can destroy the accounts after jumping through a lot of hoops (password reset, login, accept t&c, etc - for each child account)

right- I guess what I was trying to say is- is that process documented anywhere?

I know I’m in a bad position, I am trying to find a similar checklist

aha! yes, good question

no, we have not documented it. @Jeremy G (Cloud Posse) would it be relatively quick for you to open an issue against the reference-architectures that describes the process you took? It's been a long enough time that I can't recall every step.

i just remember doing the password reset on every child account so that I could log in

then i think there was some kind of terms & conditions I had to accept (checkbox/click)

and after that, I think it's sufficient enough to retry the terraform destroy

that said, I don't think it's possible to reuse the same email address on round 2

@oscarsullivan_old or @Jan or @Josh Larsen might have some more recent recollection

@Erik Osterman (Cloud Posse) I did not delete any accounts. Not only are there lots of hoops to go through, but also deleting accounts take a long time (90 days, I think) and the account email address remains permanently associated with the "deleted" account, and therefore cannot be used on a new account.

@Mike Pfaffroth What I recommend is to reuse the created accounts. Run make root up to the point where terraform complains that it cannot create the accounts, then import the accounts into Terraform using terraform import.

interesting… I will try that approach. Appreciate the tip.

hm… even on a brand new account I am not able to create these- it always dies when trying to create the others:

* aws_organizations_account.default: Error creating account: AccessDeniedException: You don't have permissions to access this resource.
	status code: 400, request id: 7de7c003-6846-11e9-9002-752f3d4639b5

I changed the email address as well, just to make sure it wasn’t trying to create ones that were already there

I am the root account:

➜  reference-architectures git:(master) ✗ aws sts get-caller-identity
{
    "UserId": "1redacted7",
    "Account": "1redacted7",
    "Arn": "arn:aws:iam::1redacted7:root"
}

is there a special permissions I need? or is there a documentation process for signing up for an account in the right way within an organization?

There are no events this week

Join us for "Office Hours" every Wednesday 11:30AM (PST, GMT-7).

This is an opportunity to ask us questions about INLINECODE_0, get live demos and learn from others using it. Next one is Mar 20, 2019 11:30AM.

👉️ Add it to your calendar

https://zoom.us/j/684901853

#office-hours (our channel)