#geodesic - March 2019 | Slack Archive

geodesic

335 messages

Discussions related to https://github.com/cloudposse/geodesic

Archive: https://archive.sweetops.com/geodesic/

Feb 2019

March 2019

Apr 2019

Erik Osterman (Cloud Posse)over 7 years ago

Are you referring to S3 backends?

Erik Osterman (Cloud Posse)over 7 years ago

aha yes

Erik Osterman (Cloud Posse)over 7 years ago

so this is solved with environment variables

Erik Osterman (Cloud Posse)over 7 years ago(edited)

we use direnv to define them for terraform consumption

Erik Osterman (Cloud Posse)over 7 years ago

so in a project, you'll define a .envrc

Erik Osterman (Cloud Posse)over 7 years ago

and in the Dockerfile you'll define some globals

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/cloudposse/testing.cloudposse.co/blob/master/Dockerfile#L30-L33

Erik Osterman (Cloud Posse)over 7 years ago

Here's a sample .envrc

Erik Osterman (Cloud Posse)over 7 years ago(edited)

https://github.com/cloudposse/testing.cloudposse.co/blob/master/conf/chamber/.envrc

Erik Osterman (Cloud Posse)over 7 years ago(edited)

use terraform + use tfenv setups up the TF_CLI_ARGS_init

Erik Osterman (Cloud Posse)over 7 years ago

using the the cloudposse direnv stdlib is totally optional.

Erik Osterman (Cloud Posse)over 7 years ago

the direnv stdlib is defined here: https://github.com/cloudposse/geodesic/tree/master/rootfs/etc/direnv/rc.d

Erik Osterman (Cloud Posse)over 7 years ago

note, the use terraform helper will do this:

Erik Osterman (Cloud Posse)over 7 years ago


export TF_BUCKET_PREFIX=${TF_BUCKET_PREFIX:-$(basename $(pwd))}

Erik Osterman (Cloud Posse)over 7 years ago

this is so that each of your project folders gets it's own state folder inside the state bucket

Erik Osterman (Cloud Posse)over 7 years ago

by default it uses the current folder name

lorenover 7 years ago

Long as it's consistent throughout a project, life is good! Thank you editorconfig!

lorenover 7 years ago

If it's your project, that's your prerogative 😋

lorenover 7 years ago

I like to lint for editorconfig violations in CI, for those few editors that don't honor its settings by default, https://github.com/jedmao/eclint

chrismover 7 years ago

What OS was https://github.com/cloudposse/github-authorized-keys tested on (im assuming in aws)?

Erik Osterman (Cloud Posse)over 7 years ago

It was tested on quite a few Linux distos

Erik Osterman (Cloud Posse)over 7 years ago

However the problem is usually that useradd varies by distro so you always need to update the template

Erik Osterman (Cloud Posse)over 7 years ago

It defaults to alpine

chrismover 7 years ago

I was running it on ub18 in docker (just following the readme tbh)
Not wholly sure why it failed tbh, said it couldnt talk to github which seems self explanatory; I was just wondering if the pathing / commands its expecting to execute on the host are correct

Erik Osterman (Cloud Posse)over 7 years ago

can you share the error @chrism

Erik Osterman (Cloud Posse)over 7 years ago

devil is in the details 😃

chrismover 7 years ago

yep 1 mo

chrismover 7 years ago

the instructions seem wonky as well; --expose is an override of EXPOSE in docker but the docs show it like port -p 301:301

chrismover 7 years ago


{"level":"info","msg":"Run syncUsers job on start","time":"2019-03-06T17:06:42Z"}
{"job":"syncUsers","level":"error","msg":"Connection to <http://github.com|github.com> failed","subsystem":"jobs","time":"2019-03-06T17:07:12Z"}
{"level":"info","msg":"Run ssh integration job on start","time":"2019-03-06T17:07:12Z"}
{"job":"sshIntegrate","level":"info","msg":"Ensure file /usr/bin/github-authorized-keys","subsystem":"jobs","time":"2019-03-06T17:07:12Z"}
{"job":"sshIntegrate","level":"info","msg":"Ensure exec mode for file /usr/bin/github-authorized-keys","subsystem":"jobs","time":"2019-03-06T17:07:12Z"}
{"job":"sshIntegrate","level":"info","msg":"Ensure AuthorizedKeysCommand line in sshd_config","subsystem":"jobs","time":"2019-03-06T17:07:12Z"}
{"job":"sshIntegrate","level":"info","msg":"Ensure AuthorizedKeysCommandUser line in sshd_config","subsystem":"jobs","time":"2019-03-06T17:07:12Z"}
{"job":"sshIntegrate","level":"info","msg":"Restart ssh","subsystem":"jobs","time":"2019-03-06T17:07:12Z"}
{"job":"sshIntegrate","level":"info","msg":"Output: ","subsystem":"jobs","time":"2019-03-06T17:07:12Z"}
{"job":"sshIntegrate","level":"error","msg":"Error: fork/exec /usr/sbin/service: no such file or directory","subsystem":"jobs","time":"2019-03-06T17:07:12Z"}
{"level":"info","msg":"Start jobs scheduler","time":"2019-03-06T17:07:12Z"}
[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
 - using env:   export GIN_MODE=release
 - using code:  gin.SetMode(gin.ReleaseMode)

[GIN-debug] GET    /user/:name/authorized_keys --> <http://github.com/cloudposse/github-authorized-keys/server.Run.func1|github.com/cloudposse/github-authorized-keys/server.Run.func1> (3 handlers)
[GIN-debug] Listening and serving HTTP on :301

chrismover 7 years ago(edited)


docker run   -v /:/host   --expose "301"   -p 127.0.0.1:301:301  -e GITHUB_API_TOKEN=x   -e GITHUB_ORGANIZATION=x
  -e GITHUB_TEAM=ops   -e SYNC_USERS_INTERVAL=200   -e LISTEN=:301   -e INTEGRATE_SSH=true        cloudposse/github-authorized-keys

Erik Osterman (Cloud Posse)over 7 years ago

{"job":"sshIntegrate","level":"error","msg":"Error: fork/exec /usr/sbin/service: no such file or directory","subsystem":"jobs","time":"2019-03-06T17:07:12Z"}

Erik Osterman (Cloud Posse)over 7 years ago

this is assuming a systemd setup

Erik Osterman (Cloud Posse)over 7 years ago

sounds like you don’t have systemd

Erik Osterman (Cloud Posse)over 7 years ago

try INTEGRATE_SSH=false

Erik Osterman (Cloud Posse)over 7 years ago

I want to incorporate this in our docs.

erikover 7 years ago

Erik Osterman (Cloud Posse)over 7 years ago

even explaining this is a big win as a way to show how the abstraction works.

Erik Osterman (Cloud Posse)over 7 years ago

Ideally if you open that as a PR we can review it and leave comments

Erik Osterman (Cloud Posse)over 7 years ago

Hey @oscarsullivan_old sorry been so focused on the conference. Back to life as normal next week.

Erik Osterman (Cloud Posse)over 7 years ago

If I find some time today will take a look.

Erik Osterman (Cloud Posse)over 7 years ago

will reach out this afternoon

Erik Osterman (Cloud Posse)over 7 years ago

in the middle of a meeting atm

Alex Siegmanover 7 years ago

@Erik Osterman (Cloud Posse) I think I found the docs you're referring to here: https://github.com/osulli/geodesic-getting-started If those are they, I'll read through this and ping back if I have any issues

Alex Siegmanover 7 years ago

Well, I'm working through making a root account, which it looks like you might have skipped?

Alex Siegmanover 7 years ago

Ah, yeah, my company is at the point we want to build out something similar to the ref architecture, so I was just going to spin it up quick and see how these tools handle it, and maybe just use them

Alex Siegmanover 7 years ago

So, we also have an existing account I could use as our root account, and that might make the most sense, but I wanted to test this on a blank canvas to be sure I didn't trash anything there.

Alex Siegmanover 7 years ago

Exactly. I'm just thinking forward, because eventually once I get these k8s clusters built inside the proper account arch, I'm going to have to deal with moving data

Alex Siegmanover 7 years ago

and that's not going to be a small project

Alex Siegmanover 7 years ago

Yeah, that's my concern also

Alex Siegmanover 7 years ago

But I just started looking a couple hours ago, not really a fair shake :)

Erik Osterman (Cloud Posse)over 7 years ago

@oscarsullivan_old are you around?

Erik Osterman (Cloud Posse)over 7 years ago

I can help jump on a zoom quickly and see if I can get you unstuck

Erik Osterman (Cloud Posse)over 7 years ago

ah bummer

Erik Osterman (Cloud Posse)over 7 years ago

stick those in rootfs/etc/profile.d/mycompanyname.sh

Erik Osterman (Cloud Posse)over 7 years ago

or some filename like that

Erik Osterman (Cloud Posse)over 7 years ago

profile.d is loaded by the shell when you start up

raehikover 7 years ago

Hey all, question that's been on my mind. AFAIK Geodesic sets remote state S3 key according to the directory basename/leaf directory

raehikover 7 years ago

has the possibility of using the whole directory come up? e.g. (/conf/)cluster1/{vpc,kops}, (/conf/)cluster2/{vpc,kops}

Erik Osterman (Cloud Posse)over 7 years ago

It would be nice to have to generate the path w.r.t. /conf

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/cloudposse/geodesic/blob/master/rootfs/etc/direnv/rc.d/terraform#L16

Erik Osterman (Cloud Posse)over 7 years ago

we generate it here

Thibaultover 7 years ago

since switching to .envrc files, I'm having the same issue

raehikover 7 years ago

@Erik Osterman (Cloud Posse) yeah it's a really easy change, just curious if it came up

Erik Osterman (Cloud Posse)over 7 years ago

just lack of time

Erik Osterman (Cloud Posse)over 7 years ago

😃

Thibaultover 7 years ago

aka the bucket prefix doesn't seem to be respected

raehikover 7 years ago

rc.d/terraform :


pwd_tmp=$(pwd)
export TF_BUCKET_PREFIX=${TF_BUCKET_PREFIX:-${pwd_tmp:1}}

raehikover 7 years ago

(or so. actually we need to remove /conf, oops)

Erik Osterman (Cloud Posse)over 7 years ago


export TF_BUCKET_PREFIX=${TF_BUCKET_PREFIX:-$(pwd | cut -d/ -f2-)}

Erik Osterman (Cloud Posse)over 7 years ago

this will strip off the first part (/conf)

raehikover 7 years ago

are you sure it's not -f3-

raehikover 7 years ago

that works perfect on my Geodesic and local

Erik Osterman (Cloud Posse)over 7 years ago

state folders should be relate to conf

Erik Osterman (Cloud Posse)over 7 years ago(edited)

e.g. ordinarily, we would have /conf/vpc, so the state bucket folder should be vpc

Erik Osterman (Cloud Posse)over 7 years ago

or /conf/us-west-2/vpc should be us-west-2/vpc

raehikover 7 years ago

right, would be interested in having it as an option

Erik Osterman (Cloud Posse)over 7 years ago(edited)

where s3://$BUCKET/$PREFIX/terraform.tfstate

Erik Osterman (Cloud Posse)over 7 years ago(edited)

@raehik if you want to open a PR for that change we’ll promptly review

https://github.com/cloudposse/geodesic/blob/master/rootfs/etc/direnv/rc.d/terraform#L16

raehikover 7 years ago

it's a possible breaking change, right? so how would I make it an option, check for ${TF_USE_FULL_PWD}?

Erik Osterman (Cloud Posse)over 7 years ago

yea, it’s possibly breaking, though I suspect anyone who was using subfolders would have already encountered this problem.

Erik Osterman (Cloud Posse)over 7 years ago

We can add a flag. I’d default it to the new format.

Erik Osterman (Cloud Posse)over 7 years ago

maybe we have TF_USE_CWD vs TF_USE_PWD

Erik Osterman (Cloud Posse)over 7 years ago

CWD = current working directory

Erik Osterman (Cloud Posse)over 7 years ago

Why a new var?

raehikover 7 years ago

PWD = present working directory?

Erik Osterman (Cloud Posse)over 7 years ago

raehikover 7 years ago

*print working directory, my b

Erik Osterman (Cloud Posse)over 7 years ago

so for clarification, we’re talking about defaulting TF_BUCKET_PREFIX

Erik Osterman (Cloud Posse)over 7 years ago

to something more accurate

Erik Osterman (Cloud Posse)over 7 years ago

ok, how about this

Erik Osterman (Cloud Posse)over 7 years ago

TF_BUCKET_PREFIX_FORMAT

Erik Osterman (Cloud Posse)over 7 years ago

and we can have one format be pwd and other format be basename or basename-pwd or something like that?

raehikover 7 years ago

that makes sense

raehikover 7 years ago

more descriptive than just a flag

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/cloudposse/geodesic/blob/master/rootfs/etc/direnv/rc.d/terraform#L25

Erik Osterman (Cloud Posse)over 7 years ago

that’s where it’s set

Erik Osterman (Cloud Posse)over 7 years ago

export TF_CLI_INIT_BACKEND_CONFIG_KEY="${TF_BUCKET_PREFIX}/${TF_STATE_FILE}"

Erik Osterman (Cloud Posse)over 7 years ago

@oscarsullivan_old and @raehik this is fixed now for your guys right? https://github.com/cloudposse/reference-architectures/issues/13

Josh Larsenover 7 years ago

so, are <http://root.cloudposse.co|root.cloudposse.co> and <http://test.cloudposse.co|test.cloudposse.co> repos just examples or outdated in favor of the reference-arcitechtures repo?

Erik Osterman (Cloud Posse)over 7 years ago

the *.<http://cloudposse.co|cloudposse.co> are examples of how we use geodesic; these are what we use for presentations and demos

Erik Osterman (Cloud Posse)over 7 years ago

the reference-architectures is a first stab at automating that process but makes a very strong assumption: start with a virgin AWS root account

Erik Osterman (Cloud Posse)over 7 years ago

what we implement for our customers tends to be a little bit ahead of what we have in *.<http://cloudposse.co|cloudposse.co> just b/c we don’t have the time to keep them updated

Josh Larsenover 7 years ago

so for a cold start you would recommend reference-architecture over using the examples?

Erik Osterman (Cloud Posse)over 7 years ago

it somewhat depends on your existing level of experience with AWS, docker, multi-account architectures, terraform… and also if you’re going to use k8s.

Josh Larsenover 7 years ago

ok assuming a good understanding and experience in all of those + planning to use k8s

Erik Osterman (Cloud Posse)over 7 years ago

Then I suggest kicking the tires for reference architectures. Before you start, have a look at the open issues so you can be prepared of common issues.

Erik Osterman (Cloud Posse)over 7 years ago

I can also give you a review over zoom

Erik Osterman (Cloud Posse)over 7 years ago

https://calendly.com/cloudposse

Erik Osterman (Cloud Posse)over 7 years ago

the first step is to request an increase in the number of AWS accounts

Josh Larsenover 7 years ago

yup

Josh Larsenover 7 years ago

ok thanks, i'll hit your calendar 🙂

Janover 7 years ago

hey hey

Janover 7 years ago

how is terraform module dependancy / execution order dealt with in geodesic ?

Janover 7 years ago

having /conf/tfstate-backend /conf/vpc /conf/foo /conf/bar (other than knowing that tfstate needs to be fiorst then vpc then others)

Janover 7 years ago

thinking about it more from a pipeline point of view

chrismover 7 years ago

Pretty nice concept for RKE https://github.com/yamamoto-febc/terraform-provider-rke

erikover 7 years ago

Erik Osterman (Cloud Posse)over 7 years ago

works for me 😃

Erik Osterman (Cloud Posse)over 7 years ago

If it’s out of date, submit PR here: https://github.com/cloudposse/packages/tree/master/vendor/packer

mmuehlbergerover 7 years ago

@Erik Osterman (Cloud Posse) Regarding terraform-root-modules#132 (chamber dependencies): I’ve forked the repo already to adapt it to my needs, I just wanted to point it out. Do you want me to continue to document issues like this for future reference or is that just clutter for you?

Erik Osterman (Cloud Posse)over 7 years ago

we cannot ship everything in the base image b/c it gets toooooooo big

Erik Osterman (Cloud Posse)over 7 years ago

that’s why we have the cloudposse/packages distribution

Erik Osterman (Cloud Posse)over 7 years ago(edited)

just run: RUN apk add --update packer@cloudposse to your Dockerfile

Josh Larsenover 7 years ago

@oscarsullivan_old oh you are correct... i got stonewalled by tech support saying my account was too new to increase that limit. i guess this is a new policy now. he wouldn't even give me a solid timeframe at all... just said the account has to be around for a "few weeks" before they will up the limit.

Erik Osterman (Cloud Posse)over 7 years ago

WOW

Erik Osterman (Cloud Posse)over 7 years ago

that is lame.

Josh Larsenover 7 years ago

he said it was a new policy

Thibaultover 7 years ago

could be due to the amount of people trying Cloudposse's ref arch?

Erik Osterman (Cloud Posse)over 7 years ago

haha

mmuehlbergerover 7 years ago

So basically, you need to have some spare accounts lying around to “be prepared”. 😄

Erik Osterman (Cloud Posse)over 7 years ago

or people like @Dombo who I think requested account limits of 1000

Erik Osterman (Cloud Posse)over 7 years ago

😃

Josh Larsenover 7 years ago

here is the reply from tech support.. you might add this into the cold start docs.

Erik Osterman (Cloud Posse)over 7 years ago

thanks @Josh Larsen! will open an issue incase it helps others

Alex Siegmanover 7 years ago

Is there a way to easily "unmake" the root account bootstrapping to test from a fresh place on the reference architectures? I feel like I should be able to do a terraform destroy in here somewhere and accomplish that, but not seeing built in make targets to make it easy

Erik Osterman (Cloud Posse)over 7 years ago

so, it depends.

Erik Osterman (Cloud Posse)over 7 years ago

basically, the problem is AWS makes it programatically impossible to destroy accounts created using the API without first logging in and accepting T&Cs

Erik Osterman (Cloud Posse)over 7 years ago

this is a big reason we haven't yet tried to tackle automated e2e testing

Erik Osterman (Cloud Posse)over 7 years ago

so since it means we'd need to start from point X, it's not clear where point X is

Alex Siegmanover 7 years ago

Well, with the dynamodb problems AWS had this afternoon, the bootstrapping on my root account broke

Alex Siegmanover 7 years ago(edited)

So I was just hoping I could "terraform destroy" the right way

Erik Osterman (Cloud Posse)over 7 years ago

unfortunately no make target that implements it right now

Alex Siegmanover 7 years ago

I can fix the state mismatch by hand

Alex Siegmanover 7 years ago

Was just curious 😃

Erik Osterman (Cloud Posse)over 7 years ago

but we've been talking more about this internally

Erik Osterman (Cloud Posse)over 7 years ago

ultimately, we want it to be as easy as docker-compose up

Alex Siegmanover 7 years ago

It'd be nice if we could clear out an account with a terraform destroy, even if it doesn't kill the account itself, just for testing purposes.

Erik Osterman (Cloud Posse)over 7 years ago

ya, thought this is where it gets complicated

Erik Osterman (Cloud Posse)over 7 years ago

b/c one of the steps is to create the accounts

Erik Osterman (Cloud Posse)over 7 years ago

so now to bring it back up it needs to skip that

Erik Osterman (Cloud Posse)over 7 years ago

tf is not good at skipping

Alex Siegmanover 7 years ago

tf certainly is not. Where is the state stored for the account that is brought up? in the root account? or in the subaccount?

Erik Osterman (Cloud Posse)over 7 years ago

not if 99% of everything is done using local-exec

Erik Osterman (Cloud Posse)over 7 years ago(edited)

we're using terraform to setup the reference architectures, but perhaps it was not the ideal tool for the job. we're basically using terraform to generate terraform code.

Erik Osterman (Cloud Posse)over 7 years ago

then executing that terraform code inside of docker.

Erik Osterman (Cloud Posse)over 7 years ago(edited)

because we're using local-exec a lot to call docker, there's no easy way to recover state. terraform cannot compute an accurate plan.

lorenover 7 years ago

There's also cloudnuke... Don't need to use terraform if you really want to just destroy everything

Erik Osterman (Cloud Posse)over 7 years ago

hahaha

Erik Osterman (Cloud Posse)over 7 years ago

too true

Erik Osterman (Cloud Posse)over 7 years ago

😃

lorenover 7 years ago

https://github.com/gruntwork-io/cloud-nuke

Erik Osterman (Cloud Posse)over 7 years ago(edited)

~~too bad it doesn't delete ECS too~~

Erik Osterman (Cloud Posse)over 7 years ago

a number of other tools though

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/rebuy-de/aws-nuke

lorenover 7 years ago

Yeah, it doesn't do everything, for sure, but if the idea is right, and the design is right, open a PR 😉

Erik Osterman (Cloud Posse)over 7 years ago(edited)

~~though that's the thing; they have a PR for that, but won't merge it b/c it won't selectively nuke~~ 😃

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/gruntwork-io/cloud-nuke/pull/36

Erik Osterman (Cloud Posse)over 7 years ago

oh, maybe this is out of date now

Erik Osterman (Cloud Posse)over 7 years ago

haha, it was @sarkis who opened the issue

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/gruntwork-io/cloud-nuke/issues/32

sarkisover 7 years ago

😎

Erik Osterman (Cloud Posse)over 7 years ago

thanks @loren this was a great suggestion. i think we can work with something like this.

lorenover 7 years ago

Yeah, they merged it, but removed some functionality due to limitations they couldn't figure out at the time

chrismover 7 years ago

Anyone used terraform for helm + k8 without wondering why they f'king bothered and didn't just use the 5 or 6 lines of shell they usually used to set helm up 😬

chrismover 7 years ago

some of this stuffs like self-flagellation

Erik Osterman (Cloud Posse)over 7 years ago

@chrism we have been using Helmfile

Erik Osterman (Cloud Posse)over 7 years ago

I am curious to learn what problems you have run into using helm with terraform.

Erik Osterman (Cloud Posse)over 7 years ago

(Also in our experience it’s never been 5-6 lines! So many configuration options.)

chrismover 7 years ago

I have it wired up to deploy a cluster in a private network via RKE; so that bits fine; there's a terraform plugin which does most of what's needed.
I scripted out the terraform to setup helm to deploy rancher to it. But its flaky as hell

chrismover 7 years ago

All the "workarounds" to setting up tiller basically end in people posting commands to execute manually.
And for extra fun because all this junks then in the tf state file when you destroy it locks up trying to undo helm which doesnt work very well

chrismover 7 years ago

and yeah 6 or 7 lined is a little off 12 is more accurate

chrismover 7 years ago

Ultimately though we're using rancher, then we launch clusters from rancher (cant script that yet)

chrismover 7 years ago

I've a growing hatred for acls and security groups 😆

chrismover 7 years ago

There should be a training mode where you just run shit for a while and they spit out by tag which group needs to talk to what on which ports

chrismover 7 years ago

TF should have just gone with helm file and avoided the mass of extra types it has to support and the usual code rot / delayed releases

chrismover 7 years ago

You know you're starting to lose the plot when you have helpers called fuckingssh to ssh via a bastion but making sure the stupid keys written out by terraform are chmod 400 because too many shits were given

Erik Osterman (Cloud Posse)over 7 years ago

Haha

Erik Osterman (Cloud Posse)over 7 years ago

@mumoshu once proposed a Helmfile provider for terraform

chrismover 7 years ago

just seems more logical for something that doesn't "undo" very well

chrismover 7 years ago

running helm reset only works half the time and when it does the bugger hangs around thinking about it

chrismover 7 years ago

I hate helm

chrismover 7 years ago

I get the idea

chrismover 7 years ago

But the implementation makes me want to throw shit

chrismover 7 years ago

Rancher switching to helm from rke-addons was also a total pita

chrismover 7 years ago

they took a process that reliably worked, and swapped it for one that can fail magically

chrismover 7 years ago

and added 2 more pages of instructions to the setup 😆

chrismover 7 years ago

the Rancher K2 stuff looked interesting (or was it k3) if for nothing more than them reducing the amount of shit required

chrismover 7 years ago(edited)

(something I liked about nomad)

chrismover 7 years ago

Quickly shoved a aws_security_group in my config; now to spend 20 minutes changing to separate entries because terraform thinks it needs to delete it to update 😬

Erik Osterman (Cloud Posse)over 7 years ago

I am thinking of maybe setting up a weekly recurring "office hours" that would be free for anyone to join

Erik Osterman (Cloud Posse)over 7 years ago

it would be specifically to help those using geodesic and our modules

Erik Osterman (Cloud Posse)over 7 years ago

Thinking 11am PST on Wednesdays.

Erik Osterman (Cloud Posse)over 7 years ago

Yes that works

chrismover 7 years ago

anyone got a TLDR price comparison of running a cluster in EKS vs running it manually with partially reserved nodes.

chrismover 7 years ago

Saw an article by cloudhealth but you know when you really cba reading pages of shit just for a THIS COSTS MORE 😛

mmuehlbergerover 7 years ago

I don’t have a cost comparison, but it basically replaces your master nodes for a $144 per month fee.

mmuehlbergerover 7 years ago

Too expensive for small clusters, but better when sizing up and desiring HA.

mmuehlbergerover 7 years ago

Update on AWS account limit increase: They approved an account I created last Thursday for 10 sub-accounts. (The account is completely independent, but same company had another account previously though, which shared the same billing address, not sure if that matters).

chrismover 7 years ago

Fun fact; when you setup rancher you have to give it a hostname for the cluster to use.
Fair enough you might think.
But if you're setting it up on a private network with an internal lb and planning to expose it from the public subnet via nginx, IF you use a real domain name all the k8 clusters you make HAVE to be able to talk to that dns entry.

There's no x-host override to pass down (so say you could set rancher to use myinternallb.local and nginx could have mycluster.domain.com)

chrismover 7 years ago

If you set nginx to pass the lb name as host header down because of the way they've written the damn UI it fails

chrismover 7 years ago

the ui loads

chrismover 7 years ago

but all the calls are fully resolved by URL so it tries calling into nginx with the local lb name 😱

chrismover 7 years ago

Other things have a similar issue like identity server; but they had the foresight to allow you to pass additional headers in.

erikover 7 years ago

added an integration to this channel: Google Calendar for Team Events

Unknown Userover 7 years ago

Alex Siegmanover 7 years ago(edited)

Curious, does the calendar figure out the right time zone to show me based on my settings in slack? 1:30 PM with no TZ info is basically a number without units, useless! (Pet peeve, I work from 3 timezones in the US plus Japan regularly, so I have to deal with TZ translation all the time)

Erik Osterman (Cloud Posse)over 7 years ago(edited)

Thanks I will update the title to include the TZ

Erik Osterman (Cloud Posse)over 7 years ago

11:30am PST

Erik Osterman (Cloud Posse)over 7 years ago

Thanks @oscarsullivan_old! Will review that

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/cloudposse/terraform-root-modules/pulls

Erik Osterman (Cloud Posse)over 7 years ago

@tamsky has also been working on some .envrc docs

Erik Osterman (Cloud Posse)over 7 years ago(edited)

if you can make it on the call on wednesday, we can review the multi-region discussion

Erik Osterman (Cloud Posse)over 7 years ago

should be quite straight forward actually

joshmyersover 7 years ago

multi region is interesting, where does your state live?

joshmyersover 7 years ago

hope ya’ll are namespacing things by region!

SweetOps #geodesicover 7 years ago

Alex Siegmanover 7 years ago

Dangit, I meant to show up to that so I could listen in. I need to add these to my calendar!

Erik Osterman (Cloud Posse)over 7 years ago

@Alex Siegman we'll have another one next week

Erik Osterman (Cloud Posse)over 7 years ago

same time

lorenabout 7 years ago

if you do choose to use envs, you need to quote the assignment for complex var-types...

lorenabout 7 years ago

https://www.terraform.io/docs/configuration/variables.html#complex-typed-values

lorenabout 7 years ago

for a map:


export TEST='{bar="baz", foo="zzz"}'

chrismabout 7 years ago

When you type yes outside of a question and /usr/bin/yes goes into an infinite loop that ctrl+c wont break 🙄

Erik Osterman (Cloud Posse)about 7 years ago

With terraform it supports a flag to default to yes on all prompts

Erik Osterman (Cloud Posse)about 7 years ago

No need to script it

lorenabout 7 years ago

whew, tks for reminding me! i went to apply a config and got distracted, forgot it was waiting on my input 😂

lorenabout 7 years ago

of course my temp cred had expired. sigh

lorenabout 7 years ago

another thing i'm looking forward to in 0.12, support for credential_process providers

Erik Osterman (Cloud Posse)about 7 years ago

aws-vault exec --server is to address this problem provided a sufficiently long session duration (max 12 h)

lorenabout 7 years ago

i unfortunately do not have permissions to do that in this particular environment. it's a federated identity, 1hr max. only way to get a cred is with custom utilities. i might have to steal that --server idea though... i can easily write my own tool...

Erik Osterman (Cloud Posse)about 7 years ago

yes, there are some IAM metadata proxies out there that emulate the AWS behavior

Erik Osterman (Cloud Posse)about 7 years ago

could be a good place to start @loren

Erik Osterman (Cloud Posse)about 7 years ago

https://www.google.com/search?q=aws+metadata+proxy&rlz=1C5CHFA_enUS755US755&oq=aws+metadata+proxy&aqs=chrome..69i57.2121j0j1&sourceid=chrome&ie=UTF-8

lorenabout 7 years ago

Yep, on it, thanks! 😄

Erik Osterman (Cloud Posse)about 7 years ago

of those of you using geodesic (recent release), does your terminal look like this when you use assume-role?

erikabout 7 years ago

Erik Osterman (Cloud Posse)about 7 years ago

here’s an animated gif (slightly washed out) https://sweetops.com/wp-content/uploads/2019/03/geodesic-demo-1.gif

Erik Osterman (Cloud Posse)about 7 years ago

or an SVG animation: https://sweetops.com/wp-content/uploads/2019/03/termtosvg_fmnxoium.svg

Erik Osterman (Cloud Posse)about 7 years ago

Basically, what I want to understand if it renders “normally” for you, or if you have some ugliness

joshmyersabout 7 years ago

Is that not working as expected?

SweetOps #geodesicabout 7 years ago

There is 1 event this week

Abel Luckabout 7 years ago

which terraform module is responsible for creating a billing user/group such that a user can access the billing section of the console

Abel Luckabout 7 years ago

it seems to me that the admin users created have admin perms on the sub accounts, but zero perms on the root account. i'd like to be able to add an additional group policy to allow some of these users to access billing

Erik Osterman (Cloud Posse)about 7 years ago

We don't have any turnkey groups like that right now. Just haven't gotten around to it.

Erik Osterman (Cloud Posse)about 7 years ago(edited)

Maybe add the group+policy here? https://github.com/cloudposse/terraform-root-modules/tree/master/aws/root-iam

caseyabout 7 years ago

has anyone been able to port forward from the geodesic shell? Im trying to view kubernetes-dashboard on my host machine but unable to get port forwarding working.

caseyabout 7 years ago

i have tried that but saying site cannot be reached when going to localhost:54515

Andriy Knysh (Cloud Posse)about 7 years ago

@casey take a look here https://github.com/cloudposse/docs/issues/428

caseyabout 7 years ago

thank you, that worked!

Erik Osterman (Cloud Posse)about 7 years ago(edited)

Awesome @Abel Luck ! Sounds spot on

SweetOps #geodesicabout 7 years ago

chrismabout 7 years ago

What's the deal with teleport (noticed it slid into geodesic)

caseyabout 7 years ago

Has anyone had this problem with ubuntu 16.04? I updated Geodesic to 0.85.0 and I am now getting the following when I go to assume role:

caseyabout 7 years ago

Erik Osterman (Cloud Posse)about 7 years ago

@Jeremy G ☝️

Erik Osterman (Cloud Posse)about 7 years ago

Can you try the most recent previous release and see if it works? If so, then we will know it was a recent change

caseyabout 7 years ago

0.86.0?

caseyabout 7 years ago

oh miss read sorry

Erik Osterman (Cloud Posse)about 7 years ago

0.84.1

caseyabout 7 years ago

yeah one sec

caseyabout 7 years ago

another note is that it is working on mac for 0.85.0

Erik Osterman (Cloud Posse)about 7 years ago

ohhh interesting

Erik Osterman (Cloud Posse)about 7 years ago

is this issue on your WSL machine?

caseyabout 7 years ago

no im on ubuntu 16.04

Erik Osterman (Cloud Posse)about 7 years ago

@oscarsullivan_old reported some problem too (he’s also on ubuntu)

Erik Osterman (Cloud Posse)about 7 years ago

i’ll be free this afternoon to take a look

caseyabout 7 years ago

yeah, confirming 0.84.1 is not working either

Erik Osterman (Cloud Posse)about 7 years ago

ohhh interesting

caseyabout 7 years ago

I think it has something to do with the interactive prompt that shows, I found this issue but not sure what the resolution was

caseyabout 7 years ago

https://github.com/cloudposse/geodesic/issues/391

Erik Osterman (Cloud Posse)about 7 years ago

maybe 0.80.0 broke it?

Erik Osterman (Cloud Posse)about 7 years ago

when we went from alpine 3.8 -> 3.9

caseyabout 7 years ago

ill try 0.79.0 one sec

caseyabout 7 years ago

nope 0.79.0 same issue

caseyabout 7 years ago

0.56.0 working, but non-interactive

caseyabout 7 years ago

*0.57.0

Jeremy G (Cloud Posse)about 7 years ago

@casey I don't have access to an Ubuntu machine running Docker running Geodesic to test, but I am responsible for most of the recent changes to Geodesic and am happy to help. Do you see any problems before trying to assume-role?

caseyabout 7 years ago

assume-role is working on geodesic 0.57.0

caseyabout 7 years ago

i bumped to 0.85.0 working on my mac, and everything was fine on osx

caseyabout 7 years ago

went back ubuntu 16.04 at home and it was not working

Jeremy G (Cloud Posse)about 7 years ago

Please try echo foo | fzf --height 30% --preview 'echo bar' and LMK if that works in 0.8[56].0 on Ubuntu

Jeremy G (Cloud Posse)about 7 years ago

You should see an interactive prompt, and then just hit return and the prompt should clear.

caseyabout 7 years ago

getting

caseyabout 7 years ago

Failed to read /dev/tty

caseyabout 7 years ago

Jeremy G (Cloud Posse)about 7 years ago

OK, the problem is fzf. What does env | grep TERM print?

caseyabout 7 years ago

TERM=xterm-256color

Jeremy G (Cloud Posse)about 7 years ago

Are you in fact running some kind of xterm or are you running a script without a tty?

caseyabout 7 years ago

I dont know

caseyabout 7 years ago

im just using the default terminal on ubuntu

Jeremy G (Cloud Posse)about 7 years ago

Can you change the terminal window size?

caseyabout 7 years ago

yeah

Jeremy G (Cloud Posse)about 7 years ago

When you resize the window, does echo ${COLUMNS}x${LINES} change?

caseyabout 7 years ago

yeah

Jeremy G (Cloud Posse)about 7 years ago

Let's simplify a bit and unset FZF_DEFAULT_OPTS and then try the above fzf command again.

caseyabout 7 years ago

same

caseyabout 7 years ago

Failed to read /dev/tty

Jeremy G (Cloud Posse)about 7 years ago

OK. Let's back up. (If you have time to work through this now).

caseyabout 7 years ago

i can go at it for another 10 min

caseyabout 7 years ago

then we can resume when I am back if its not enough time

Jeremy G (Cloud Posse)about 7 years ago(edited)

great. Try ([[ -t 1 ]] && echo true) || echo false

caseyabout 7 years ago

im doing this inside geodesic container correct?

Jeremy G (Cloud Posse)about 7 years ago

Do it inside the Geodesic container and then open a second window on the Ubuntu machine (not Geodesic) and run that same command line.

caseyabout 7 years ago

saying bash syntax error near `true)'

caseyabout 7 years ago

`bash: conditional binary operator expected
bash: syntax error near `true)'
`

Jeremy G (Cloud Posse)about 7 years ago

Sorry, typo, fixed

caseyabout 7 years ago

both are saying true

Jeremy G (Cloud Posse)about 7 years ago

Let me do a little research. When can you pick this up again?

caseyabout 7 years ago

just ping me

caseyabout 7 years ago

ill be back in like an hour or 2

Jeremy G (Cloud Posse)about 7 years ago

Ping me with @Jeremy G 6 hours from now if you haven't heard from me before then.

Jeremy G (Cloud Posse)about 7 years ago

As a workaround until this is fixed, setting export ASSUME_ROLE_INTERACTIVE=false should get you back to work. You can use our new customization features to set this up automatically each time you run the container. See https://github.com/cloudposse/geodesic/pull/422 for how to do that.

Jeremy G (Cloud Posse)about 7 years ago

@casey The assume-role failure is due to a bug in the Ubuntu kernel. See https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1813873

Jeremy G (Cloud Posse)about 7 years ago(edited)

Solution is to upgrade your Ubuntu kernel to 4.4.0-143.169 or 4.15.0-46. See https://github.com/cloudposse/geodesic/issues/427#issuecomment-477805621

Erik Osterman (Cloud Posse)about 7 years ago

Awesome sleuthing

caseyabout 7 years ago

that was it!

caseyabout 7 years ago

how did you figure that out?! thank you though

Erik Osterman (Cloud Posse)about 7 years ago

@Jeremy G knows this kind of stuff inside and out

caseyabout 7 years ago

clearly. That was impressive

February 2019 Browse by date April 2019