#terraform - August 2018 | Slack Archive

terraform

661 messages

Archive: https://archive.sweetops.com/terraform/

Jul 2018

August 2018

Sep 2018

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/cloudposse/terraform-aws-iam-assumed-roles/issues/6

Erik Osterman (Cloud Posse)over 7 years ago

Would like feedback on this. Am I missing something here?

pmullerover 7 years ago

@pmuller has joined the channel

pmullerover 7 years ago

one thing i'd like to do here is to restrict aws account usage by source ip address

pmullerover 7 years ago

if your users all work in the same office

pmullerover 7 years ago

you could restrict mfa management to your office ip address

Erik Osterman (Cloud Posse)over 7 years ago

yea, that could be an enhancement

pmullerover 7 years ago

that way an attacker should steal api keys or password + be physically close to your office and hack your wifi to abuse your aws account

Erik Osterman (Cloud Posse)over 7 years ago

one thing I’ve loved about Duo is the ability to do geofencing

pmullerover 7 years ago

did not try this yet but i'd like to do this here

pmullerover 7 years ago

Duo ?

Erik Osterman (Cloud Posse)over 7 years ago

yea, they are a MFA saas

Erik Osterman (Cloud Posse)over 7 years ago

(and now also do SSO)

Erik Osterman (Cloud Posse)over 7 years ago

http://duo.com/

Erik Osterman (Cloud Posse)over 7 years ago

We have support for this in our bastion container: https://github.com/cloudposse/bastion

pmullerover 7 years ago

so basically Duo will store your users secret mfa keys?

Erik Osterman (Cloud Posse)over 7 years ago

yep

Erik Osterman (Cloud Posse)over 7 years ago

and offer multiple forms of 2FA

Erik Osterman (Cloud Posse)over 7 years ago

you can choose which ones you allow

pmullerover 7 years ago

interesting ! (i dropped the google authenticator idea precisely because i was afraid of storing those mfa secrets)

Erik Osterman (Cloud Posse)over 7 years ago

you can also require, for example, that their MFA device is up to date

Erik Osterman (Cloud Posse)over 7 years ago

that their OS is current

Erik Osterman (Cloud Posse)over 7 years ago

etc..

Erik Osterman (Cloud Posse)over 7 years ago

so policy management at a higher level

Erik Osterman (Cloud Posse)over 7 years ago

yea, GA is a nice contribution by google, but seriously difficult to truly secure.

Erik Osterman (Cloud Posse)over 7 years ago

Also, are you familiar with Teleport for SSH?

pmullerover 7 years ago

their current os ? is it using TNC ?

Erik Osterman (Cloud Posse)over 7 years ago

their = ?

pmullerover 7 years ago

This "their" 🙂

pmullerover 7 years ago

how Duo is checking for OS version ?

Erik Osterman (Cloud Posse)over 7 years ago

since one factor is mobile push notifications

Erik Osterman (Cloud Posse)over 7 years ago

they can get a lot of metadata

Erik Osterman (Cloud Posse)over 7 years ago

including gps location

Erik Osterman (Cloud Posse)over 7 years ago

OS version

Erik Osterman (Cloud Posse)over 7 years ago

authenticator version

Erik Osterman (Cloud Posse)over 7 years ago

device os

Erik Osterman (Cloud Posse)over 7 years ago

etc

pmullerover 7 years ago

very interesting 🙂

pmullerover 7 years ago

about teleport, i would like to use it and log everything with it, but haven't taken time to try it yet

pmullerover 7 years ago

my current compromise is logging a lot with auditd and pam_audit_tty

erikover 7 years ago

pmullerover 7 years ago

that's from Duo ?

Erik Osterman (Cloud Posse)over 7 years ago

yea

pmullerover 7 years ago

nice!

erikover 7 years ago

pmullerover 7 years ago

maybe i am a bit too paranoid for the cloud era ... i grew up in datacenters, and started using aws in 2016 ... but i still have too much of the data center / oss guy thinking: DIY !
so at first i even had a hard time trusting AWS (then I realized how absurd it was..) 😉
let alone talking about a 3rd party for authentication

pmullerover 7 years ago

but yeah, i sometimes feel like i am wasting my time 😉

Erik Osterman (Cloud Posse)over 7 years ago(edited)

override.tf appears to have no impact/influence over terraform { ... } block

pmullerover 7 years ago

too bad

pmullerover 7 years ago

cannot terragrunt be helpful with your sed hack? (haven't tried it yet)

Erik Osterman (Cloud Posse)over 7 years ago(edited)

i am not well versed in terragrunt

Erik Osterman (Cloud Posse)over 7 years ago

i know terragrunt can do what our module does out of the box

Erik Osterman (Cloud Posse)over 7 years ago

but i like that we’re defining the infrastructure for bucket/dynamodb with terraform rather than another tool

pmullerover 7 years ago

i like ideas terragrunt brings on the table, but i know from experience it's better to use vanilla tools in the long term ... wrappers comes and go ... and creates debt!

Erik Osterman (Cloud Posse)over 7 years ago

yes - agree

Erik Osterman (Cloud Posse)over 7 years ago

i would consider using it for specific cases such as this

Erik Osterman (Cloud Posse)over 7 years ago

small in scope

pmullerover 7 years ago

i'd be happy to hear about your feedback

pmullerover 7 years ago

btw, do you test your tf modules ? if so, how ? (doing so with all the code i write... except tf, so it's frustrating)

Erik Osterman (Cloud Posse)over 7 years ago

we’re not quite there yet. right now, we do lint testing only.

pmullerover 7 years ago

same here

pmullerover 7 years ago

not even that ... we just run "terraform fmt" and check if there is a change in the repo

pmullerover 7 years ago

i feel poor with just this 😉

Erik Osterman (Cloud Posse)over 7 years ago

See our plan of attack here: https://sweetops.slack.com/archives/CB84E9V54/p1532332704000172

Erik Osterman (Cloud Posse)over 7 years ago

in the #geodesic channel

Erik Osterman (Cloud Posse)over 7 years ago

basically, we package all infrastructure in a container along with terraform

Erik Osterman (Cloud Posse)over 7 years ago

thus, it’s an app like everything else

Erik Osterman (Cloud Posse)over 7 years ago

our strategy is to use our testing account to run: plan, apply, plan, destroy, plan

Erik Osterman (Cloud Posse)over 7 years ago

I am less enthused by atlantis and terratest

pmullerover 7 years ago

haven't heard about atlantis

pmullerover 7 years ago

terratest requires to write tests in go, which is rather unpleasant

pmullerover 7 years ago

wants a nice test runner/framework like pytest ;)

pmullerover 7 years ago

lunch time, bbl

Erik Osterman (Cloud Posse)over 7 years ago

cool, ttyl!

pmullerover 7 years ago

ior have a good night 🙂

Erik Osterman (Cloud Posse)over 7 years ago

(yea, I’m in Los Angeles)

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/wata727/tflint

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/akaspin/terraform-provider-generic

Erik Osterman (Cloud Posse)over 7 years ago

this is an interesting concept. it reminds me of the “porcelain” plugin module used in helm, but applied to terraform providers

Erik Osterman (Cloud Posse)over 7 years ago

using this, it’s possible to define a “basic” provider in any language, and then accept targets like apply, destroy, etc

Erik Osterman (Cloud Posse)over 7 years ago

the implementation above seems like a POC, but if anyone else has seen one, lmk

Erik Osterman (Cloud Posse)over 7 years ago

note, this is very different from the data external provider which does not distinguish between phases

pmullerover 7 years ago

oh nice!

Erik Osterman (Cloud Posse)over 7 years ago

Related to the earlier question of terraform cicd, this looks interesting

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/mercari/tfnotify/blob/master/README.md

Erik Osterman (Cloud Posse)over 7 years ago

This will take the output of terraform plan/apply and post it back to GitHub or Slack.

Erik Osterman (Cloud Posse)over 7 years ago

@Andriy Knysh (Cloud Posse) wrote a generic tool for this, not specialized for terraform called ‘github-commenter’

pmullerover 7 years ago

wow, thanks, those projects look great!

pmullerover 7 years ago

something related I would like to implement: https://github.com/mozilla-services/pytest-services

Andriy Knysh (Cloud Posse)over 7 years ago

hi @pmuller

Andriy Knysh (Cloud Posse)over 7 years ago

for security testing on AWS, take a look at this tool https://github.com/dowjones/hammer

Andriy Knysh (Cloud Posse)over 7 years ago

https://blogs.wsj.com/cio/2018/07/24/dow-jones-develops-automated-security-tool/

i5okieover 7 years ago

hi again

i5okieover 7 years ago

so im looking at terraform-aws-s3-log-storage module. looks awesome.
Here’s a loaded question.. I’ve a got a custom elastic beanstalk platform, using your terraform-aws-elasticbeanstalk module..

Is there a way you’d recommend to make that bucket name exposed to eb instances? (not manually).

Erik Osterman (Cloud Posse)over 7 years ago

Hrmm... so basically, how to pass the bucket name from the log storage module to the beanstalk module?

i5okieover 7 years ago

I think I’ll be using fluentd to push logs, and I’ve ran into a bit of head scratcher, the config file for td-agent doesn’t support environmental vars so

i5okieover 7 years ago

yes

Erik Osterman (Cloud Posse)over 7 years ago

td-agent ~ fluentd?

i5okieover 7 years ago

yes

i5okieover 7 years ago

just a non-gem version

Erik Osterman (Cloud Posse)over 7 years ago

ok, so taking a further step back

Erik Osterman (Cloud Posse)over 7 years ago(edited)

you're running fluentd on the beanstalk instances to forward logs

i5okieover 7 years ago

yep

Erik Osterman (Cloud Posse)over 7 years ago

ok, gotcha. sec

Erik Osterman (Cloud Posse)over 7 years ago

Ok, so the tf-agent config should get parameterized

Erik Osterman (Cloud Posse)over 7 years ago

We typically use gomplate for that

Erik Osterman (Cloud Posse)over 7 years ago

then using gomplate, you can consume env variables

Erik Osterman (Cloud Posse)over 7 years ago

then pass the bucket name as an env to the beanstalk

i5okieover 7 years ago

I haven’t completely settled on fluentd. but it leaves a bit of flexibility instead of marrying to data firehose..

i5okieover 7 years ago

Erik Osterman (Cloud Posse)over 7 years ago

https://docs.cloudposse.com/tools/gomplate/

i5okieover 7 years ago

thanks

Erik Osterman (Cloud Posse)over 7 years ago

https://gomplate.hairyhenderson.ca/

i5okieover 7 years ago

okay that just looks like magic

i5okieover 7 years ago

lol

i5okieover 7 years ago

hey, cloudposse/terraform-aws-cloudwatch-logs in description says … for use with fluentd.
Do you guys have an example of how you use it with fluentd?

Erik Osterman (Cloud Posse)over 7 years ago

We use it with Kubernetes

Erik Osterman (Cloud Posse)over 7 years ago

I can share how we do that

Erik Osterman (Cloud Posse)over 7 years ago

hey @Ziad Hilal! Glad you signed up 😃

Ziad Hilalover 7 years ago

@Ziad Hilal has joined the channel

Maarten van der Hoefover 7 years ago

Not Terraform, but .. should be here I think..
I had a fight with a daemon which was running inside a wrapper. It is sidekiq, and it's important that sidekiq receives the SIGTERM. The wrapper script was necessary as it does a few other things.

Having this inside the dispatch.sh


   SIDEKIQ_COUNT=3 SIDEKIQ_MAXMEM_MB=2000 SIDEKIQ_PRELOAD=sidekiq_swarm exec sidekiqswarm -t 25 -C config/sidekiq.yml

is not enough as it would still be a child of the entrypoint.

So in the Dockerfile this is what did the trick.


CMD exec /$APP_DIR/bin/dispatch.sh

Erik Osterman (Cloud Posse)over 7 years ago

@Maarten van der Hoef was that meant for this channel?

Maarten van der Hoefover 7 years ago

yeah,.. I just wanted to share something others might stumble upon at one moment.. Unsuited ?

Erik Osterman (Cloud Posse)over 7 years ago

Oh, just maybe don't get the full context.

Erik Osterman (Cloud Posse)over 7 years ago

😃

Erik Osterman (Cloud Posse)over 7 years ago

Aha, adding the exec part

Maarten van der Hoefover 7 years ago

exactly.. on 2 places

Erik Osterman (Cloud Posse)over 7 years ago

so basically, it's running /bin/sh -c 'exec /$APP_DIR/bin/dispatch.sh', which then replaces PID1 with dispatch.sh

Erik Osterman (Cloud Posse)over 7 years ago

I've also run into problems with signal handling and shell scripts with docker.

Maarten van der Hoefover 7 years ago(edited)

and then inside dispatch sidekiq replaces it

Maarten van der Hoefover 7 years ago

we can also have a docker channel

Maarten van der Hoefover 7 years ago

haha

Erik Osterman (Cloud Posse)over 7 years ago

yea, probably a good idea.

Maarten van der Hoefover 7 years ago

I'll remove my stuff here then

Erik Osterman (Cloud Posse)over 7 years ago

#docker created

Erik Osterman (Cloud Posse)over 7 years ago

that's fine - we'll start new ones there.

Erik Osterman (Cloud Posse)over 7 years ago

😃

i5okieover 7 years ago

hey so for terraform-aws-elasticsearch module. its saying that EBS storage must be selected for t2.small.elasticsearch.

i5okieover 7 years ago

is that not what’s used by default?

Erik Osterman (Cloud Posse)over 7 years ago

@Andriy Knysh (Cloud Posse) ☝️

i5okieover 7 years ago

i’m gonna do more research into ES for now

i5okieover 7 years ago

well apparently i fail it setting up ES from this tf module. i’ve got a route53 domain, that all worked, the vpc endpoint is in public subnets.. but I cannot access ES or Kibana endpoints at all.

i5okieover 7 years ago

and security group allows all traffic from 0.0.0.0/0

Erik Osterman (Cloud Posse)over 7 years ago

you deployed it in the public subnets and opened up the security groups?

i5okieover 7 years ago

i believe so. I’ve just added the default security group just to try. will see if that helps

i5okieover 7 years ago

i have iam access policy configured. im thinking thats the issue

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/cloudposse/terraform-root-modules/blob/master/aws/backing-services/

Erik Osterman (Cloud Posse)over 7 years ago

here’s an example of how andriy deployed it

i5okieover 7 years ago

tnx

i5okieover 7 years ago

might be a dump question. but what is the purpose of this:

i5okieover 7 years ago

in terraform-aws-elastic-beanstalk-environment

Erik Osterman (Cloud Posse)over 7 years ago

It's part of this horible/nasty hack to make it easy to pass envs

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/cloudposse/terraform-aws-elastic-beanstalk-environment/blob/master/main.tf#L648

Erik Osterman (Cloud Posse)over 7 years ago

perhaps there's a better way of doing it now adays in HCL

Erik Osterman (Cloud Posse)over 7 years ago(edited)

We had a more elegant way using null_resource, but it would lead to frequent errors like cannot compute count of dynamic variable (or something like that)

Erik Osterman (Cloud Posse)over 7 years ago

So basically, the module always defines N fixed environment variables

Erik Osterman (Cloud Posse)over 7 years ago

If the user provides it, it uses it, other wise you see something like DEFAULT_ENV_20=UNSET in your beanstalk environment

Erik Osterman (Cloud Posse)over 7 years ago

(which is just a place holder)

i5okieover 7 years ago

odd

i5okieover 7 years ago

I was looking at an existing platform. I wonder why they’re not showing how to use their container config in examples.

i5okieover 7 years ago

with a custom platform, they just say here, specify :environment settings, and bam you got a platform. but then for example, looking at Ruby / passenger platform they have :container config as well, with nicely defined json for settings and what not.

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/cloudposse/terraform-aws-elastic-beanstalk-environment/pull/14

Erik Osterman (Cloud Posse)over 7 years ago

Here is more context

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/cloudposse/terraform-aws-elastic-beanstalk-environment/pull/11

Erik Osterman (Cloud Posse)over 7 years ago(edited)

@pmuller I still think we should consider this:https://github.com/cloudposse/terraform-aws-rds-cluster/issues/26

Erik Osterman (Cloud Posse)over 7 years ago

It would be a nice security enhancement

pmullerover 7 years ago

hehe, love this approach!

pmullerover 7 years ago

did someone tried https://github.com/juliosueiras/vim-terraform-completion ?

Erik Osterman (Cloud Posse)over 7 years ago

@Jamie Nelson (BitFlight) https://github.com/cloudposse/terraform-aws-iam-assumed-roles/pull/7

Erik Osterman (Cloud Posse)over 7 years ago

any opinions on taking this approach?

Erik Osterman (Cloud Posse)over 7 years ago(edited)

@sarkis @Andriy Knysh (Cloud Posse) ☝️

Andriy Knysh (Cloud Posse)over 7 years ago

please see my comments on that: https://github.com/cloudposse/terraform-aws-iam-assumed-roles/pull/7#issuecomment-411117761

Erik Osterman (Cloud Posse)over 7 years ago

@Jamie Nelson (BitFlight) shouldn't we be using a label here? https://github.com/cloudposse/terraform-aws-efs-cloudwatch-sns-alarms/blob/master/alarms.tf#L13

Jamie Nelson (BitFlight)over 7 years ago

Yes

Erik Osterman (Cloud Posse)over 7 years ago

Ok, I'll log an issue

Jamie Nelson (BitFlight)over 7 years ago

All of the modules need to have a revision to handle context

Erik Osterman (Cloud Posse)over 7 years ago

how are you doing man?

Jamie Nelson (BitFlight)over 7 years ago

Exhausted!

Erik Osterman (Cloud Posse)over 7 years ago(edited)

from vacation? 😃

Jamie Nelson (BitFlight)over 7 years ago

Yeah

Jamie Nelson (BitFlight)over 7 years ago

Way behind from vacation so playing hard catchup

Erik Osterman (Cloud Posse)over 7 years ago

My ticket is up tomorrow! Tulum, MX

Andriy Knysh (Cloud Posse)over 7 years ago

@Jamie Nelson (BitFlight) some of your changes to terraform-null-label do not respect the enabled flag. So when I test it with var.enabled=false, Terraform still wants to recreate some resources

Andriy Knysh (Cloud Posse)over 7 years ago

when you change the module, you might want to take a look at that too

Jamie Nelson (BitFlight)over 7 years ago

I will address that. Sorry that should have been picked up at review. But it was pushed through quickly.

Jamie Nelson (BitFlight)over 7 years ago

Thank you for testing.

Dominic Monroeover 7 years ago

I'm trying to figure out how to use https://github.com/cloudposse/terraform-aws-rds-cloudwatch-sns-alarms with https://github.com/cloudposse/terraform-aws-sns-lambda-notify-slack am I missing something obvious?

paulover 7 years ago

👋 Hello! 🙂 I'm attempting to use terraform-aws-cloudtrail for the first time and just hitting an issue with the event_selector. It defaults to {} but this causes aws_cloudtrail to create an event selector anyway.

paulover 7 years ago

We then get this on every apply:

  ~ module.cloudtrail.module.cloudtrail.aws_cloudtrail.default
      event_selector.#:                           "0" => "1"
      event_selector.0.include_management_events: "" => "true"
      event_selector.0.read_write_type:           "" => "All"

paulover 7 years ago

Removing the event_selector variable/parameter from terraform-aws-cloudtrail 'fixes' the issue.

paulover 7 years ago

Looks like null / undefined parameters will be coming in HCL2 but until then I'm not sure what the answer is.

Erik Osterman (Cloud Posse)over 7 years ago

Hrmmm I am not sure off the top of my head. @Andriy Knysh (Cloud Posse) might have some suggestions. He originally implemented it.

paulover 7 years ago

🤞 🙂

antonbabenkoover 7 years ago

@antonbabenko has joined the channel

Andriy Knysh (Cloud Posse)over 7 years ago

hi @paul, give me some time, I’ll look into terraform-aws-cloudtrail event_selector. You can open a PR with your fixes so it’d be easier to review and test

paulover 7 years ago

@Andriy Knysh (Cloud Posse) 👋 At the moment I don't have a fix, other than commenting out the event_selector. We're discussing it internally at the moment, if we think of a neat way around it then I'll PR it.

Dominic Monroeover 7 years ago

@Andriy Knysh (Cloud Posse) @paul out of curiosity, does the string "" work as blank? Or maybe doing event_selector = []?

paulover 7 years ago

I'll give it a go, let's find out.

paulover 7 years ago(edited)

Actually, the variable type is set to a ~~map~~ list.

Dominic Monroeover 7 years ago

hehe 😉 I hit something similar this morning.

paulover 7 years ago

of maps.

paulover 7 years ago

🤔

paulover 7 years ago

Giving it a go.

Dominic Monroeover 7 years ago

Yep. It's the same for launch_configuration root_block_device, for no particular reason, perhaps for nicer syntax as:


event_selector {
  …
}

Rather than


event_select = {
  …
}

Dominic Monroeover 7 years ago

tbh, the pain of this goes away with terraform 12 if you can wait, now that conditionals can return lists you can do:


event_selector = "${var.event_selector == {} ? [] : [var.event_selector]}"

Dominic Monroeover 7 years ago(edited)

https://github.com/hashicorp/terraform/issues/12453#issuecomment-327266951 you could try this though if you're a masochist.

Dominic Monroeover 7 years ago(edited)

(Basically join the k/vs of the event_selector with a separator, and then split it outside the conditional)

paulover 7 years ago

🤔 So if I do this..

variable "event_selector" {
  type        = "list"
  description = "Specifies an event selector for enabling data event logging. See: <https://www.terraform.io/docs/providers/aws/r/cloudtrail.html> for details on this map variable"
  default     = []
}

paulover 7 years ago

It then works correctly.

paulover 7 years ago

It would then require people to supply a list of maps though, so it wouldn't be backwards compatible.

paulover 7 years ago

I think.

Dominic Monroeover 7 years ago(edited)

You could also do that, which is what I decided to do internally. However, I think that changes usage slightly. It might be as simple as changing event_selector = {} to event_selector { }

paulover 7 years ago

Where would I make that change?

Dominic Monroeover 7 years ago

when using the module, that would be, so still a breaking change.

paulover 7 years ago

Ah.

paulover 7 years ago

Fiddling with different combinations. 🙂

Danover 7 years ago

Anyone have experience deploying AWS AD service with terraform?(includes creating vpc, subnets, jumpboxes etc)

paulover 7 years ago

bah. No joy.

Dominic Monroeover 7 years ago

A map() function would make this so much easier (in the fp sense), as you could use compact & map.

Dominic Monroeover 7 years ago

@paul how about slice(list(var.event_selector), 0, length(var.event_selector) > 0 ? 1 : 0)

Dominic Monroeover 7 years ago

that took way too much fiddling to discover. Nice trick once you know it though 🙂

paulover 7 years ago

Oh, interesting. I'll give that a try later this evening. (BST timezone)

paulover 7 years ago

Thanks.

Dominic Monroeover 7 years ago

Fellow brit? 😄 👋

paulover 7 years ago

👋 🙂

Jamie Nelson (BitFlight)over 7 years ago

👋

Andriy Knysh (Cloud Posse)over 7 years ago

@paul @Dominic Monroe so I tested this example https://github.com/cloudposse/terraform-aws-cloudtrail/blob/master/examples/complete/main.tf

Andriy Knysh (Cloud Posse)over 7 years ago

when event_selector = [] (empty list), Terraform never tries to recreate resources

Andriy Knysh (Cloud Posse)over 7 years ago

if we put a map inside a list, it always tries to recreate regardless if the map is empty or populated

Andriy Knysh (Cloud Posse)over 7 years ago

event_selector = [{}]

Andriy Knysh (Cloud Posse)over 7 years ago

  event_selector = [{
    read_write_type           = "All"
    include_management_events = true
  }]

Andriy Knysh (Cloud Posse)over 7 years ago

in both cases

  ~ module.cloudtrail.aws_cloudtrail.default
      event_selector.#:                           "0" => "1"
      event_selector.0.include_management_events: "" => "true"
      event_selector.0.read_write_type:           "" => "All"

Andriy Knysh (Cloud Posse)over 7 years ago

looks like a feature/bug

Andriy Knysh (Cloud Posse)over 7 years ago

also in the docs, it does not mention at all that event_selector should be a list, it looks like a map

Andriy Knysh (Cloud Posse)over 7 years ago

https://www.terraform.io/docs/providers/aws/r/cloudtrail.html

Andriy Knysh (Cloud Posse)over 7 years ago

so we can change it to

variable "event_selector" {
  type        = "list"
  description = "Specifies an event selector for enabling data event logging. See: <https://www.terraform.io/docs/providers/aws/r/cloudtrail.html> for details on this map variable"
  default     = []
}

Andriy Knysh (Cloud Posse)over 7 years ago

which will silence it in the case when we don’t need any event selectors

Andriy Knysh (Cloud Posse)over 7 years ago

but will not help in other cases

Andriy Knysh (Cloud Posse)over 7 years ago

@Jamie Nelson (BitFlight) any ideas on that?

Jamie Nelson (BitFlight)over 7 years ago

I’ll have to look soon. I’m out with people

Dominic Monroeover 7 years ago(edited)

@Andriy Knysh (Cloud Posse) I think we can have the best of both worlds using the slice() trick above. The other being that we don't break backwards compatibility.

Andriy Knysh (Cloud Posse)over 7 years ago

don’t worry about backwards compatibility, we use tags in all modules like here https://github.com/cloudposse/terraform-root-modules/blob/master/aws/cloudtrail/main.tf#L34

Andriy Knysh (Cloud Posse)over 7 years ago

so if we need to update to the new version, we update the tag and update the top-level module

Dominic Monroeover 7 years ago

I think backwards compatibility is always important, regardless of pinned versions. I imagine this is something that differs across programming communities. I also think the current interface is the most idiomatic.

Dominic Monroeover 7 years ago

@Jamie Nelson (BitFlight) Regarding SNS, RDS etc. I just after much hacking around terraform, managed to create a "terraform-sns-claims". Essentially modules like the RDS alarms module export a variable of claims with a value like ["cloudwatch", "rds"]. Internally, the sns module stores a statement relating to that claim. It then generates a policy from the set of claims. I need to find out if anything else actually pushes to SNS (e.g. autoscaling events perhaps), but this is a fairly tidy system for allowing the sns policy to be dictated by the alarms you want.

If you're hacking on the slack sns stuff tomorrow, I can probably publish my SNS work, and you can bring it into cloudposse if you wish (whatever license I need to use, I will). I'd love to not have to fork slack too 🙂 I'll probably look into contributing RDS event support at the same time.

Jamie Nelson (BitFlight)over 7 years ago(edited)

I’ll do it tomorrow then. Please push any suggestions you want and I’ll merge them as needed! Please and thank you!

paulover 7 years ago

Apologies, I didn't get chance to try out the suggested solutions this evening. It's in my diary for the morning.

paulover 7 years ago

I think I'm leaning towards the changing to a list solution. It's far cleaner solution than slicing and the AWS provider is expecting a list anyway.

Dominic Monroeover 7 years ago

@Jamie Nelson (BitFlight) references for the gh repos:

https://github.com/SevereOverfl0w/terraform-aws-rds-cloudwatch-sns-alarms
https://github.com/SevereOverfl0w/terraform-aws-sns-claims

Usage example for both found in the rds repo.

Dominic Monroeover 7 years ago

https://github.com/cloudposse/terraform-aws-cloudtrail-cloudwatch-alarms/blob/master/main.tf#L8 might be a better approach overall, uncertain of how multiple policies apply to an sns.

Dominic Monroeover 7 years ago

@Jamie Nelson (BitFlight) I ended up forking the slack lambda, https://github.com/SevereOverfl0w/terraform-aws-notify-slack it's not really in a pull-requestable state, but if you're hacking on this it might serve as inspiration.

Jamie Nelson (BitFlight)over 7 years ago

Thank you

Jamie Nelson (BitFlight)over 7 years ago

I'm on them now 🙂

Dominic Monroeover 7 years ago

I'm now onto monitoring our autoscaling groups 😄 I see you're also responsible for ec2 😉

Jamie Nelson (BitFlight)over 7 years ago

FYI team, this environment variable isn;t in the documentation last time i checked, but can get you around some annoying errors when destroying resources.

Jamie Nelson (BitFlight)over 7 years ago

export TF_WARN_OUTPUT_ERRORS=1

Jamie Nelson (BitFlight)over 7 years ago

It converts the errors that would normally halt the destruction of resources from a module

Jamie Nelson (BitFlight)over 7 years ago

into warnings

Jamie Nelson (BitFlight)over 7 years ago

and therefore allows you to complete the destruction of a TF template

Maarten van der Hoefover 7 years ago

For terraformers, who like dirty hacks, and who are encountering issues with count.index inside a conditionally created resource with count to length(var.of_a_list).. here's something that worked for me 😎

create a data "template_file" with count of the length of the list with no conditions.


data "template_file" "custom_listen_host" {
  count = "${length(var.custom_listen_hosts)}"
...

And refer to the template_file from the resource with the condition..


resource "aws_lb_listener_rule" 
"host_based_routing_custom_listen_host" {
....
  count = "${local.create && length(var.custom_listen_hosts) > 0 ? length(var.custom_listen_hosts) : 0 }"

    values = ["${data.template_file.custom_listen_host.*.rendered[count.index]}"]
}

Andriy Knysh (Cloud Posse)over 7 years ago

thanks @Maarten van der Hoef for the example

Maarten van der Hoefover 7 years ago

Until 0.12 is there, everything is allowed I thought @Andriy Knysh (Cloud Posse) 😉

Andriy Knysh (Cloud Posse)over 7 years ago

not every hack I guess, but if it looks good, why not 🙂

Andriy Knysh (Cloud Posse)over 7 years ago

this count = "${local.create && length(var.custom_listen_hosts) > 0 ? length(var.custom_listen_hosts) : 0 }" could be simplified to count = "${local.create ? length(var.custom_listen_hosts) : 0 }"

Maarten van der Hoefover 7 years ago

hey, that's true

Maarten van der Hoefover 7 years ago

thanks

Andriy Knysh (Cloud Posse)over 7 years ago

@paul

I think I’m leaning towards the changing to a list solution. It’s far cleaner solution than slicing and the AWS provider is expecting a list anyway

Andriy Knysh (Cloud Posse)over 7 years ago

want to test and open a PR? 🙂

paulover 7 years ago

Yeah, happy to give that a go tomorrow afternoon. Off the clock now 🙂

Andriy Knysh (Cloud Posse)over 7 years ago

perfect

Andriy Knysh (Cloud Posse)over 7 years ago

i tested it a little bit https://sweetops.slack.com/archives/CB6GHNLG0/p1533750657000315

Igor Rodionovover 7 years ago

@Igor Rodionov has joined the channel

paulover 7 years ago

I spent a bit of time on the issue with event_selector this afternoon but haven't got it working for all use cases (no event_selectors, a single event selector, multiple event selectors). I'm going to return to it towards the end of next week.

Andriy Knysh (Cloud Posse)over 7 years ago

thanks @paul

Andriy Knysh (Cloud Posse)over 7 years ago

(agree, not easy to make it working in all cases)

paulover 7 years ago

No problem. 🙂 I found lots of github issues around Terraform asking for it to support exactly this kind of thing but no concrete solutions have materialised.

paulover 7 years ago

I'm going to see if I can get a colleague or two of mine to have a look next week to see if we can think of a way around it. 🤞

i5okieover 7 years ago

hi again

i5okieover 7 years ago

I’ve just ran into a head-scratcher issue with terraform..beanstalk module i think.

i5okieover 7 years ago

I’ve got 5 environments. it created 4. but this last one its got this issue. and I can’t figure out for the life of me where its picking up “elb-logs-makeshift” bucket name from.

i5okieover 7 years ago

in the main.tf of the module, this is the only reference:

i5okieover 7 years ago

Erik Osterman (Cloud Posse)over 7 years ago

that's really odd! haven't seen elb-logs-makeshift before. I grepped through all of our modules and don't see it and it doesn't appear in any of our code. https://github.com/search?q=org%3Acloudposse+makeshift&type=Code

i5okieover 7 years ago

ikr

i5okieover 7 years ago

and terraform plan, grepped for elb, also doesn’t produce it.

Erik Osterman (Cloud Posse)over 7 years ago

what about your ENV?

Erik Osterman (Cloud Posse)over 7 years ago

export | grep makeshift

i5okieover 7 years ago

makeshift is a known name. but “elb-logs-makeshift” in that combination is not

i5okieover 7 years ago

uhm

i5okieover 7 years ago

export locally yes, but that’d affect other environments tho..

Erik Osterman (Cloud Posse)over 7 years ago(edited)

in this case, looks like namespace=elb, stage=logs, name=makeshift (if you're using our label module)

i5okieover 7 years ago

not explicitly no. just using your beanstalk environment module

Erik Osterman (Cloud Posse)over 7 years ago

i5okieover 7 years ago

we had this environment up before too. this is so bizarre. we used an older module, and im now rewriting with the new one. but then even in your module, the way you define it, it’d be elb-logs ending.. not elb-logs+string

i5okieover 7 years ago

#killmenow

Erik Osterman (Cloud Posse)over 7 years ago

yea, makes no sense. must be some silly error somewhere.

i5okieover 7 years ago

this has to be some bug with aws or something. because the rest of the environments just use a normal elb-logs-<random_digits> s3 bucket.

i5okieover 7 years ago

Don’t think i like EB anymore. So high maintenance

Erik Osterman (Cloud Posse)over 7 years ago

And very slow to iterate 😃

i5okieover 7 years ago

i’d ask amazon but don’t have support subscription yet

i5okieover 7 years ago

and funny thing is, aws cli says that s3 bucket doesn’t exist.

i5okieover 7 years ago

but when trying to create it, it says “$ aws s3 mb <s3://elb-logs-makeshift> makeshift2.3.7
make_bucket failed: <s3://elb-logs-makeshift> An error occurred (BucketAlreadyExists) when calling the CreateBucket operation: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.”

i5okieover 7 years ago

“$ aws s3 ls elb-logs-makeshift makeshift2.3.7

An error occurred (NoSuchBucket) when calling the ListObjects operation: The specified bucket does not exist”

Erik Osterman (Cloud Posse)over 7 years ago

oh, maybe someone else on AWS owns that bucket.

Erik Osterman (Cloud Posse)over 7 years ago

that would explain why you cannot list it

Andriy Knysh (Cloud Posse)over 7 years ago

@i5okie we suggest using label or a similar naming pattern for consistency and to eliminate naming contention

Andriy Knysh (Cloud Posse)over 7 years ago

yes, all buckets are global

Andriy Knysh (Cloud Posse)over 7 years ago

except when you do aws s3 ls, it shows only your account

i5okieover 7 years ago

yeah that makes sense.

Andriy Knysh (Cloud Posse)over 7 years ago

you said elb-logs-<randomstring>

Andriy Knysh (Cloud Posse)over 7 years ago(edited)

could makeshift be generated as one of the random strings ? 🙂

i5okieover 7 years ago

but why would it try to create/use a bucket that nothing asked it to use. and the other environments created with the exact same module and config essentially, just use the standard elb-logs-3434343434 thing that aws picks by default.

i5okieover 7 years ago

no, because other environments go into elb-logs-<randomdigits>

i5okieover 7 years ago

the same one

i5okieover 7 years ago

without that being explicitly specified anywhere in the config files.

i5okieover 7 years ago

oh i see. its actually elb-logs-<accountnumber>

Andriy Knysh (Cloud Posse)over 7 years ago

😉

Andriy Knysh (Cloud Posse)over 7 years ago

just use this https://github.com/cloudposse/terraform-null-label

Andriy Knysh (Cloud Posse)over 7 years ago

or this https://github.com/cloudposse/terraform-terraform-label

i5okieover 7 years ago

it makes no sense that from scratch I’ve just created other environments with this module and it went fine, and used the normal per-account elb-logs bucket.
and this one odd-ball decides not to.
I’m using this: https://github.com/cloudposse/terraform-aws-elastic-beanstalk-environment

i5okieover 7 years ago

for all of our environments.

i5okieover 7 years ago

so why would i use label separetly?

Andriy Knysh (Cloud Posse)over 7 years ago

not sure how it relates to the <accountnumber>

Andriy Knysh (Cloud Posse)over 7 years ago

can you share your code?

i5okieover 7 years ago

thats just what EB does naturally creates per-account elb-logs s3 bucket, and uses it for all environments. terraform no terraform, its just what it does

i5okieover 7 years ago

which code

i5okieover 7 years ago

i have to use local copy of the module because to modify the healthchec url varriable a bit.

Andriy Knysh (Cloud Posse)over 7 years ago

what’s the diff between those 5 envs? stage?

i5okieover 7 years ago

no. app names

i5okieover 7 years ago

5 applications, 3 stages each

i5okieover 7 years ago

but im only re-doing the staging env for now

i5okieover 7 years ago

which is in its own vpc, etc etc

i5okieover 7 years ago

so “name” variable would be different between the 5. plus app versions, and env vars.

i5okieover 7 years ago

terraform plan, or apply output does not mention “elb-logs-makeshift”. in fact here’s a line

Andriy Knysh (Cloud Posse)over 7 years ago

the EB logs bucket get created here https://github.com/cloudposse/terraform-aws-elastic-beanstalk-environment/blob/master/main.tf#L965

Andriy Knysh (Cloud Posse)over 7 years ago

but @i5okie still, how all of this relates to what you said about elb-logs-<randomdigits>?

Andriy Knysh (Cloud Posse)over 7 years ago

what is randomdigits here and where they from?

i5okieover 7 years ago

its actually account number.

i5okieover 7 years ago

ELB doesn’t use these buckets that are created.

i5okieover 7 years ago

it ends up using the account-default elb-logs-<accountnumber> bucket instead.

i5okieover 7 years ago

don’t know why

Andriy Knysh (Cloud Posse)over 7 years ago

it’s not in the image above

i5okieover 7 years ago

im thinking this is a bug at AWS side.

i5okieover 7 years ago

because I could let you read all of my terraform files, and you wouldn’t find anything that would put the strings together to say “elb-logs-makeshift”

i5okieover 7 years ago

oh i have an idea. i’ll delete the .terraform folder and try again

Andriy Knysh (Cloud Posse)over 7 years ago

yea try that

Andriy Knysh (Cloud Posse)over 7 years ago

can you show all bucket names from the 5 env that EB created?

i5okieover 7 years ago

there is in fact 5th one

Andriy Knysh (Cloud Posse)over 7 years ago

but where is randomdigits in there?

i5okieover 7 years ago

but they are all empty. and all elb logs go to the elb-logs-<accountnumber> bucket instead.

Andriy Knysh (Cloud Posse)over 7 years ago

it should not add any random things

i5okieover 7 years ago

your module doesn’t

i5okieover 7 years ago

its just what elb does

i5okieover 7 years ago

Andriy Knysh (Cloud Posse)over 7 years ago

i think you have naming collision somewhere

i5okieover 7 years ago

we used an older module of yourse for this. all names were the same. in fact the only difference i noticed between the versions of modules is the order of names/namespace/stage is different.

i5okieover 7 years ago

not that it matters

i5okieover 7 years ago

ok deleting terraform folder and re-starting didn’t help

i5okieover 7 years ago

and its lying

i5okieover 7 years ago

it did create the load balancer

i5okieover 7 years ago

but it quits on actions and doesn’t add instances

i5okieover 7 years ago

i’ll have to wait an hour before the “(terminated)” envs dissapear from console, and create the env manually. then try to import it into the terraform.

Andriy Knysh (Cloud Posse)over 7 years ago

@i5okie just to confirm. The bucket that terraform-aws-elastic-beanstalk-environment creates, is for Load balancer logs https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/command-options-general.html#command-options-general-elbv2

Andriy Knysh (Cloud Posse)over 7 years ago

they have the names as you showed in the images above

Andriy Knysh (Cloud Posse)over 7 years ago(edited)

at the same time, EB creates a bucket per region where it stores all EB stuff (e.g. app versions)

Andriy Knysh (Cloud Posse)over 7 years ago

and that bucket name looks like this elasticbeanstalk-us-east-1-<accountId>

i5okieover 7 years ago

i have both

Andriy Knysh (Cloud Posse)over 7 years ago

that’s the only thing with Account ID there

i5okieover 7 years ago

i have the buckets that you specify in the terraform module for elb logs, the elb-logs-<accountnumber> bucket that elb spits elb logs into when i enable those. and the elastic-beanstalk-<region>-<accountnumber> buckets

i5okieover 7 years ago

i don’t ask it to change which buckets to put logs into. i’m just going with what it does.

i5okieover 7 years ago

it probably does it when i say to forward logs to cloudwatch

i5okieover 7 years ago

in web console

i5okieover 7 years ago

its aws

i5okieover 7 years ago

just created environment from scratch by manually on web console. same error.

Andriy Knysh (Cloud Posse)over 7 years ago

hmm

Andriy Knysh (Cloud Posse)over 7 years ago

what’s the error?

i5okieover 7 years ago

the reference shows up in cloudformation template

i5okieover 7 years ago

how it gets to that template, i have no friggin idea

i5okieover 7 years ago

im going to delete the stack and try creating it manually again

Andriy Knysh (Cloud Posse)over 7 years ago

yea sounds like you need to delete everything first

i5okieover 7 years ago

which is odd. i deleted the environment, waited for it to disappear from the console, then manually created it. and it ran into the same error.
I guess it kept the cloudformation stack template and kept trying to re-create the stack from template? so odd

i5okieover 7 years ago

ok, time for aws support.
deleted environment. deleted cf template.
applied terraform. same error

Andriy Knysh (Cloud Posse)over 7 years ago

you can try to create a new EB application and deploy the env into it

Andriy Knysh (Cloud Posse)over 7 years ago

(it’s actually a good idea to deploy each env into its own application)

i5okieover 7 years ago

hmm

i5okieover 7 years ago

a new application for each stage?

Andriy Knysh (Cloud Posse)over 7 years ago

yes

i5okieover 7 years ago

hmm

Andriy Knysh (Cloud Posse)over 7 years ago

it’s better for many reasons

Erik Osterman (Cloud Posse)over 7 years ago

Since each stage should be in a separate AWS account to ensure isolation.

Erik Osterman (Cloud Posse)over 7 years ago

Which then necessitates using a separate app.

i5okieover 7 years ago

yeah separate account makes sense

Andriy Knysh (Cloud Posse)over 7 years ago

for now each stage could/should be in a separate app

i5okieover 7 years ago

not using EB is probably also a good idea

Andriy Knysh (Cloud Posse)over 7 years ago

not really 🙂 It’s good at what it does. Never seen the issues like you are having

i5okieover 7 years ago

no not like this. But the hours I’ve spent “trying” to make things work with EB.. i probably could have learned how to use ECS properly.

Andriy Knysh (Cloud Posse)over 7 years ago

maybe

Andriy Knysh (Cloud Posse)over 7 years ago

but EB is actually much simpler in many cases

i5okieover 7 years ago

true very much so

i5okieover 7 years ago

guess what

i5okieover 7 years ago

new application, manual environment setup in web console.

i5okieover 7 years ago

same issue

i5okieover 7 years ago

lol. alright i’ll see what amazon says.

i5okieover 7 years ago

wow. all the time i was pulling my hair out.

i5okieover 7 years ago

just to find the “elb-logs-makeshift” in .ebextensions of the app code itself.

Andriy Knysh (Cloud Posse)over 7 years ago

well, glad you found the issue @i5okie

Andriy Knysh (Cloud Posse)over 7 years ago

how the other 4 env were deployed? without .ebextensions?

i5okieover 7 years ago

different applications

Maarten van der Hoefover 7 years ago

Funny thing I just found out.

It is possible to chain aws_iam_policy_documents by taking the .json-output of one aws_iam_policy_document's as input with the parameter source_json for another aws_iam_policy_document.
This way it's possible to conditionally add statements to a single policy. Context is that not all sources support multiple policies ecr_repo is one of them.

Erik Osterman (Cloud Posse)over 7 years ago

that's a nice trick!

Maarten van der Hoefover 7 years ago

Anyone figured out how to add multiple principles using aws_iam_policy_document ?


 principals {
        identifiers = ["${var.allowed_read_principals}"]
        type        = "AWS"
      }

gets rendered to this json:
"Principal": {
"AWS": "arn

iam::==:root", "arn

iam::=:root", "arn

iam::==:root", "arn

iam::==:root"
}

Maarten van der Hoefover 7 years ago

nevermind, it's something else, replacing var.allowed_read_principals with a self-made list does work

Andriy Knysh (Cloud Posse)over 7 years ago

@Maarten van der Hoef is var.allowed_read_principals a list?

Maarten van der Hoefover 7 years ago

I should have deleted the question 🙂

Maarten van der Hoefover 7 years ago

no it was not in the end

Maarten van der Hoefover 7 years ago

all fixed

Andriy Knysh (Cloud Posse)over 7 years ago

👍️

Andrew Jeffreeover 7 years ago

Greetings. I've been testing out your beanstalk-environment module and I've been having the problem that every-time I run terraform it plans to change everything, even immediately after the initial environment creation etc.

Andrew Jeffreeover 7 years ago

I haven't updated anything or changed any values and from the output I'm seeing Terraform wants to change values from their current value to the exact same value.

Andrew Jeffreeover 7 years ago

In some cases it's removing a setting and then re-adding it later on

Andrew Jeffreeover 7 years ago


      setting.3926968379.name:                                                                              "" => "SSHSourceRestriction"
      setting.3926968379.namespace:                                                                         "" => "aws:autoscaling:launchconfiguration"
      setting.3926968379.resource:                                                                          "" => ""
      setting.3926968379.value:                                                                             "" => "tcp, 22, 22, 0.0.0.0/0"
      setting.502734328.name:                                                                               "SSHSourceRestriction" => ""
      setting.502734328.namespace:                                                                          "aws:autoscaling:launchconfiguration" => ""
      setting.502734328.resource:                                                                           "" => ""
      setting.502734328.value:                                                                              "tcp,22,22,0.0.0.0/0" => ""

Andrew Jeffreeover 7 years ago

in other cases it's the below:

Andrew Jeffreeover 7 years ago


      setting.3402994671.name:                                                                              "Statistic" => "Statistic"
      setting.3402994671.namespace:                                                                         "aws:autoscaling:trigger" => "aws:autoscaling:trigger"
      setting.3402994671.resource:                                                                          "" => ""
      setting.3402994671.value:                                                                             "Average" => "Average"

Erik Osterman (Cloud Posse)over 7 years ago

Hrm... don't believe that should be the case.

Erik Osterman (Cloud Posse)over 7 years ago

@i5okie are you seeing this behavior?

Erik Osterman (Cloud Posse)over 7 years ago

also, what version of terraform and aws provider are you using?

Andriy Knysh (Cloud Posse)over 7 years ago

@Andrew Jeffree please show the output from terraform init and terraform plan, and also the code how you instantiate the module

Andriy Knysh (Cloud Posse)over 7 years ago

for reference, this is how we deployed it before https://github.com/cloudposse/terraform-aws-jenkins/blob/master/main.tf#L14

Andriy Knysh (Cloud Posse)over 7 years ago

(although it probably was tested with older TF version)

Andrew Jeffreeover 7 years ago

Sure gimme a min

Andrew Jeffreeover 7 years ago

So I think at least with the first snippet I pasted it's an issue between how terraform provides the string to beanstalk and how beanstalk returns it.

i5okieover 7 years ago

@Erik Osterman (Cloud Posse) I’m seeing terraform re-do almost every setting. Very similar behaviour to re-setting the Tags even if they were not changed.

i5okieover 7 years ago

it doesn’t replace any environment, or instances. just the setting.xxxxxxxxx.xxxx stuff.

Andrew Jeffreeover 7 years ago

https://gist.github.com/ImperialXT/0cb538bcca4c1596a84676cceaaed63e - terraform init
https://gist.github.com/ImperialXT/13dc04ff25fb3dde8a463255059087f9 - terraform plan
https://gist.github.com/ImperialXT/b4833763f6801054a44e8f8d833f3443 - code

Andrew Jeffreeover 7 years ago

using terraform 0.11.8

Erik Osterman (Cloud Posse)over 7 years ago

Hrmmm my guess is it’s a regression in terraform

Erik Osterman (Cloud Posse)over 7 years ago

We didn’t see this behavior in earlier versions but sounds like it’s a problem now.

Erik Osterman (Cloud Posse)over 7 years ago

It’s probably because maps in golang aren’t stable

Erik Osterman (Cloud Posse)over 7 years ago

Stable in the sense they are not ordered the same between executions

Erik Osterman (Cloud Posse)over 7 years ago

I don’t know how to fix this but will gladly accept any PRs

Andrew Jeffreeover 7 years ago

Haha

Andrew Jeffreeover 7 years ago

Yeah I figure it's either terraform or the beanstalk api.

Andrew Jeffreeover 7 years ago

I've had to hassle AWS to fix a few bugs in the beanstalk api recently

Andrew Jeffreeover 7 years ago

so I wouldn't be surprised if it has further issues.

Andrew Jeffreeover 7 years ago

If I figure it out I'll certainly submit a PR

Andrew Jeffreeover 7 years ago

Thanks for at least confirming I'm not going insane and missing something super obvious 🙂

Erik Osterman (Cloud Posse)over 7 years ago

Ahhh yea…. you’re not going insane 😃

Andrew Jeffreeover 7 years ago

not yet at least 😛

Danover 7 years ago

So for the ssm parameters piece. If I have the ssm stored in AWS and just want to pull that. Do i just use data AWS ssm parameters to get the keys?

Andriy Knysh (Cloud Posse)over 7 years ago

@Dan after you store params in SSM, you can read their values from another TF module: https://github.com/cloudposse/terraform-aws-ssm-parameter-store#simple-read-parameter-example

Andriy Knysh (Cloud Posse)over 7 years ago(edited)

you need to know the names of the params when you write them and then when you read them from other modules

Erik Osterman (Cloud Posse)over 7 years ago(edited)

https://github.com/cloudposse/build-harness/pull/95

Erik Osterman (Cloud Posse)over 7 years ago

Added make target to upgrade all terraform module sources to latest version

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/dependabot/dependabot-core/pull/648

tamskyover 7 years ago

that's amazing -- it's merged already

Erik Osterman (Cloud Posse)over 7 years ago

yea, he's fast!

Erik Osterman (Cloud Posse)over 7 years ago(edited)

i've heard many-a-times a developer say "i have some free time, let me implement that today"

Erik Osterman (Cloud Posse)over 7 years ago

and 2 mo later it's not done.

tamskyover 7 years ago

even more impressive is the non-trivial nature and volume of code. agree on all points

Erik Osterman (Cloud Posse)over 7 years ago(edited)

yea... i thought so too

Erik Osterman (Cloud Posse)over 7 years ago

Some nice modules here: https://github.com/devops-workflow

Alex Somesanover 7 years ago

@Alex Somesan has joined the channel

Gonzalo Peciover 7 years ago

Broad topic, but how are you guys handling multi account AWS deployments?
- Creds per account (interpolated somehow in CI)
- Cred with assume roles?
- some other?

Im just curious to see how others are doing this

lorenover 7 years ago

we use assume role

Andriy Knysh (Cloud Posse)over 7 years ago

@Gonzalo Peci we use separate AWS accounts per stage (prod, staging, dev) and also a separate account (we call it root, although a better name might be identity) where we provision all IAM users and roles. We then use roles to login to the member accounts

Andriy Knysh (Cloud Posse)over 7 years ago

take a look at our reference architectures https://docs.cloudposse.com/reference-architectures/

lorenover 7 years ago

what @Andriy Knysh (Cloud Posse) said 🙂

lorenover 7 years ago(edited)

the terraform provider looks something like this:


provider "aws" {
  profile = "<profile with credential allowed to assume role>"

  assume_role {
    role_arn = "<role arn in target account>"
  }
}

Gonzalo Peciover 7 years ago

@Andriy Knysh (Cloud Posse) we use something similar for our users and assume roles for entering the accounts

Gonzalo Peciover 7 years ago

but for CI I was unsure, we are going the way of per AWS Account->CI User

Gonzalo Peciover 7 years ago

so we are sure that we limit the scope of the blast

Gonzalo Peciover 7 years ago

but at the end of the day we have to interpolate the correct CI User for each stage of the deployment

Gonzalo Peciover 7 years ago

which is meh

Andriy Knysh (Cloud Posse)over 7 years ago

oh, we are working on that now too 🙂

Gonzalo Peciover 7 years ago

and CI has all users, so the blast could be really big

Andriy Knysh (Cloud Posse)over 7 years ago

so we are going to assume roles with MFA by using this tool https://piotrkazmierczak.com/2016/mfa-tokens-in-your-terminal/

Gonzalo Peciover 7 years ago

I saw your <http://prod.name.com|prod.name.com> <http://this.name.com|this.name.com> structure for the AWS Accounts and think its great for the accounts, but I dont think it translates to app deployment

Andriy Knysh (Cloud Posse)over 7 years ago

yes

Andriy Knysh (Cloud Posse)over 7 years ago

we just added it to geodesic https://github.com/cloudposse/geodesic/pull/248

Gonzalo Peciover 7 years ago

Interesting

Andriy Knysh (Cloud Posse)over 7 years ago

geodesic is our container which we use to login to the accounts and provision resources

Gonzalo Peciover 7 years ago

Yep

Andriy Knysh (Cloud Posse)over 7 years ago(edited)

so we create a CI/CD user per stage (prod, staging, dev, etc) and then use oath-toolkit to get the MFA token and then assume role to login to the account

Gonzalo Peciover 7 years ago

but doesnt generating the token on your computer defy the objective of MFA? as you have the MFA gen and AWS creds on the same place

Gonzalo Peciover 7 years ago

In any case, I believe that might be a different converesation

Andriy Knysh (Cloud Posse)over 7 years ago

yea, it’s a long conversation 🙂

Gonzalo Peciover 7 years ago

My original quesiton was more about, how do you deploy apps to the different environments with multiple AWS accounts

Andriy Knysh (Cloud Posse)over 7 years ago

ah, it’s easy 🙂 let me show

Gonzalo Peciover 7 years ago

In your case its Kube, so you have to pass the kube endpoint and creds for each env

Gonzalo Peciover 7 years ago

depending on the branch or stage in the pipeline

Andriy Knysh (Cloud Posse)over 7 years ago

so we have a collection of TF modules which we use for all stages/environments. The modules have no identity (you can say they are just templates) https://github.com/cloudposse/terraform-root-modules

Andriy Knysh (Cloud Posse)over 7 years ago

then in the geodesic shell for a member account, we pull the resources we need

Gonzalo Peciover 7 years ago

1 moment, that part I follow

Gonzalo Peciover 7 years ago

As I follow your cloudposse project, thanks a lot btw for some of the ideas/concepts

Andriy Knysh (Cloud Posse)over 7 years ago

https://github.com/cloudposse/prod.cloudposse.co/blob/master/Dockerfile#L1

Andriy Knysh (Cloud Posse)over 7 years ago

https://github.com/cloudposse/prod.cloudposse.co/blob/master/Dockerfile#L36

Andriy Knysh (Cloud Posse)over 7 years ago

no problem

Andriy Knysh (Cloud Posse)over 7 years ago

so the identity comes into play in the accounts repos

Andriy Knysh (Cloud Posse)over 7 years ago

which we use from geodesic shells per account

Andriy Knysh (Cloud Posse)over 7 years ago

and we store all credentials in AWS SSM and use chamber to read them

Andriy Knysh (Cloud Posse)over 7 years ago

https://docs.cloudposse.com/tools/chamber/

Gonzalo Peciover 7 years ago

Yeah we use a similar workflow for that part

Gonzalo Peciover 7 years ago

with chamber

Gonzalo Peciover 7 years ago

But those are a repo per env, so those are a bit different I guess.
After you have KOPS and you your PaaS in a way, lets say now you have app1 how do you deploy it?

Gonzalo Peciover 7 years ago

EG: some people have a var set (all fake vars, just used as an example)
KOPS_ENDPOINT
KOPS_USER
KOPS_PASS
then on each stage of the pipeline, or branch depending on how you deploy, they interpolate the right values there.

Gonzalo Peciover 7 years ago

even if they are on chamber, you have to store either 3 AWS accounts for chamber, or 1 AWS account for chamber, with multiple secrets

Andriy Knysh (Cloud Posse)over 7 years ago

we use CodeFresh pipelines to deploy apps to k8s clusters

Andriy Knysh (Cloud Posse)over 7 years ago

and we use geodesic which has chamber inside

Andriy Knysh (Cloud Posse)over 7 years ago

in the pipelines (per account/stage) we read the ENV vars from SSM and CodeFresh applies them

Gonzalo Peciover 7 years ago

so CodeFresh has permissions to access each SSM for each env?

Andriy Knysh (Cloud Posse)over 7 years ago

Codefresh uses containers per step, so it’s easy to use geodesic there

Gonzalo Peciover 7 years ago

Ah right

Gonzalo Peciover 7 years ago

and you have the creds on each container

Gonzalo Peciover 7 years ago

Nice

Gonzalo Peciover 7 years ago

Thanks for your answer, It gave me some ideas

Andriy Knysh (Cloud Posse)over 7 years ago

you welcome. I believe @Erik Osterman (Cloud Posse) has a lot more to add to this

Gonzalo Peciover 7 years ago

We are going a different way right now, we will a have a CI AWS Account, whith a CI chamber, that we can get the envs for each environment on each stage of the deploy

Gonzalo Peciover 7 years ago

But was looking for alternatives/improvements

Gonzalo Peciover 7 years ago

BTW Im trying to fix https://github.com/segmentio/chamber/pull/70 so it finally merges and we get per env/custom paths

Andriy Knysh (Cloud Posse)over 7 years ago

interesting

Andriy Knysh (Cloud Posse)over 7 years ago(edited)

since chamber can accept multiple services when reading (e.g. chamber read service1 service2 key) we use multiple services to override the default values (if needed)

Andriy Knysh (Cloud Posse)over 7 years ago

e.g. chamber read kops app1 db_password

Gonzalo Peciover 7 years ago

Yeah, we use something like chamber exec ci-development ci-app1

Gonzalo Peciover 7 years ago

But I want to have a cleaner SSM, as the interface sucks a bit

Gonzalo Peciover 7 years ago

so having /devel/ci or similar will be ideal

Gonzalo Peciover 7 years ago

or something like /external/thisprovider for any sahred keys we want to actually share

Gonzalo Peciover 7 years ago(edited)

for CI of apps we are currently moving to something like:


(CI has CI AWS Account creds)
chamber exec aws-development -- 
(CI AWS Account is overwritten by Dev AWS Account)
chamber exec ci-app1 -- example command

Gonzalo Peciover 7 years ago

aws-development secret lives in CI Acccount
ci-app1 secret lives in Dev Account

Andriy Knysh (Cloud Posse)over 7 years ago

That’s nice

Andriy Knysh (Cloud Posse)over 7 years ago

We use the same secret names for all accounts since SSM is per account, and this way all our code remains the same

Gonzalo Peciover 7 years ago

yeah we would only interpolate the first chamber for the same reason, after that all SSM secerts are the same across environments.
we thought about moving ci-app1 to the CI Account and changing it to something like /dev/ci-app1 or so, so we can use the fact that chamber can read multiple secrets at once, but ultimately we prefer to have less interpolation of environment names etc

Andrew Jeffreeover 7 years ago

So my customer wants to pay someone to fix that bug around Terraform and beanstalk settings, that we discussed here last week.

Andrew Jeffreeover 7 years ago

I'm not sure that even if someone can figure out where the bug is and patch it they'll be able to get it merged in a timely manner.

Andrew Jeffreeover 7 years ago

comments/thoughts/suggestions appreciated.

Andriy Knysh (Cloud Posse)over 7 years ago

@Andrew Jeffree are you asking about this one https://sweetops.slack.com/archives/CB6GHNLG0/p1534814857000100

Andriy Knysh (Cloud Posse)over 7 years ago

i’ll take a look at it

Danover 7 years ago

If you created a vpc in one tf file... How do you use another tf file to find that vpc and those subnets to deploy resources too

Andriy Knysh (Cloud Posse)over 7 years ago

@Dan Terraform works per-folder, meaning everything in the folder will be used. So if you create a VPC in one file, you can use it in all other files in the same folder

Andriy Knysh (Cloud Posse)over 7 years ago

Here is an example

Andriy Knysh (Cloud Posse)over 7 years ago

https://github.com/cloudposse/terraform-root-modules/blob/master/aws/backing-services/vpc.tf

Danover 7 years ago

Im using separate folders

Andriy Knysh (Cloud Posse)over 7 years ago(edited)

Terraform also supports modules in local directories, identified by a relative path starting with either ./ or ../. Such local modules are useful to organize code in more complex repositories, and are described in more detail in Creating Modules. https://www.terraform.io/docs/modules/usage.html#source

Andriy Knysh (Cloud Posse)over 7 years ago

^ one way of doing it

Andriy Knysh (Cloud Posse)over 7 years ago

another way is to look up the VPC using data sources - you create it in one folder and do terraform apply, then in another module in another folder you look it up

Andriy Knysh (Cloud Posse)over 7 years ago

example:

Andriy Knysh (Cloud Posse)over 7 years ago

https://github.com/cloudposse/terraform-aws-kops-metadata/blob/master/main.tf#L3

Danover 7 years ago

I'll take a look thank you!

Maarten van der Hoefover 7 years ago

Anyone ever used: aws_iam_account_alias ? What is the practical use of it ?

Andriy Knysh (Cloud Posse)over 7 years ago

it’s a friendly name for the account instead of account ID (which is not easy to remember)

Andriy Knysh (Cloud Posse)over 7 years ago

so the URL where you login would be like this:

Andriy Knysh (Cloud Posse)over 7 years ago

https://my-company.signin.aws.amazon.com/console

Maarten van der Hoefover 7 years ago

hm that's quite nice actually

Maarten van der Hoefover 7 years ago

but also an information give-away in some cases

Maarten van der Hoefover 7 years ago

🙂

Andriy Knysh (Cloud Posse)over 7 years ago

yep

Andriy Knysh (Cloud Posse)over 7 years ago

Here is a Terraform trick we used recently.
Looks simple, but we really did not know how to do it before 🙂
It might help somebody.

In some cases, you have a TF module and want to provide some settings in a list or map.
And you have a conditional variable (let’s say var.condition1) which changes the settings for the module.


locals {

  settings1 = [
    {
      name = "1a"
      value = "1a"
    },
    {
      name = "1b"
      value = "1b"
    }
  ]

  settings2 = [
    {
      name = "2a"
      value = "2b"
    },
    {
      name = "1b"
      value = "1b"
    }
  ]
}

module "example" {
	settings = "${var.condition1 ? local.settings1 : local.settings2}"
}

won’t work because Terraform does not support list and map in conditional expressions (maybe V2 will do it better, but we don’t really know).
So here is the slice pattern 🙂 (for the lack of a better name)


locals {
	settings = [
	  [
	    {
	      name = "1a"
	      value = "1a"
	    },
	    {
	      name = "1b"
	      value = "1b"
	    }
	  ],
	  [
	    {
	      name = "2a"
	      value = "2b"
	    },
	    {
	      name = "1b"
	      value = "1b"
	    }
	  ]
  	]

  	from_index    = "${var.condition1 ? 0 : 1}"
  	to_index         = "${var.condition1 ? 1 : 2}"
  	settings_final = "${slice(local.settings, local.from_index, local.to_index)}"
}

module "example" {
	settings = "${local.settings_final}"
}

Andriy Knysh (Cloud Posse)over 7 years ago

Implemented here as an example:
https://github.com/cloudposse/terraform-aws-dynamodb/blob/master/main.tf#L11

lorenover 7 years ago

another way of doing this is to put the conditional eval in the index of the map... map[condition ? true : false]

lorenover 7 years ago

where true and false are the key in your lookup dictionary

Andriy Knysh (Cloud Posse)over 7 years ago(edited)

Yep, for maps it will work as well

Andriy Knysh (Cloud Posse)over 7 years ago(edited)

The slice pattern works better if you need to remove some settings based on condition (and you can’t send empty or default values to the module)

Maarten van der Hoefover 7 years ago

love it! 🙂

Maarten van der Hoefover 7 years ago

https://www.reddit.com/r/Terraform/comments/99zri8/gke_regional_cluster_deployment_defining_zones/ you can publish your slice pattern here 🙂

Andriy Knysh (Cloud Posse)over 7 years ago

🙂

Danover 7 years ago

@Andriy Knysh (Cloud Posse) i cant get this to work^

Andriy Knysh (Cloud Posse)over 7 years ago

what’s the error?

Danover 7 years ago

Andriy Knysh (Cloud Posse)over 7 years ago

so, since you are using data sources to lookup the VPC and subnets, two possible issues here:

Andriy Knysh (Cloud Posse)over 7 years ago

1. Did you already provision those VPC and subnets

Danover 7 years ago

^yes

Danover 7 years ago

the tags match

Andriy Knysh (Cloud Posse)over 7 years ago

2. Did you specify the correct filters to look them up?

Danover 7 years ago

the filter im not sure about

Danover 7 years ago

this makes sense ot me

Danover 7 years ago

filter {
name = "tag:Name"
Values = "${var.AD-Private-Subnet2}"

Danover 7 years ago

filte for

Danover 7 years ago

filter for

Danover 7 years ago

tag name

Danover 7 years ago

with the value of this variable

Danover 7 years ago

oh i dont want the tag name tho i want the subnet

Danover 7 years ago

hmm

Andriy Knysh (Cloud Posse)over 7 years ago

you can use tags, but you need to create them with the same tags

Danover 7 years ago

yes thats what i did

Danover 7 years ago

the tags match whats in aws

Danover 7 years ago

and match the other tf file thats creating them

Danover 7 years ago

the other tf file is in sep folder

Andriy Knysh (Cloud Posse)over 7 years ago

I’ll take a look in 30 mins (in a meeting now)

Danover 7 years ago

no rush

Danover 7 years ago

thank you so much!

Danover 7 years ago

sorry im still learning!

Andriy Knysh (Cloud Posse)over 7 years ago(edited)

@Dan to lookup VPC, you can use id and filters https://www.terraform.io/docs/providers/aws/d/vpc.html

Andriy Knysh (Cloud Posse)over 7 years ago

but for subnets, you use tags https://www.terraform.io/docs/providers/aws/d/subnet_ids.html

Andriy Knysh (Cloud Posse)over 7 years ago

and vpc_id is required

Danover 7 years ago

yea i was tyring to avoid the id due to the fact it changes each time a vpc is created

Danover 7 years ago

i was looking for a way to have it pass through the current environment

Andriy Knysh (Cloud Posse)over 7 years ago

you can use the id from the VPC you look up data.vpc.my_vpc.id

Andriy Knysh (Cloud Posse)over 7 years ago

https://www.terraform.io/docs/providers/aws/d/vpc.html#id

Andriy Knysh (Cloud Posse)over 7 years ago

All of the argument attributes except filter blocks are also exported as result attributes

Danover 7 years ago

without inputing the varibale

Danover 7 years ago

variable

Andriy Knysh (Cloud Posse)over 7 years ago

for subnets:

Andriy Knysh (Cloud Posse)over 7 years ago

vpc_id = “${data.aws_vpc.my_vpc.id}”

Andriy Knysh (Cloud Posse)over 7 years ago

it will first lookup the VPC and then use its ID (and tags if you want) to lookup the subnets

Andriy Knysh (Cloud Posse)over 7 years ago(edited)

(you prob don’t need the tags, you can get all subnets from the VPC by its ID)

Danover 7 years ago

will i need this variable "AD-Private-Subnet1" {}

Erik Osterman (Cloud Posse)over 7 years ago

(nitpick, consider using all lower case and and underscores - it’s the most common convention for terraform resource names)

Danover 7 years ago

sorry!

Danover 7 years ago

is there a way to do it without inputing the variable?

Danover 7 years ago

im missing somthing here

Andriy Knysh (Cloud Posse)over 7 years ago

@Dan you can explain what you want to achieve and paste your complete code here (or DM me). (sorry, don’t want it to be http://xyproblem.info :))

Andriy Knysh (Cloud Posse)over 7 years ago(edited)

@Andrew Jeffree so I tested terraform-aws-elastic-beanstalk-environment and yes, it does re-create all settings on each plan/apply


      setting.1039973377.name:               "InstancePort" => "InstancePort"
      setting.1039973377.namespace:          "aws:elb:listener:22" => "aws:elb:listener:22"
      setting.1039973377.resource:           "" => ""
      setting.1039973377.value:              "22" => "22"
      setting.1119692372.name:               "" => "ListenerEnabled"
      setting.1119692372.namespace:          "" => "aws:elbv2:listener:443"
      setting.1119692372.resource:           "" => ""
      setting.1119692372.value:              "" => "false"
      setting.1136119684.name:               "RootVolumeSize" => "RootVolumeSize"
      setting.1136119684.namespace:          "aws:autoscaling:launchconfiguration" => "aws:autoscaling:launchconfiguration"
      setting.1136119684.resource:           "" => ""
      setting.1136119684.value:              "8" => "8"
      setting.1201312680.name:               "ListenerEnabled" => "ListenerEnabled"
      setting.1201312680.namespace:          "aws:elb:listener:443" => "aws:elb:listener:443"
      setting.1201312680.resource:           "" => ""
      setting.1201312680.value:              "false" => "false"

Andriy Knysh (Cloud Posse)over 7 years ago

This feature/bug was present for years and still is not fixed:

https://github.com/hashicorp/terraform/issues/6729
https://github.com/terraform-providers/terraform-provider-aws/pull/901
https://github.com/hashicorp/terraform/issues/6729
https://github.com/hashicorp/terraform/issues/6257
https://github.com/terraform-providers/terraform-provider-aws/issues/280
https://github.com/hashicorp/terraform/issues/11056
https://github.com/terraform-providers/terraform-provider-aws/issues/461

Andriy Knysh (Cloud Posse)over 7 years ago

nobody is sure who’s bug it is, Terraform or AWS

Andriy Knysh (Cloud Posse)over 7 years ago

(i tested some ideas from the links above, nothing worked)

Andriy Knysh (Cloud Posse)over 7 years ago

the only possible solution is to add this:

Andriy Knysh (Cloud Posse)over 7 years ago

lifecycle {
	ignore_changes = ["setting"]
}

Andriy Knysh (Cloud Posse)over 7 years ago

but it’s a hack since it will not update the env if you update any of the settings

Erik Osterman (Cloud Posse)over 7 years ago

@Andriy Knysh (Cloud Posse) can you open a new issue here: https://github.com/cloudposse/terraform-aws-elastic-beanstalk-environment/issues

Erik Osterman (Cloud Posse)over 7 years ago

with all your research above?

Erik Osterman (Cloud Posse)over 7 years ago

that way we can track it since this comes up quite frequently

Erik Osterman (Cloud Posse)over 7 years ago

hate for you to have to explain it everytime

Erik Osterman (Cloud Posse)over 7 years ago

😃

Andrew Jeffreeover 7 years ago

@Andriy Knysh (Cloud Posse) yep, am aware it's an open issue in multiple places. For now we're ignoring changes to settings in the lifecycle but as you mentioned it's a hack. The customer wants to pay someone to fix it, but I don't like their odds.

Erik Osterman (Cloud Posse)over 7 years ago

sounds like it would be a difficult thing to give an estimate on, so it would probably need to be T&E

Erik Osterman (Cloud Posse)over 7 years ago

is anyone using private submodules with codebuild?

Andriy Knysh (Cloud Posse)over 7 years ago(edited)

Regarding terraform-aws-elastic-beanstalk-environment recreating the settings all the time, here what I think is happening:

- Terraform sends all settings to AWS, but some of them are not relevant to the environment you are deploying
- Elastic Beanstalk accepts all settings, applies the relevant ones, and throws away the rest
- Next time Terraform asks about the settings, Elastic Beanstalk returns a subset of the values and probably in different order
- Terraform can’t decide/calculate if the settings are the same - they sure look different (and would require an advanced algorithm to determine if they are the same)
- Terraform assigns new ID to the entire array of settings and tries to recreate all of them
- Elastic Beanstalk accepts the settings, applies the relevant ones, and throws away the rest - the cycle repeats

What’s a possible solution?
Introduce var.settings (list of maps) to be able to provide all the required settings from outside of the module.
It might work, but in practice would be very difficult to know all the needed settings and tedious to implement.

Andriy Knysh (Cloud Posse)over 7 years ago

@Andrew Jeffree ^

Andriy Knysh (Cloud Posse)over 7 years ago

(opened an issue to track any progress on this https://github.com/cloudposse/terraform-aws-elastic-beanstalk-environment/issues/43)

Andrew Jeffreeover 7 years ago

interesting

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/cloudposse/terraform-null-label/pull/38

Erik Osterman (Cloud Posse)over 7 years ago

@Jamie Nelson (BitFlight) any insights?

Erik Osterman (Cloud Posse)over 7 years ago

@Andriy Knysh (Cloud Posse)

Andriy Knysh (Cloud Posse)over 7 years ago

checking

Andriy Knysh (Cloud Posse)over 7 years ago

we can override delimiter for any label. If it’s not provided and context provided, it will be taken from the context. If nothing is provided, the default will be used

Andriy Knysh (Cloud Posse)over 7 years ago

i’m reviewing the PR

Jamie Nelson (BitFlight)over 7 years ago

Its an easy fix 🙂

Jamie Nelson (BitFlight)over 7 years ago

I see that in the PR it has been broken out into less condense parts so that each step can be explained.

Andriy Knysh (Cloud Posse)over 7 years ago

@mrwacky yea looks like it will be resolved in 0.12, thanks