#office-hours - September 2020

@here office hours is starting in 30 minutes! Remember to post your questions here.

i'm curious to know what the overall strategy is for handling the new version of the aws provider.

Erik Osterman (Cloud Posse) has joined Public "Office Hours"

Jeff Wozniak has joined Public "Office Hours"

Anton Shakh has joined Public "Office Hours"

Vlad Ionescu has joined Public "Office Hours"

Soham Jadiya has joined Public "Office Hours"

Sheldon Hull has joined Public "Office Hours"

17133029948 has joined Public "Office Hours"

Ian Bartholomew has joined Public "Office Hours"

@here our devops #office-hours starting now! join us to talk shop https://cloudposse.zoom.us/j/508587304

Andrey Nazarov has joined Public "Office Hours"

Michael Holt has joined Public "Office Hours"

Kareem Shahin has joined Public "Office Hours"

Eric Berg has joined Public "Office Hours"

Neil Gealy has joined Public "Office Hours"

nat lie has joined Public "Office Hours"

Adam Crown has joined Public "Office Hours"

Hugo Samayoa has joined Public "Office Hours"

Matt Gowie has joined Public "Office Hours"

Jawwad Yunus has joined Public "Office Hours"

Isa Aguilar has joined Public "Office Hours"

James Connolly has joined Public "Office Hours"

Babajide Hassan has joined Public "Office Hours"

Sean Conley has joined Public "Office Hours"

Marc Tamsky has joined Public "Office Hours"

Nick James has joined Public "Office Hours"

Eric Berg has joined Public "Office Hours"

John D has joined Public "Office Hours"

For versioning this is nice.

I have this running right now in a similar manner. I use gitversion that calculates the semver based on branching. If you do a breaking change you manually set the tag to bump otherwise all the patch versions generate pre-release draft releases on branch and normal minor patch.

Sheldon Hull has joined Public "Office Hours"

Andrew Roth has joined Public "Office Hours"

Olivier Chaine has joined Public "Office Hours"

Zadkiel AHARONIAN has joined Public "Office Hours"

I adopted the null label stuff and love it. All my resources have randomized pet names with standard prefix.
I have wanted to figure out the null label stuff so I’m excited to try this. The submodules having null label has confused me but this looks like it will help with this problem

Nothing like provisioning a bunch of servers and my coworkers seeing “snarky-puppy-rds-foobar” 😆

Sheldon Hull has joined Public "Office Hours"

Rube goldberg lol. Totally.

I really like this solution for DB operations too: https://aws.amazon.com/blogs/database/building-a-cross-account-continuous-delivery-pipeline-for-database-migrations/

Interesting. You are saying on IAM Service accounts that you wouldn’t manage this user provisioning through a master terraform security repo for example? How do you setup the user provisioning to be IAC at that point?

Can we get a picture of this diagram and mind addressing sometime why terraform is only on foundational, when i’d guess that it has impact in all of the tiers

Sheldon Hull has joined Public "Office Hours"

If we have time at the end, I want to know what others are doing to provision their IAM user and defined role/groups across accounts via code. Are you using terraform pull request driven workflow, lambda with json in s3 buckets, etc?

https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs :

To use these credentials with the Kubernetes provider, they can be interpolated into the respective attributes of the Kubernetes provider configuration block.

IMPORTANT WARNING _When using interpolation to pass credentials to the Kubernetes provider from other resources, these resources SHOULD NOT be created in the same apply operation where Kubernetes provider resources are also used. This will lead to intermittent and unpredictable errors which are hard to debug and diagnose. The root issue lies with the order in which Terraform itself evaluates the provider blocks vs. actual resources. Please refer to this section of Terraform docs for further explanation._

The best-practice in this case is to ensure that the cluster itself and the Kubernetes provider resources are managed with separate apply operations. Data-sources can be used to convey values between the two stages as needed.

GKE module also has some workarounds:
https://github.com/terraform-google-modules/terraform-google-kubernetes-engine/blob/4d33759bb6e913586f9d0e2705d6eb2fb6c43a23/cluster.tf#L252

Managing terraform workspaces with terraformenterprise provider (import from yaml perhaps) is the only scalable way to do this

You have to manage terraform workspaces via code at that point

The challenge with managing terraform with terraform is pretty much that on free tier there is no additional levels of permission for folks. You can’t have readers, just admins. So you have to bump up the pay and then ensure that workspaces are NOT allowed to be created by any other method than code, or I feel it’s a lost cause to ensure this is managed consistently.

Kinda frustrating but i don’t see how you can effectively manage manual + automated workspaces in a solid way if you don’t just have it all managed by a service account instead.

^ this sounds like an awesome topic for next #office-hours . What changes come from managing 2-3 workspaces, 10s of workspaces, 100s, 1000s

👉️ New Zoom Recording from our Office Hours session on 2020-09-02 is now available.

Following up this multi-level or multi-tier structure you showed. Having this stuff decoupled means that you define different pipelines for them. Is this like a pipeline per level? Separate repo for each? Or it might be several pipelines within the same level? By the pipeline I essentially mean terraform apply command which applies a set of modules. What is the CloudPosse approach?

How do you deal with different chicken-and-egg scenarios? Like you deploy Gitlab and its runners as level 3, but you need runners to run terraform commands on level 1 or even to deploy this Gitlab:)

@here office hours is starting in 30 minutes! Remember to post your questions here.

Hi, I am trying to create a security group with

module "app_db_sg" {
  source = "terraform-aws-modules/security-group/aws//modules/postgresql"

  name   = "${local.environment}-db-sg"
  vpc_id = module.vpc.vpc_id

  description = "Security group that controls access to DB"
  use_name_prefix = false

  computed_ingress_with_source_security_group_id = [
    {
      rule                     = "postgresql-tcp"
      source_security_group_id = module.app_beanstalk_environment[0].security_group_id
    }
  ]
  number_of_computed_ingress_with_source_security_group_id = 1

}

but I am getting One of ['cidr_blocks', 'ipv6_cidr_blocks', 'self', 'source_security_group_id', 'prefix_list_ids'] must be set to create an AWS Security Group Rule. I just want to know how I can check that the value returned by module.app_beanstalk_environment[0].security_group_id is right. I am using tfctl so terraform console does not work for me (or I am not sure how to use it).

@here our devops #office-hours starting now! join us to talk shop https://cloudposse.zoom.us/j/508587304

Gitpod did this recently, a full setup of their EKS environment and export of the terraform plan and more with a single docker run. I was pretty impressed, esp as never having used helm it was amazing to see it all pretty much just work 🙂

David Karlsen has joined Public "Office Hours"

raphael francis has joined Public "Office Hours"

Andrew Roth has joined Public "Office Hours"

Erik Osterman (Cloud Posse) has joined Public "Office Hours"

Vlad Ionescu has joined Public "Office Hours"

Sheldon Hull has joined Public "Office Hours"

Question:
• GitHub Actions —> Any easy way to trigger an action on demand?
• GitHub Actions --> Any update on any dashboard/centralized reporting for actions that have been run in an organization?

Anton Shakh has joined Public "Office Hours"

Isa Aguilar has joined Public "Office Hours"

Adam Crown has joined Public "Office Hours"

Ian Bartholomew has joined Public "Office Hours"

PePe Amengual has joined Public "Office Hours"

Kareem Shahin has joined Public "Office Hours"

GitHub Actions —> Any easy way to trigger an action on demand?

Yup, you can triger them manually now. They have a button! https://github.blog/changelog/2020-07-06-github-actions-manual-triggers-with-workflow_dispatch/

Victor Ma has joined Public "Office Hours"

Babajide Hassan has joined Public "Office Hours"

Taras Dyshkant has joined Public "Office Hours"

Andrew Elkins has joined Public "Office Hours"

Robert Horrox has joined Public "Office Hours"

Christopher Picht has joined Public "Office Hours"

Chef got acquired. I think that’s a big change 🙂

eeic berg has joined Public "Office Hours"

sri has joined Public "Office Hours"

Zadkiel AHARONIAN has joined Public "Office Hours"

Pedro Torres has joined Public "Office Hours"

There's also https://github.com/philips-labs/terraform-aws-github-runner

Michael Martin has joined Public "Office Hours"

Sri has joined Public "Office Hours"

Eric Berg has joined Public "Office Hours"

Maged Abdelmoeti has joined Public "Office Hours"

@Erik Osterman (Cloud Posse) can you share the parsing logic of the yaml? I’ve not found many good “flatten” examples. That part would be useful in my own work if possible

Such a great community. Always enjoy chatting with all of you 🙂

^AWS SaaS Factory presentation on programatic Control Planes.

Screenshot from the above video

Haven’t got a chance to participate today. Looking forward to watching the recorded version

Full disclosure: I'll miss office-hours next week as I have a conflict

👉️ New Zoom Recording from our Office Hours session on 2020-09-09 is now available.

I’m watching the latest episode. Regarding of version-checker , Lens had the same functionality, there were some bugs in it, but it was more or less usable. Don’t know its current state though.

As of fat module vs decomposition I would join a @Vlad Ionescu (he/him)’s camp. In the past we struggled a lot managing everything via just one tf apply. It looked cool at first that you theoretically could fire up all the things from the ground up. But then came the pain. Mostly it came firstly, as Vlad pointed out, from fundamental changes in modules and secondly - from unstable third-party or home-grown tf providers. And we’d encountered spoiled state quite often until we decomposed things in a way similar to CloudPosse’s 4-layered approach.

But, yes, it’s a matter of your use cases. For some fat modules might work perfectly.

Just my two cents on this.

@here office hours is starting in 30 minutes! Remember to post your questions here.

Erik Osterman (Cloud Posse) has joined Public "Office Hours"

Taras Dyshkant has joined Public "Office Hours"

Vicken Simonian has joined Public "Office Hours"

Giles Billenness has joined Public "Office Hours"

Adam Crown has joined Public "Office Hours"

Neil Gealy has joined Public "Office Hours"

Andrew Roth has joined Public "Office Hours"

@Andrew Roth https://github.com/hashicorp/terraform/issues/25631#issuecomment-693523596

Paul Obalonye has joined Public "Office Hours"

@Jeremy G (Cloud Posse) what's the link to the cycle issue you reported? ☝️

@here starting now

@Erik Osterman (Cloud Posse) https://github.com/hashicorp/terraform/issues/26226

Matt Gowie has joined Public "Office Hours"

Alex Siegman has joined Public "Office Hours"

Anyone use https://github.com/jckuester/awsweeper ? Is there a better tool out there for blanking out an AWS account? When I am trying to make sure that my Code creates all of the Infrastructure I have, I find destroying to be nearly as important as creating.

Brian Tai has joined Public "Office Hours"

Christopher Picht has joined Public "Office Hours"

Jeremy CloudPosse has joined Public "Office Hours"

Ian Bartholomew has joined Public "Office Hours"

Paul Obalonye has joined Public "Office Hours"

David Lundgren has joined Public "Office Hours"

Sri has joined Public "Office Hours"

Kareem Shahin has joined Public "Office Hours"

Oludahun Bade-Ajidahun has joined Public "Office Hours"

Robert Horrox has joined Public "Office Hours"

Andrew Elkins has joined Public "Office Hours"

Jim Park has joined Public "Office Hours"

Sri has joined Public "Office Hours"

azam has joined Public "Office Hours"

Anton Shakh has joined Public "Office Hours"

My question here might be worth discussing if we need a topic: https://github.com/cloudposse/terraform-aws-acm-request-certificate/pull/25#issuecomment-693419593

Sri has joined Public "Office Hours"

Anybody use a bot to merge code? I’m wondering what that looks like under the hood

Marc Tamsky has joined Public "Office Hours"

Omer Sen has joined Public "Office Hours"

alejandro chacon has joined Public "Office Hours"

Zadkiel AHARONIAN has joined Public "Office Hours"

pepe amengual has joined Public "Office Hours"

Isa Aguilar has joined Public "Office Hours"

Adam Blackwell has joined Public "Office Hours"

ivan pedro has joined Public "Office Hours"

Eric Berg has joined Public "Office Hours"

Juan Soto has joined Public "Office Hours"

Blaise pabon has joined Public "Office Hours"

sri has joined Public "Office Hours"

Ian Bartholomew has joined Public "Office Hours"

Side question about IAM access, we no longer use cross account assumptions for console access and instead use OneLogin. I'm curious if this diverges from the CloudPosse reference architecture and if others do something similar with Okta or OneLogin. If there are downsides that I'm not aware of, I'd love to know about them.

(in these sample screenshots I, as an SRE, only have admin and readonly for each account, but developers often have various other roles)

Second side question: aws-nuke was mentioned in the beginning of office hours, which I know is used in this reference architecture:

https://github.com/cloudposse/testing.cloudposse.co/blob/master/.github/workflows/aws-nuke.yml
https://github.com/cloudposse/testing.cloudposse.co/blob/4d02425da9a97bb8e7cbe61987d511f0ed6d1e4c/.github/workflows/aws-nuke.yml

I'm curious if others use this, but chose to run the workflow on private runners and use an IAM role to avoid needing to give AWS credentials to Github and if there are cons to the second approach.

👉️ New Zoom Recording from our Office Hours session on 2020-09-16 is now available.

hey, apparently all the docs arent letting me do any of this due to the provider if im reading it correct

action {
name = "${var.application_name}-ecs-worker"
category = "Deploy"
owner = "AWS"
provider = "ECS"
input_artifacts = ["task"]
version = "1"
configuration = {
ClusterName = aws_ecs_cluster.ecs_cluster.name
ServiceName = "${var.application_name}-worker"
# ActionMode = "REPLACE_ON_FAILURE"
# OutputFileName = "CreateStackOutput.json"
# StackName = "MyStack"
# ImageDefinitionsFile = "worker-imagedefinitions.json"
# TaskDefinitionTemplateArtifact = "task"
# TaskDefinitionTemplatePath = "worker-imagedefinitions.json"
}
}

id love to talk about this if you are open to it today

@here office hours is starting in 30 minutes! Remember to post your questions here.

Erik Osterman (Cloud Posse) has joined Public "Office Hours"

Adam Crown has joined Public "Office Hours"

Vlad Ionescu has joined Public "Office Hours"

Jeremy CloudPosse has joined Public "Office Hours"

pepe amengual has joined Public "Office Hours"

Fernando Castillo has joined Public "Office Hours"

Anere Faithful has joined Public "Office Hours"

Marcin Brański has joined Public "Office Hours"

Patrick Joyce has joined Public "Office Hours"

Michael Londeen has joined Public "Office Hours"

Vitali Bystritski has joined Public "Office Hours"

Christian Roy has joined Public "Office Hours"

Justin Ober has joined Public "Office Hours"

Matt Gowie has joined Public "Office Hours"

Sri has joined Public "Office Hours"

Kareem Shahin has joined Public "Office Hours"

vicken has joined Public "Office Hours"

Brian Tai has joined Public "Office Hours"

Oliver Schoenborn has joined Public "Office Hours"

Andrey Nazarov has joined Public "Office Hours"

Nigel Kirby has joined Public "Office Hours"

David Lundgren has joined Public "Office Hours"

Topic I’m interested in if we have time: Grafana users, have you found any useful community dashboards that you would recommend / what is the general opinion about community dashboards. Alternatively, how do you manage your Grafana dashboards? Should it only be codified + read only in the UI

Anere Faithful has joined Public "Office Hours"

Eric Berg has joined Public "Office Hours"

Vlad Ionescu has joined Public "Office Hours"

Michael Holt has joined Public "Office Hours"

Zadkiel AHARONIAN has joined Public "Office Hours"

Nick James has joined Public "Office Hours"

Christopher Picht has joined Public "Office Hours"

Marc Tamsky has joined Public "Office Hours"

For downscaling k8s deployments on a schedule: https://github.com/hjacobs/kube-downscaler

Neil Gealy has joined Public "Office Hours"

Isa Aguilar has joined Public "Office Hours"

Jim Park has joined Public "Office Hours"

Juan Soto has joined Public "Office Hours"

Laurence Giglio has joined Public "Office Hours"

Andrew Roth has joined Public "Office Hours"

sri has joined Public "Office Hours"

waf?

ahhhh!! sorry, we ran out of time today @Patrick Joyce

we'll get to WAF next wednesday

👉️ New Zoom Recording from our Office Hours session on 2020-09-23 is now available.

Spent like 6 hours figuring this out. Thought it might be useful for someone else…

Idempotently create a Personal Access Token for a user in GitLab running in Kubernetes

https://gist.github.com/RothAndrew/e1c8d3e183293d3fadb6cdbf64a3475d

Guys? I have a question ... What are You using for logging in k8s ( forwarders ) ?
I am using Loki & Opendistro ( Elasticsearch ), problem is that I want to use Fluentbit + FluentD combo ( tls forward, exposed separate loadbalancer ), what is problem that there is not complete & matured solution for it, which is weird ..
• fluentbit -> there isnt support for hotreloading, nor API endpoints, signaling option in application
• fluentd -> there is no good helm chart with elasticsearch & loki output and sidecar for reloading after config change ( not only config, but mainly secret (tls) change )
• logging-operator by banzaicloud -> systemd and host logging is behind paywall via logging-opeator-extensions, which is nogo for me
• kubesphere/fluentbit-operator -> seems unfinished ( no helm chart ), but promising
• vmware/kube-fluentd-operator -> helm chart available, its promising
Any other alternatives? I can probably use beats & logstash, but whole community is using fluentbit/fluentd combo..., but this ecosystem is not matured yet...
Ideas? Thanks

@here office hours is starting in 30 minutes! Remember to post your questions here.

I am interested to know what some peoples experiences are with AWS WAF alternatives. Bonus if on-prem but not req. These are some of the more popular ones Ive found Imperva, Fortinet, Signal Sciences, Barracuda, Sophos, F5 and obvs cloudflare.

Robert Horrox has joined Public "Office Hours"

@here our devops #office-hours starting now! join us to talk shop https://cloudposse.zoom.us/j/508587304

Erik Osterman (Cloud Posse) has joined Public "Office Hours"

vicken has joined Public "Office Hours"

Alex Siegman has joined Public "Office Hours"

Nigel Kirby has joined Public "Office Hours"

Andrew Roth has joined Public "Office Hours"

Vlad Ionescu has joined Public "Office Hours"

Raja Tejas Yerramalli has joined Public "Office Hours"

David Scott has joined Public "Office Hours"

Zadkiel AHARONIAN has joined Public "Office Hours"

Fernando Castillo has joined Public "Office Hours"

sri has joined Public "Office Hours"

vicken has joined Public "Office Hours"

Patrick Joyce has joined Public "Office Hours"

Alex Pereyra has joined Public "Office Hours"

15139103984 has joined Public "Office Hours"

rhenusonerosalia has joined Public "Office Hours"

Zachary Loeber has joined Public "Office Hours"

Fernando Castillo has joined Public "Office Hours"

Tim Gourley has joined Public "Office Hours"

Neil Gealy has joined Public "Office Hours"

Kareem Shahin has joined Public "Office Hours"

Jeremy (Cloud Posse) has joined Public "Office Hours"

pepe amengual has joined Public "Office Hours"

Rohit G has joined Public "Office Hours"

Ω has joined Public "Office Hours"

Marc Tamsky has joined Public "Office Hours"

Michael Londeen has joined Public "Office Hours"

Eric Berg has joined Public "Office Hours"

Related to the current conversation: https://github.com/cloudflare/cf-terraforming

David Lundgren has joined Public "Office Hours"

charles pogi has joined Public "Office Hours"

Nicolás de la Torre has joined Public "Office Hours"

VJ D has joined Public "Office Hours"

pepe amengual has joined Public "Office Hours"

Durgesh Manohar has joined Public "Office Hours"

👉️ New Zoom Recording from our Office Hours session on 2020-09-30 is now available.