#office-hours - May 2023

For discussion: GenAI bundled in with your observability tools....

I say yes...with caveats. 😅

https://newrelic.com/blog/nerdlog/new-relic-grok

Erik Osterman (Cloud Posse) has joined Public "Office Hours"

@here office hours is starting in 30 minutes! Remember to post your questions here.

Erik Osterman (Cloud Posse) has joined Public "Office Hours"

Jim Park has joined Public "Office Hours"

Manoj Bhagwat has joined Public "Office Hours"

Nenna Salinas has joined Public "Office Hours"

Michael Jenkins has joined Public "Office Hours"

Michael Pursifull has joined Public "Office Hours"

Elisha Bello has joined Public "Office Hours"

Diego Maia has joined Public "Office Hours"

Kris Musard has joined Public "Office Hours"

Ishank Bansal has joined Public "Office Hours"

Vlad Ionescu has joined Public "Office Hours"

Charles Smith has joined Public "Office Hours"

Ian Bartholomew has joined Public "Office Hours"

Isaac M has joined Public "Office Hours"

Jonathan Eunice has joined Public "Office Hours"

Josh has joined Public "Office Hours"

Wasim Khan has joined Public "Office Hours"

Afolabi Omotoso has joined Public "Office Hours"

Ozzy Al has joined Public "Office Hours"

David Hawthorne has joined Public "Office Hours"

dave lundgren has joined Public "Office Hours"

Abdul Aziz Tetteh has joined Public "Office Hours"

Wasim Khan has joined Public "Office Hours"

Tim Gourley has joined Public "Office Hours"

https://aws.amazon.com/blogs/aws/new-set-up-your-aws-notifications-in-one-place/ and https://aws.amazon.com/about-aws/whats-new/2023/04/amazon-rds-m7g-r7g-database-instances/ 🙂

Jeremy / Nuru (Cloud Posse) has joined Public "Office Hours"

Matt Calhoun has joined Public "Office Hours"

Jose Figueredo has joined Public "Office Hours"

Michael Vasilenko has joined Public "Office Hours"

John Mitchell has joined Public "Office Hours"

Michael Vasilenko has joined Public "Office Hours"

Matt Gowie has joined Public "Office Hours"

Sandro Aldave has joined Public "Office Hours"

Adelia R has joined Public "Office Hours"

Adebiyi Adegboye has joined Public "Office Hours"

Adelia R has joined Public "Office Hours"

Adebiyi Adegboye has joined Public "Office Hours"

Adelia R has joined Public "Office Hours"

Adebiyi Adegboye has joined Public "Office Hours"

Wasim Khan has joined Public "Office Hours"

John Mitchell has joined Public "Office Hours"

Links from today's office hours:

https://github.com/Madh93/tpm
https://github.com/paololazzari/terraform-repl
https://github.com/cloudposse/bastion
https://github.com/hashicorp/terraform-provider-aws/milestone/226
https://aws.amazon.com/about-aws/whats-new/2022/12/aws-compute-optimizer-amazon-ecs-services-aws-fargate/
https://aws.amazon.com/about-aws/whats-new/2023/04/amazon-s3-security-best-practices-buckets-default/
https://aws.amazon.com/about-aws/whats-new/2023/04/aws-verified-access-generally-available/
https://www.hashicorp.com/blog/kubernetes-vault-integration-via-sidecar-agent-injector-vs-csi-provider
https://aws.amazon.com/about-aws/whats-new/2023/04/amazon-rds-m7g-r7g-database-instances/
https://aws.amazon.com/blogs/aws/new-set-up-your-aws-notifications-in-one-place/
https://github.com/hashicorp/terraform-provider-aws/issues/29689
https://github.com/hashicorp/terraform-provider-aws/issues/30380
https://github.com/cloudposse/geodesic
https://github.com/cloudposse/packages/tree/master/vendor
https://aws.amazon.com/verified-access/pricing/
https://docs.aws.amazon.com/chatbot/latest/adminguide/what-is.html
https://marbot.io
https://cloudonaut.io/
https://cloudonaut.io/ec2-checklist-seven-things-to-do-after-launching-an-instance/
https://aws.amazon.com/marketplace/pp/prodview-sykoblbsdgw2o
https://bucketav.com/features/
https://docs.konghq.com/hub/
https://github.com/1debit/alternat
https://www.krakend.io/
https://github.com/int128/terraform-aws-nat-instance

@here office hours is starting in 30 minutes! Remember to post your questions here.

Note new releases to fix S3 bucket creation problems:

• terraform-aws-s3-bucket v3.1.1 released
• terraform-aws-s3-log-storage v1.3.1 released
• terraform-aws-vpc-flow-logs-s3-bucket v1.0.1 released
• terraform-aws-lb-s3-bucket v0.16.4 released

Erik Osterman (Cloud Posse) has joined Public "Office Hours"

Andy Roth has joined Public "Office Hours"

Dave Gregory has joined Public "Office Hours"

Dan Medeiros has joined Public "Office Hours"

Isa has joined Public "Office Hours"

Kris Musard has joined Public "Office Hours"

dag viggo lokoeen has joined Public "Office Hours"

Nenna Salinas has joined Public "Office Hours"

Jonathan Eunice has joined Public "Office Hours"

dag viggo lokoeen has joined Public "Office Hours"

Ishank Bansal has joined Public "Office Hours"

Vlad Ionescu has joined Public "Office Hours"

Michael Jenkins has joined Public "Office Hours"

Matt Calhoun has joined Public "Office Hours"

David Hawthorne has joined Public "Office Hours"

Henri Gabriel has joined Public "Office Hours"

Pamela Hita has joined Public "Office Hours"

Andrew Vitko has joined Public "Office Hours"

Jeremy White has joined Public "Office Hours"

Alex Atkinson has joined Public "Office Hours"

Mike Martin has joined Public "Office Hours"

Paul Marcelin has joined Public "Office Hours"

Isaac M has joined Public "Office Hours"

Rahul has joined Public "Office Hours"

Michael Pursifull has joined Public "Office Hours"

Tim Gourley has joined Public "Office Hours"

Eric Berg has joined Public "Office Hours"

Chris Dobbyn has joined Public "Office Hours"

Alex Siegman has joined Public "Office Hours"

Matt Gowie has joined Public "Office Hours"

Darren Pham has joined Public "Office Hours"

Johnmary Odenigbo has joined Public "Office Hours"

https://aws.amazon.com/about-aws/whats-new/2023/05/aws-appsync-graphql-apis-private-api-support/ and https://github.blog/changelog/2023-05-10-github-actions-actions-runner-controller-public-beta/ 🙂

Hao Wang has joined Public "Office Hours"

I'm having problems with setting incomingCidrs , using aws-load-balancer-controller Helm chart...depoyed as a helm_release resource. I'm getting the list of IPs from the Cloudflare provider, which returns lists of CIDR blocks, but I can't seem to get the escaping or whatever right. I keep getting

│ Error: failed parsing key "inboundCidrs" with value "103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,104.16.0.0/13,104.24.0.0/14,108.162.192.0/18,131.0.72.0/22,141.101.64.0/18,162.158.0.0/15,172.64.0.0/13,173.245.48.0/20,188.114.96.0/20,190.93.240.0/20,197.234.240.0/22,198.41.128.0/17,2400:cb00::/32,2405:8100::/32,2405:b500::/32,2606:4700::/32,2803:f800::/32,2a06:98c0::/29,2c0f:f248::/32", key "0/22" has no value (cannot end with ,)

I tried "\"${join(",", data.cloudflare_ip_ranges.cloudflare.ipv4_cidr_blocks)}\"" , but same problem.

This appears to me to be related to helm's wrapping long lines, which results in these YAML key issues.

Links from today's office hours:
https://servian.dev/terraform-local-providers-and-registry-mirror-configuration-b963117dfffa
https://aws.amazon.com/about-aws/whats-new/2023/05/aws-backup-cross-region-backups-four-regions/
https://techcrunch.com/2023/05/10/aws-open-sources-snapchange-and-cedar-sdk/amp/
https://github.com/cedar-policy/
https://www.bleepingcomputer.com/news/security/github-now-auto-blocks-token-and-api-key-leaks-for-all-repos/amp/
https://www.githubstatus.com/history
https://help.evernote.com/hc/en-us/articles/12748274247059-Collaborative-Editing-Overview
https://obsidian.md/
https://sweetops.slack.com/archives/CHDR1EWNA/p1683742684063129
https://github.com/develeap/terraform-provider-chatgpt
https://janik6n.net/posts/manage-multiple-terraform-projects-in-monorepo/
https://atmos.tools/
https://aws.amazon.com/about-aws/whats-new/2023/05/aws-appsync-graphql-apis-private-api-support/
https://github.blog/changelog/2023-05-10-github-actions-actions-runner-controller-public-beta/
https://www.osohq.com
https://logseq.com/
https://wiki.dendron.so/
https://goblin.tools
https://reinventedsoftware.com/keepit/
https://twitter.com/slackhq/status/521894442064560128
https://ourtechroom.com/tech/slack-technology-stack/
https://reg.rainfocus.com/flow/github/universe23/cfp/page/cfslandingpage
https://github.com/kubernetes-sigs/aws-load-balancer-controller

As a counterpoint to the seemingly prevailing view (e.g. from the previous week's OH) away from KRM (Crossplane/Anthos Config Connector), this is interesting to see Spotify blog about their terraform migration.

Their two reasons for ruling out terraform were they wanted it to be fully declarative (config as data, which I guess Cloud Posse does to an extent) and additionally break-glass functionality (able to do a quick e.g. kubectl edit cloudsql without running a pipeline). They seem to have it working across 3,000 GCP projects, so hopefully they give a talk about the rough edges they encountered.
https://twitter.com/bgrant0607/status/1654870283394891776
https://engineering.atspotify.com/2023/05/fleet-management-at-spotify-part-2-the-path-to-declarative-infrastructure/

And fwiw, Alibaba's Alipay have brewed something similar internally.
https://github.com/KusionStack

Just heard about this: https://github.com/flux-subsystem-argo/flamingo not sure when it came out but is anyone using it? It's looks like it's the best of both (ArgoCD + FluxCD)

Cross-posting this from #aws for discussion in this weeks #office-hours (which I may not be able to make, but I will watch the recording if ya'll discuss this!): https://sweetops.slack.com/archives/CCT1E7JJY/p1683939758963199

I won't make it to the call today, sorry 😞

@here office hours is starting in 30 minutes! Remember to post your questions here.

FYI - https://www.hashicorp.com/blog/terraform-cloud-updates-plans-with-an-enhanced-free-tier-and-more-flexibility

Erik Osterman (Cloud Posse) has joined Public "Office Hours"

Rinchin Shoysoronov has joined Public "Office Hours"

Nenna Salinas has joined Public "Office Hours"

venkata mutyala has joined Public "Office Hours"

Emile Fugulin has joined Public "Office Hours"

Hao Wang has joined Public "Office Hours"

David Hawthorne has joined Public "Office Hours"

Joe Caulfield has joined Public "Office Hours"

Jonathan Poczatek has joined Public "Office Hours"

Dave Gregory has joined Public "Office Hours"

Michael Jenkins has joined Public "Office Hours"

Menekse Tok has joined Public "Office Hours"

Jim Park has joined Public "Office Hours"

dag viggo lokoeen has joined Public "Office Hours"

solomon onwuasoanya has joined Public "Office Hours"

Jonathan Eunice has joined Public "Office Hours"

Stevan Arychuk has joined Public "Office Hours"

Paul Marcelin has joined Public "Office Hours"

Isa has joined Public "Office Hours"

Henri Gabriel has joined Public "Office Hours"

Isaac M has joined Public "Office Hours"

Sean Roberts has joined Public "Office Hours"

Matt Calhoun has joined Public "Office Hours"

Oliver Schoenborn has joined Public "Office Hours"

Ian Bartholomew has joined Public "Office Hours"

Brad Curfman has joined Public "Office Hours"

Ozzy Al has joined Public "Office Hours"

Ishank Bansal has joined Public "Office Hours"

Isa has joined Public "Office Hours"

Dariusz Panasiuk has joined Public "Office Hours"

Eduardo Wohlers has joined Public "Office Hours"

Jeremy White has joined Public "Office Hours"

Alex Atkinson has joined Public "Office Hours"

Kris Musard has joined Public "Office Hours"

Amer Zec has joined Public "Office Hours"

Michael Pursifull has joined Public "Office Hours"

Marc Tamsky has joined Public "Office Hours"

dag viggo lokoeen has joined Public "Office Hours"

Oliver Schoenborn has joined Public "Office Hours"

Dave Gregory has joined Public "Office Hours"

Junior Jimenez has joined Public "Office Hours"

Links from today's office hours:

https://blog.cloudflare.com/r2-super-slurper-ga/
https://www.hashicorp.com/blog/terraform-cloud-updates-plans-with-an-enhanced-free-tier-and-more-flexibility
https://news.trendmicro.com/2023/05/13/openai-chatgpt-data-breach/
https://techcrunch.com/2023/05/12/aws-announces-new-version-of-aurora-database-that-strips-out-i-o-costs/amp/
https://github.com/liggitt/audit2rbac
https://newsletter.pragmaticengineer.com/p/datadogs-65myear-customer-mystery
https://github.com/hashicorp/tfc-workflows-github
https://blog.aquasec.com/leveraging-kubernetes-rbac-to-backdoor-clusters
https://navendu.me/posts/gateway-and-mesh/
https://itnext.io/k8sgpt-localai-unlock-kubernetes-superpowers-for-free-584790de9b65
https://github.com/flux-subsystem-argo/flamingo
https://github.com/AlexNabokikh/tfsort
https://www.theregister.com/2023/05/16/alpine_linux_318/
https://martinheinz.dev/blog/92
https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
https://github.com/env0/custom-flows-examples/tree/main/dynamic-backend

If you want to auto-detect the specific GH Actions deprecations mentioned today across all your repos, you can do this:
https://gist.github.com/AlexAtkinson/b08037e721671ad15fa356d54f6d22e8
Just a quick job... not a pretty pretty pony of a script. Maybe one day.

Looks like I might miss today too 😞 I am so sorry!

@here office hours is starting in 30 minutes! Remember to post your questions here.

Not specifically Terraform, but interested if anyone has recommended platforms for getting security certifications and answering security questionnaires. Ones we know about: OneTrust (formerly known as Tugboat Logic), Vanta, Drata, Secureframe. There seem to be about 73 different options out there, and don’t want to boil the entire ocean, but if anyone has such a platform they love, would appreciate hearing about it.

What’s up with the Datadog metric kubernetes_state.node.age? It doesn’t seem to report in any logical units, like seconds or minutes, and doesn’t seem well documented, according to The Google. Anyone know how to interpret it?

Finally, we recently had a situation with Karpenter stopping autoscaling (up or down). A bunch of host-xyz pods were present (to our eyes, a surprising number of those pods). Soon as those were deleted, everything returned to normal / good auto-scaling. Ideas?

Erik Osterman (Cloud Posse) has joined Public "Office Hours"

Alex Atkinson has joined Public "Office Hours"

Michael Jenkins has joined Public "Office Hours"

Dave Gregory has joined Public "Office Hours"

Nenna Salinas has joined Public "Office Hours"

Henri Gabriel has joined Public "Office Hours"

Sean Roberts has joined Public "Office Hours"

Matt Calhoun has joined Public "Office Hours"

Vlad Ionescu has joined Public "Office Hours"

Jonathan Eunice has joined Public "Office Hours"

Vicken Simonian has joined Public "Office Hours"

Kris Musard has joined Public "Office Hours"

Adebiyi Adegboye has joined Public "Office Hours"

Paul Bullock has joined Public "Office Hours"

https://aws.amazon.com/about-aws/whats-new/2023/05/aws-global-accelerator-extends-tcp-termination-ipv6-traffic/ and https://aws.amazon.com/about-aws/whats-new/2023/05/improved-end-to-end-latencies-amazon-eventbridge-event-buses/

Jonathan Eunice has joined Public "Office Hours"

Marc Tamsky has joined Public "Office Hours"

Dariusz Panasiuk has joined Public "Office Hours"

CHristopher Pieper has joined Public "Office Hours"

Paul Marcelin has joined Public "Office Hours"

Hao Wang has joined Public "Office Hours"

Denis Baryshev has joined Public "Office Hours"

Office Hours Q:
1. We use terraform output to inform other systems (such as helm values). Is anyone else caching the output somewhere (file in git, S3, …)?
a. Side note: terraform init should support just fetching the state file.
i. Instead it requires you to download all providers and modules. For our 1000+ tf roots, that takes a LONG TIME.
ii. So I wrote a one-liner that does a simple aws s3 sync then terraform output of all those tfstates (that doesn’t require an init). Fetched that 1000+ tfstates and printed all outputs in minutes.

Links from today's office hours:

https://openai.com/blog/introducing-the-chatgpt-app-for-ios
https://medium.com/@bobbyrsec/the-dangers-of-googles-zip-tld-5e1e675e59a5
https://nyxt.atlas.engineer/
https://zed.dev/
https://www.pcworld.com/article/1919392/spacetop-is-the-first-laptop-without-a-screen.html
https://www.docker.com/blog/welcome-tilt-fixing-the-pains-of-microservice-development-for-kubernetes/
https://blog.visionarycto.com/p/my-20-year-career-is-technical-debt
https://github.com/datarootsio/tf-profile/
https://github.com/paololazzari/fuzzy-terraform-rm
https://gist.github.com/AlexAtkinson/b08037e721671ad15fa356d54f6d22e8
https://aws.amazon.com/about-aws/whats-new/2023/05/aws-global-accelerator-extends-tcp-termination-ipv6-traffic/
https://aws.amazon.com/about-aws/whats-new/2023/05/improved-end-to-end-latencies-amazon-eventbridge-event-buses/
https://en.wikipedia.org/wiki/Year_2038_problem
https://twitter.com/__steele/status/1659822002470014976
https://github.com/gabrie30/ghorg
https://twitter.com/nickste/status/1626642619395883008
https://bitesizedserverless.com/bite/serverless-messaging-latency-compared/
https://medium.com/postnl-engineering/improved-eventbridge-latency-opens-up-new-use-cases-at-postnl-910fdf6b5dde
https://github.com/DataDog/documentation/blob/master/content/en/integrations/kubernetes_state_core.md
https://github.com/aws/karpenter/issues/2021#issuecomment-1485431932
https://developer.hashicorp.com/terraform/language/resources/provisioners/local-exec
https://helm.sh/docs/topics/charts_hooks/
https://github.com/argoproj/argo-cd/issues/12060

Sorry for the harshness on consul at the end there. Every app decision I make these days depends on how well maintained it is regarding CVEs.

Thanks to the SweetOps office hours discussion, kubernetes_state.node.age confusion resolved:
1. The metric indeed measured in seconds. Hat tip: @Matt Calhoun.
2. Fargate nodes are indeed nodes. Hat tip: @Vlad Ionescu (he/him) True even if Fargate notes more like the cluster control plane than all other nodes, at least in our configuration. Fargate nodes naturally have much much longer run times , and therefore entirely skew statistics like average and max values.
3. Solution: filter out Fargate nodes (e.g. kubernetes_state.node.age{stage:prod,!node:fargate-*} ) and add formula to /60 or /3600 to move to minutes or hours. Age values now make sense.
4. Final point: De-provisioned nodes are not nodes, even if they recently were nodes. Possibly obvious...but Datadog’s instantaneous, point-in-time values may not comport with your intuitive feel about cluster/node behavior aggregated over a larger duration. Did not in our case. But with above filters, the numbers now make sense.

Also, average/arithmetic mean is a terrible statistical aggregator, and can get in the way of easily understanding disparate value sizes. Geometric mean better, but AFAIK that not available in Datadog.

AWS's Platform Engineering webinar is going on now. They recommend uploading outputs, such as subnet id's, etc., to Parameter Store for consumption by other modules.
So that's AWS's official recommendation for abstracting references away from the state files.

And in the same segment, they advise using Hashicorp Vault. It's on their diagram. Secondarily, kinda as an afterthought they mention AWS Secrets Manager.

@here office hours is starting in 30 minutes! Remember to post your questions here.

Any thoughts or experience with BastionZero? Another of the “we secure your dev and ops connections” contenders (cf Teleport, StrongDM, …)

Erik Osterman (Cloud Posse) has joined Public "Office Hours"

Nenna Salinas has joined Public "Office Hours"

Adebiyi Adegboye has joined Public "Office Hours"

Michael Pursifull has joined Public "Office Hours"

Andy Wortman has joined Public "Office Hours"

Isaac M has joined Public "Office Hours"

Vlad Ionescu has joined Public "Office Hours"

Nick Janjghava has joined Public "Office Hours"

David Hawthorne has joined Public "Office Hours"

Sean Roberts has joined Public "Office Hours"

Soren Jensen has joined Public "Office Hours"

Paul Bullock has joined Public "Office Hours"

Marc Tamsky has joined Public "Office Hours"

Jonathan Eunice has joined Public "Office Hours"

Michael Jenkins has joined Public "Office Hours"

Eric Berg has joined Public "Office Hours"

Allan Mohr has joined Public "Office Hours"

Matt Gowie has joined Public "Office Hours"

Matt Calhoun has joined Public "Office Hours"

Charles Smith has joined Public "Office Hours"

Aaron Cutchin has joined Public "Office Hours"

Jim C has joined Public "Office Hours"

Amer Zec has joined Public "Office Hours"

Oliver Schoenborn has joined Public "Office Hours"

https://aws.amazon.com/about-aws/whats-new/2023/05/amazon-eks-eks-distro-kubernetes-version-1-27/ and https://aws.amazon.com/about-aws/whats-new/2023/05/aws-appsync-merged-apis-graphql-federation/ 🙂

Vicken Simonian has joined Public "Office Hours"

Rinchin Shoysoronov has joined Public "Office Hours"

Ozzy Al has joined Public "Office Hours"

Sergei Valevka has joined Public "Office Hours"

Jeremy White has joined Public "Office Hours"

Dan Hamilton has joined Public "Office Hours"

Links from today's office hours:

https://www.theregister.com/2023/05/26/microsoft_azure_linux_container/
https://www.digitaltrends.com/computing/keepass-master-password-plain-text-vulnerability/
https://thehackernews.com/2023/05/severe-flaw-in-google-clouds-cloud-sql.html
https://github.com/charmbracelet/mods
https://www.reddit.com/r/Terraform/comments/13ru7f9/heads_up_terraform_aws_provider_500/
https://registry.terraform.io/providers/hashicorp/aws/latest/docs/guides/version-5-upgrade
https://aws.amazon.com/about-aws/whats-new/2023/05/invoice-summary-now-available/
https://blog.brainboard.co/16-best-tools-to-design-your-cicd-engine-%EF%B8%8F-1b3533748a82
https://mkbaio.substack.com/p/please-stop-sending-me-emails-written
https://supabase.com/blog/chatgpt-plugins-support-postgres
https://www.hashicorp.com/blog/terraform-aws-provider-5-0-adds-updates-to-default-tags
https://aws.amazon.com/about-aws/whats-new/2023/05/amazon-eks-eks-distro-kubernetes-version-1-27/
https://aws.amazon.com/about-aws/whats-new/2023/05/aws-appsync-merged-apis-graphql-federation/
https://docs.aws.amazon.com/appsync/latest/devguide/security-authz.html
https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html#kubernetes-1.27
https://docs.aws.amazon.com/appsync/latest/devguide/WAF-Integration.html
https://aws.amazon.com/blogs/mobile/appsync-waf/
https://github.com/bottlerocket-os/bottlerocket/issues/1667
https://www.bastionzero.com/
https://github.com/kvaps/kubectl-node-shell
https://cloud.google.com/security/compliance/fips-140-2-validated/
https://learn.microsoft.com/en-us/azure/aks/enable-fips-nodes
https://signoz.io/
https://github.com/redpanda-data/redpanda
https://newsletter.pragmaticengineer.com/p/the-scoop-47?utm_source=post-email-title&publication_id=458709&post_id=120772763&isFreemail=false&utm_medium=email