51 messages
👽️
Junk11 months ago
Hey all,
I’ve got a couple of questions around using Atmos and Terraform in a modular setup — would love to hear your thoughts or experiences.
1. Sharing EKS auth info across components
I’m deploying an EKS cluster as one component using Atmos, and I need to share its authentication info with other components that have different lifecycles.
In the past, when using a more traditional Terraform workspace setup, I just used a data block to grab the cluster info. But now, with components separated in Atmos, I’m not sure what the best pattern is for securely sharing this kind of data.
Using templates or some shared data store(!data) might work, but I’m a bit concerned about sensitive info being exposed in plain text.
How are people handling this kind of cross-component dependency, especially when it comes to sensitive data?
2. Applying Cloudposse-style conventions to 3rd-party modules
For modules not created by Cloudposse, I still want to apply consistent naming and tagging using the terraform-null-label pattern.
In my case, modifying the root module by injecting(with mixin) context.tf seems risky, as it could break the module’s independence and cause upgrade pain later.
On the other hand, wrapping third-party modules inside another "adapter" module feels like it adds a lot of maintenance overhead.
Has anyone found a clean way to apply conventions to external modules without sacrificing maintainability or upgradeability?
Appreciate any input — happy to hear how others are dealing with similar situations.
I’ve got a couple of questions around using Atmos and Terraform in a modular setup — would love to hear your thoughts or experiences.
1. Sharing EKS auth info across components
I’m deploying an EKS cluster as one component using Atmos, and I need to share its authentication info with other components that have different lifecycles.
In the past, when using a more traditional Terraform workspace setup, I just used a data block to grab the cluster info. But now, with components separated in Atmos, I’m not sure what the best pattern is for securely sharing this kind of data.
Using templates or some shared data store(!data) might work, but I’m a bit concerned about sensitive info being exposed in plain text.
How are people handling this kind of cross-component dependency, especially when it comes to sensitive data?
2. Applying Cloudposse-style conventions to 3rd-party modules
For modules not created by Cloudposse, I still want to apply consistent naming and tagging using the terraform-null-label pattern.
In my case, modifying the root module by injecting(with mixin) context.tf seems risky, as it could break the module’s independence and cause upgrade pain later.
On the other hand, wrapping third-party modules inside another "adapter" module feels like it adds a lot of maintenance overhead.
Has anyone found a clean way to apply conventions to external modules without sacrificing maintainability or upgradeability?
Appreciate any input — happy to hear how others are dealing with similar situations.
Dan Hansen11 months ago
I'm trying to update my atmos runner image that uses
Is this just because the latest version was released too recently, or is it something else?
alpine , but apk cannot find any atmos packages.Is this just because the latest version was released too recently, or is it something else?
Miguel Zablah11 months ago
Hi! I was reading throw the docs example on !terraform.output and what dose this part refers to?
https://atmos.tools/core-concepts/stacks/yaml-functions/terraform.output#specifying-atmos-stack
also I think there is a typo
https://atmos.tools/core-concepts/stacks/yaml-functions/terraform.output#specifying-atmos-stack
also I think there is a typo
!temolate it should probabl be !templatePetr Dondukov11 months ago
Hi, in the latest version 1.169.0 I stopped having this construction in the tmpl file working:
virtual_network:
address_space: !include stacks/networks.yaml ".virtual_networks.{{ .common_name }}"Chris Harden11 months ago(edited)
Hi, is it possible to pull Atmos configuration values into a workflow shell command?
I've tried several variations unsuccessfully
Thanks in advance!
running version 1.169.0
name: "Test Workflow"
workflows:
test-command:
stack: test-stack
description: |
Verify interpolation really happens in shell commands
steps:
- name: start
type: shell
command: |
echo "Start"
echo "{{ .stack }}"
echo "End"
### Output
❯ atmos workflow test-command -f test-workflow
Start
{{ .stack }}
EndI've tried several variations unsuccessfully
echo "{{ .ComponentConfig.vars.stack }}"
echo "!terraform.output vpc vpc_id"
echo "{{ (atmos.Component "vpc" .stack).outputs.vpc_id }}"Thanks in advance!
running version 1.169.0
Kane11 months ago(edited)
Hi, having a confusion on tainting / replacing vm created from a map on Azure. What is the correct command to taint them all please ?
vm map
a :
size : "Standard_D32ls_v6"
private_ip : "1.2.3.4"
<snip>
b :
size : "Standard_D32ls_v6"
private_ip : "1.2.3.5"
<snip>
c :
size : "Standard_D32ls_v6"
private_ip : "1.2.3.6"
<snip>
changed userdata
azurerm_linux_virtual_machine.vm["a"]: Modifications complete after 12s
azurerm_linux_virtual_machine.vm["b"]: Modifications complete after 12s
azurerm_linux_virtual_machine.vm["c"]: Modifications complete after 13s
atmos terraform taint -s azure/prod/uksouth 'netflow-ingest-virtual-machines azurerm_linux_virtual_machine.vm["a"]'
< reads all map values correctly >
│ Error: No such resource instance
│
│ There is no resource instance in the state with the address azurerm_linux_virtual_machine.vm. If the resource configuration has just been added,
│ you must run "terraform apply" once to create the corresponding instance(s) before they can be tainted.
╵
no config found for the component 'netflow-ingest-virtual-machines azurerm_linux_virtual_machine.vm["a"]' in the stack manifest 'azure/prod/uksouth'
Atmos 1.165.3 on darwin/amd64 (zsh shell)
vm map
a :
size : "Standard_D32ls_v6"
private_ip : "1.2.3.4"
<snip>
b :
size : "Standard_D32ls_v6"
private_ip : "1.2.3.5"
<snip>
c :
size : "Standard_D32ls_v6"
private_ip : "1.2.3.6"
<snip>
changed userdata
azurerm_linux_virtual_machine.vm["a"]: Modifications complete after 12s
azurerm_linux_virtual_machine.vm["b"]: Modifications complete after 12s
azurerm_linux_virtual_machine.vm["c"]: Modifications complete after 13s
atmos terraform taint -s azure/prod/uksouth 'netflow-ingest-virtual-machines azurerm_linux_virtual_machine.vm["a"]'
< reads all map values correctly >
│ Error: No such resource instance
│
│ There is no resource instance in the state with the address azurerm_linux_virtual_machine.vm. If the resource configuration has just been added,
│ you must run "terraform apply" once to create the corresponding instance(s) before they can be tainted.
╵
no config found for the component 'netflow-ingest-virtual-machines azurerm_linux_virtual_machine.vm["a"]' in the stack manifest 'azure/prod/uksouth'
Atmos 1.165.3 on darwin/amd64 (zsh shell)
Gheorghe Casian11 months ago(edited)
Hello,
I am having an issue when running
The execution of the command is endless.
Any thoughts on how to debug and fix it?
I am having an issue when running
atmos describe affected --include-dependents=true --verbose=true --format yaml --file components.yaml --repo-path /tmpThe execution of the command is endless.
Any thoughts on how to debug and fix it?
J
Junk11 months ago
Hi, I'm currently implementing drift detection and remediation in GitHub Actions based on this guide:
https://docs.cloudposse.com/layers/gitops/example-workflows/#atmos-terraform-drift-remediation
During this process, I’m encountering an issue where only the remediation workflow fails to find
Strangely, the manually dispatched apply workflow runs just fine.
The
Do you happen to have any idea what might be causing this or how I could resolve it? Any insight would be appreciated.
https://docs.cloudposse.com/layers/gitops/example-workflows/#atmos-terraform-drift-remediation
During this process, I’m encountering an issue where only the remediation workflow fails to find
terraform in the PATH when running apply.Strangely, the manually dispatched apply workflow runs just fine.
The
atmos.yaml file is located in the project root, and the configuration for using tofu (including the version) is correctly specified.Do you happen to have any idea what might be causing this or how I could resolve it? Any insight would be appreciated.
Jonathan Rose11 months ago
I am trying to understand how I can use overrides to implement
requireproviders on my components. Currently, I do something like this, but I get deprecation warnings:overrides:
terraform:
providers:
aws:Weston Platter11 months ago(edited)
Question about setting up atmos ... in the CloudPosse "Jumpstart Customers" getting started section, 3rd item , it mentions that you need to setup multiple accounts.
Is there a guide for using atmos on a single aws account?
Is there a guide for using atmos on a single aws account?
Nick Dunn11 months ago
Hi, everyone. I hope this is an easy question. I upgraded Atmos today and I've noticed new behavior that I don't quite understand. When I run
Is there a new caching setting I'm not aware of?
atmos with no arguments to pull up the TUI, Atmos proceeds to take its time fetching a great many !terraform.outputs . It's certainly not all of them, but it's quite a few and I haven't yet noticed a pattern that indicates why those outputs are the ones it believes it must fetch every time I try to simply run the TUI.Is there a new caching setting I'm not aware of?
Sam Chen11 months ago
Does anyone have more context on why the devcontainer file wants 4 CPU, 8 GB of memory, and 16 GB of storage? That's like 50% of the cpu/memory on my computer.
Sam Chen11 months ago(edited)
is there a way to import sources resources into an atmos stack/component? or migrate from an existing TF statefile?
kofi11 months ago
Hello! Is it possible to include a folder instead of listing all files?
A
ameer11 months ago
hello, I am attempting to use the aws-nlb module (https://github.com/cloudposse/terraform-aws-nlb) in atmos and am setting configurations that rely on other components in the stack with something similar to
!terraform.output vpc vpc_id. i am on atmos 1.164.0. I am getting the error in the screenshot when attempting to apply the nlb component despite running the clean command and refreshing and applying the vpc component. Is this the right place to seek help?Sam Chen11 months ago(edited)
Am I going to run into trouble if my provider module? e.g. if one has more "elements" than the other.
stack.name_template is different than how I use the terraform-null-label Sam Chen11 months ago
Are data modules in plain terraform an antipattern with atmos? If not, would it be a component, static backend, a mixin, or something else?
For instance, I'm looking to pass down the GCP org ID and it's only available as a TF data source. The alternative would be to hardcode this in Atmos, which would be fine since we wouldn't make new ones often.
For instance, I'm looking to pass down the GCP org ID and it's only available as a TF data source. The alternative would be to hardcode this in Atmos, which would be fine since we wouldn't make new ones often.
ameer11 months ago
Is there a
--no-color option for atmos cli?Dan Hansen11 months ago
I don't think I've read anything else about this feature in the docs besides this blurb. Is there any more info? From https://atmos.tools/features
• Terraform Provider Atmos has a Terraform provider that can be used to manage Atmos configurations and stacks natively from HCL.
Sam Chen11 months ago
Can someone clarify, point me to any docs or prior conversations about the difference between component
imports vs inheritance? Is the former for keys only and the latter for values only? Is the former applied to all instances of a component in a manifest and the latter is for the individual instances?Sam Chen11 months ago
Maybe it's related to the above: I'm using a
_defaults.yaml manifest to define a gcs backend, but the generated backend JSON is empty. It works if I copy the same lines to the component manifest.RB10 months ago
For self hosted github runners, what do you suggest for cpu/mem or other configs in ARC to run atmos stacks ?
RB10 months ago
Upstream library of linting rules for atmos stacks? If not, what do you folks think?
Hoping to increase performance of deep merge by preventing usage of
Hoping to increase performance of deep merge by preventing usage of
atmos.Component and terraform.outputSam Chen10 months ago
Can someone highlight some differences between the two patterns in the docs:
component catalog with mixins vs layered stack configuration. They feel like they're solving the same problem but slightly differently.E
erik10 months ago
Sam Chen10 months ago
can someone point me to the special cold start steps for creating a backend bucket? I have my cold start but just want to do a sanity check. My current steps are:
1. set
2. run
3. edit stack to use the new remote backend
4. run
1. set
auto_generate_backend_file to false, to use the local backend2. run
atmos terraform plan <component> --stack <stack>3. edit stack to use the new remote backend
4. run
atmos terraform init -migrate-state Miguel Zablah10 months ago
I have a question about components I have been adding the dependency for each component but I don't see that enforce or required in anyway are they plans for this?
for example if
for example if
RDS requieres the vpc_id from vpc component I can get that using something like !terraform.output I usually add the settings.depends_on but I don't see a resound to do this anymore and I also see less and less examples of places where we use that is this deprecated? settings:
depends_on:
1:
component: "vpc"Sam Chen10 months ago
I know the docs say to avoid templates, and I agree, but is it the only way to dry up the local config (without external dependencies) if I want to use the same var for multiple stacks/components, via imports?
Jonathan Rose10 months ago
What's the recommended solution to resolve this "warning" about TTY
+ atmos vendor pull
INFO Vendoring from '/atmos/vendor.yaml'
WARN No TTY detected. Falling back to basic output. This can happen when no terminal is attached or when commands are pipelined.
INFO ✓ terraform-aws-vpc (v5.19.0)
INFO Vendored 1 components.Chris Harden10 months ago
Hi, can I define stack specific vendor files? e.g., for my test stack I'm vendoring v0.0.2 of an upstream component and my staging stack versions v0.0.1? I understand I can prevent migrating the state file by defining the workspace_key_prefix, but a single vendor file doesn't allow multiple sources with the same component name.
Additionally, I feel it would be handy in a mono repo to only vendor pull the components needed for the workflow that's being applied, rather than all the components.
Thanks in advance!
- component: "tgw-route-table"
source: "git::<https://github.com/my-org/tgw-route-table?ref={{.Version}}>"
version: "v0.0.1"
targets:
- "components/terraform/{{ .Component }}}/{{ .Version }}"
- component: "tgw-route-table"
source: "git::<https://github.com/my-org/tgw-route-table?ref={{.Version}}>"
version: "v0.0.2"
targets:
- "components/terraform/{{ .Component }}}/{{ .Version }}"Additionally, I feel it would be handy in a mono repo to only vendor pull the components needed for the workflow that's being applied, rather than all the components.
Thanks in advance!
Cristian10 months ago
Hi all,
I am getting started with
Do i need to manually create that component or should this be part of the generated tarball? Atmos doesnt show
I am getting started with
atmos using geodesic. I am currently trying to deploy the aws-sso-sync component to sync groups, as described on https://registry.terraform.io/modules/cloudposse/components/aws/latest/submodules/aws-ssosync.Do i need to manually create that component or should this be part of the generated tarball? Atmos doesnt show
aws-sso-sync only aws-sso is present.J
Junk10 months ago
Hi Atmos! I encountered the following warning while running a PLAN using
It seems like there's a failure related to Binary Cache Restore. Could this issue be affecting the performance (processing time) of the workflow?
Also, do you have any suggestions on how to resolve this problem?
github-action-atmos-terraform-plan.It seems like there's a failure related to Binary Cache Restore. Could this issue be affecting the performance (processing time) of the workflow?
Also, do you have any suggestions on how to resolve this problem?
Kane10 months ago(edited)
any chance of some help on why this fails please, the version numbers are the latest releases. I understand it isnt finding anything to download but looking at the repo's they are identical in layout etc.
apiVersion: atmos/v1
kind: AtmosVendorConfig
metadata:
name: vendor-config
description: Atmos vendoring manifest
spec:
sources:
- component: "aws-vpc"
source: "github.com/cloudposse/terraform-aws-components.git//modules/aws-vpc?ref={{.Version}}"
version: "1.537.1"
targets:
- "components/terraform/{{.Component}}/{{.Version}}"
- component: "aws-eks-cluster"
source: "github.com/cloudposse/terraform-aws-components.git//modules/aws-eks-cluster?ref={{.Version}}"
version: "1.536.0"
targets:
- "components/terraform/{{.Component}}/{{.Version}}"
- component: "aws-ssosync"
source: "github.com/cloudposse/terraform-aws-components.git//modules/aws-ssosync?ref={{.Version}}"
version: "1.535.0"
targets:
- "components/terraform/{{.Component}}/{{.Version}}"╰─➤ atmos vendor pull
INFO Vendoring from 'vendor.yaml'
x aws-vpc (1.537.1) Failed to vendor aws-vpc: error : failed to download package: error downloading '<https://github.com/cloudposse/terraform-aws-components.git?ref=1.537.1>': /usr/local/bin/git exited with 1: error: pathspec '1.537.1' did not match any file(s) known to git
x aws-eks-cluster (1.536.0) Failed to vendor aws-eks-cluster: error : failed to download package: error downloading '<https://github.com/cloudposse/terraform-aws-components.git?ref=1.536.0>': /usr/local/bin/git exited with 1: error: pathspec '1.536.0' did not match any file(s) known to git
✓ aws-ssosync (1.535.0)
Vendored 1 components. Failed to vendor 2 components. RB10 months ago(edited)
I want to use the versioned method for components and I’d like to manage that version from a single place. How do I do this?
The
Edit: for now, best solution is set it explicitly in
The
s3/test-bucket won’t inherit the component keycomponents:
terraform:
s3/defaults:
metadata:
type: abstract
component: s3-bucket/1.535.1
s3/test-bucket:
metadata:
inherits:
- s3/defaults
vars:
bucket_name: org-test-bucketEdit: for now, best solution is set it explicitly in
metadata.component per component instance OR use a template strategy s3-bucket/{{.version}}. It’s on the roadmap to inherit all keys (except inherits) of metadata so that may be the better solution.RB10 months ago
Related to the above thread, if I had a “local” component vendored as a “mixin”, how would I vendor it into every component? Can I use some kind of glob or would I need to specify every component under
targets ?cricketsc10 months ago
it seems I upgraded to 1.171.0 and am now having issues with using
(atmos.Component ...) in templates. Anyone else experiencing this?Jonathan Rose10 months ago(edited)
Creating the following validation to ensure
I currently have a test key that doesn't provide an alias, which should cause the above to fail. The component uses terraform-aws-kms/wrappers at v3.1.1 · terraform-aws-modules/terraform-aws-kms. Any suggestions? The goal is to ensure each key specifies at least one alias, which means an empty list should not be permitted.
aliases is provided for my new KMS component:{
"$id": "kms-key-component",
"$schema": "<https://json-schema.org/draft/2020-12/schema>",
"title": "vpc component validation",
"description": "JSON Schema for the 'kms' Atmos component.",
"type": "object",
"properties": {
"vars": {
"type": "object",
"properties": {
"aliases": {
"type": "array",
"prefixItems": [
{
"type": "string"
}
],
"items": false
}
}
}
}
}I currently have a test key that doesn't provide an alias, which should cause the above to fail. The component uses terraform-aws-kms/wrappers at v3.1.1 · terraform-aws-modules/terraform-aws-kms. Any suggestions? The goal is to ensure each key specifies at least one alias, which means an empty list should not be permitted.
Cyrus Dukart10 months ago
Hi folks we are trying to essentially get a "Hello World" setup here for atmos on a new AWS account.
We are currently trying to Deploy Roles. I assume i need to authenticate in through Leapp as superadmin .. but i'm not 100% the correct method to do this and get it synced up into geodesic. Can anyone point me to the correct next steps here kindly 🙂
We are currently trying to Deploy Roles. I assume i need to authenticate in through Leapp as superadmin .. but i'm not 100% the correct method to do this and get it synced up into geodesic. Can anyone point me to the correct next steps here kindly 🙂
Sam Chen10 months ago
Is there a quick guide to tearing down an atmos bootstrap, including changing from a remote backend to a local backend and destroying the remote backend? I figured it out but was wondering if that’s documented anywhere
Jonathan Rose10 months ago(edited)
When running atmos/terraform in CI, what's the recommended approach to ensuring TF output color is disabled? Set
TF_CLI_ARGS=-no-color environment variable?RB10 months ago
I'm looking for okta group, okta rules, and okta app terraform components. Any chance there are okta components that haven't been upstreamed yet ? If not, are these components that could be maintained if upstreamed by a community member ?
PePe Amengual10 months ago
How can this be used in atmos? How will the account map should look like? https://github.com/cloudposse-terraform-components/aws-account-map/pull/14
kofi10 months ago
Hello!
It would be nice to be able to deploy multiple components in a stack at the same time.
I was not able to make it working natively, but using a combination of
Is there already a way of doing it more natively?
If not, a regex command line (like the Ansible one) would be nice. Example:
Is there someone working on something similar? If there is some interest, I would like to try to put some effort on it.
It would be nice to be able to deploy multiple components in a stack at the same time.
I was not able to make it working natively, but using a combination of
xargs + TF_DATA_DIR (for terraform commands) was giving some first results. I inspired myself from (https://github.com/cloudposse/atmos/issues/225).Is there already a way of doing it more natively?
If not, a regex command line (like the Ansible one) would be nice. Example:
atmos terraform apply "~eks[2-6]" -s stack would deploy in parallel eks2, eks3, eks4, eks5, eks6 components in the stack stack. An alternative would be to use the comma notation (like again in Ansible).Is there someone working on something similar? If there is some interest, I would like to try to put some effort on it.
E
PePe Amengual10 months ago
I will use my powers....
Igor M10 months ago
When using
I get
Is this a known issue or do I have something misconfigured on my end?
atmos list valuesI get
ERRO error describing stacks: template: describe-stacks-all-sections:xx:xx: executing "describe-stacks-all-sections" at <"">: invalid value; expected stringIs this a known issue or do I have something misconfigured on my end?
Miguel Zablah10 months ago(edited)
hey did something change with atmos tf shell? when I do this I get an error that not valid credentials are being pass to it.
but when run without shell it works
but when run without shell it works
RB10 months ago
I noticed that workflow key doesnt populate when generating the atmos Atlantis yaml file. Is there a different argument i need to give it to generate this?
Igor M10 months ago
Does !include allow for a merge of list values? Is there any other way to store subsets of list configuration without changing
Use case: large dns-primary record_config, which would be nice to store across multiple files.
list_merge_strategy globally?Use case: large dns-primary record_config, which would be nice to store across multiple files.
Dan Hansen10 months ago
Is it expected behavior for
!terraform.output to return null without failing if the output being fetched does not exist?PePe Amengual10 months ago
Is it possible with atmos to grab all the outputs of a component?