v1.72.0

Update `gomplate` datasources. Add `env` and `evaluations ` sections to `Go` template configurations @aknysh (#599)

what

• Update gomplate datasources
• Add env section to Go template configurations
• Add evaluations config to Go template configurations
• Update/improve templating docs
  • https://atmos.tools/core-concepts/stacks/templating/

why

• Allow setting environment variables in Go templates to access cloud resources from datasources
• Allow configuring the number of template evaluations for template processing pipelines

description

Atmos supports configuring the number of evaluations/passes for template processing in atmos.yaml CLI config file. It effectively allows you to define template processing pipelines.

For example:

templates:
  settings:
    # Enable `Go` templates in Atmos stack manifests
    enabled: true
    # Number of evaluations/passes to process `Go` templates
    # If not defined, `evaluations` is automatically set to `1`
    evaluations: 2

• templates.settings.evaluations - number of evaluations/passes to process Go templates. If not defined, num_steps is automatically set to 1

Template evaluations are useful in the following scenarios:

• Combining templates from different sections in Atmos stack manifests
• Using templates in the URLs of datasources

Combining templates from different sections in Atmos stack manifests

You can define more than one step/pass of template processing to use and combine the results from each step.

For example:

templates:
  settings:
    enabled: true
    # Number of evaluations/passes to process `Go` templates
    evaluations: 3

settings:
  test: "{{ .atmos_component }}"
  test2: "{{ .settings.test }}"

components:
  terraform:
    vpc:
      vars:
        tags:
          tag1: "{{ .settings.test }}-{{ .settings.test2 }}"
          tag2: "{{\"{{`{{ .atmos_component }}`}}\"}}"

When executing an Atmos command like atmos terraform plan vpc -s <stack>, the above template will be processed in three phases:

• Evaluation 1

  • settings.test is set to vpc
  • settings.test2 is set to {{ .atmos_component }}
  • vpc.vars.tags.tag1 is set to {{ .atmos_component }}-{{ .settings.test }}
  • vpc.vars.tags.tag2 is set to {{<backtick>{{ .atmos_component }}<backtick>}}

• Evaluation 2

  • settings.test is vpc
  • settings.test2 is set to vpc
  • vpc.vars.tags.tag1 is set to vpc-vpc
  • vpc.vars.tags.tag2 is set to {{ .atmos_component }}

• Evaluation 3

  • settings.test is vpc
  • settings.test2 is vpc
  • vpc.vars.tags.tag1 is vpc-vpc
  • vpc.vars.tags.tag2 is set to vpc

WARNING:
The above example shows the supported functionality in Atmos templating. You can use it for some use-cases, but it does not mean that you should use it just for the sake of using, since it's not easy to read and understand what data we have after each evaluation step.

The following use-case describes a practical approach to using steps and pipelines in Atmos templates to work
with datasources.

Using templates in the URLs of datasources

Let's suppose that your company uses a centralized software catalog to consolidate all tags for tagging all the cloud resources. The tags can include tags per account, per team, per service, billing tags, etc.

NOTE: An example of such a centralized software catalog could be https://backstage.io

Let's also suppose that you have a service to read the tags from the centralized catalog and write them into an S3 bucket in one of your accounts. The bucket serves as a cache to not hit the external system's API with too many requests and not to trigger rate limiting.

And finally, let's say that in the bucket, you have folders per account (dev, prod, staging). Each folder has a JSON file with all the tags defined for all the cloud resources in the accounts.

We can then use the Gomplate S3 datasource to read the JSON file with the tags for each account and assign the tags to all cloud resources.

In atmos.yaml, we can figure 2 evaluation steps of template processing:

templates:
  settings:
    enabled: true
    gomplate:
      enabled: true
    # Number of evaluations to process `Go` templates
    evaluations: 2

In an Atmos stack manifest, we define the environment variables in the env section (AWS profile with permissions to access the S3 bucket), and the s3-tags Gomplate datasource.

In the terraform.vars.tags section, we define all the tags that are returned from the call to the S3 datasource.

import:
  # Import the default configuration for all VPCs in the infrastructure
  - catalog/vpc/defaults

# Global settings
settings:
  templates:
    settings:
      # Environment variables passed to datasources when evaluating templates
      # https://docs.gomplate.ca/functions/aws/#configuring-aws
      # https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html
      env:
        # AWS profile with permissions to access the S3 bucket
        AWS_PROFILE: "<AWS profile>"
      gomplate:
        # Timeout in seconds to execute the datasources
        timeout: 5
        # https://docs.gomplate.ca/datasources
        datasources:
          # `s3` datasource
          # https://docs.gomplate.ca/datasources/#using-s3-datasources
          s3-tags:
            # The `url` uses a `Go` template with the delimiters `${ }`,
            # which is processed as first step in the template processing pipeline
            url: "s3://mybucket/${ .vars.stage }/tags.json"

# Global Terraform config
terraform:
  # Global variables that are used by all Atmos components
  vars:
    tags:
      atmos_component: "{{ .atmos_component }}"
      atmos_stack: "{{ .atmos_stack }}"
      terraform_component: "{{ .component }}"
      terraform_workspace: "{{ .workspace }}"
      devops_team: '{{`{{ (datasource "s3-tags").tags.devops_team }}`}}'
      billing_team: '{{`{{ (datasource "s3-tags").tags.billing_team }}`}}'
      service: '{{`{{ (datasource "s3-tags").tags.service }}`}}'

# Atmos component configurations
components:
  terraform:
    vpc/1:
      metadata:
        component: vpc  # Point to the Terraform component in `components/terraform/vpc` folder
        inherits:
          # Inherit from the `vpc/defaults` base Atmos component, which defines the default 
          # configuration for all VPCs in the infrastructure.
          # The `vpc/defaults` base component is defined in the `catalog/vpc/defaults` 
          # manifest (which is imported above).
          # This inheritance makes the `vpc/1` Atmos component config DRY.
          - "vpc/defaults"
      vars:
        name: "vpc-1"

When executing an Atmos command like atmos terraform apply vpc/1 -s plat-ue2-dev, the above template will be processed in two steps:

• Evaluation 1:

  • datasources.s3-tags.url is set to s3://mybucket/dev/tags.json

  • the tags that use the datasource templates are set to the following:

    devops_team: '{{ (datasource "s3-tags").tags.devops_team }}'
    billing_team: '{{ (datasource "s3-tags").tags.billing_team }}'
    service: '{{ (datasource "s3-tags").tags.service }}'

• Evaluation 2:

  • all s3-tags datasources get executed, the JSON file s3://mybucket/dev/tags.json with the tags
    for the dev account is downloaded from the S3 bucket, and the tags are parsed and assigned in the
    terraform.vars.tags section

After executing the two evaluation steps, the resulting tags for the Atmos component vpc/1 in the stack plat-ue2-dev would look like this:

atmos_component: vpc/1
atmos_stack: plat-ue2-dev
terraform_component: vpc
terraform_workspace: plat-ue2-dev-vpc-1
devops_team: dev_networking
billing_team: billing_net
service: net

The tags will be added to all the AWS resources provisioned by the vpc Terraform component in the plat-ue2-dev stack.

v1.71.0

Update `atmos describe affected` command. Update docs @aknysh (#590)

what

• Update atmos describe affected command

• Add --clone-target-ref flag to the atmos describe affected command

• Update docs. Add "Remote State Backend" doc

  • https://atmos.tools/cli/commands/describe/affected/
  • https://atmos.tools/core-concepts/components/remote-state-backend/

why

• "Remote State Backend" doc describes how to override Terraform Backend Configuration to access components remote state, and how to do Brownfield development in Atmos

• Simplify the atmos describe affected command to not force it to clone the remote target reference (branch or tag), but instead just check it out (assuming the target reference is already on the local file system)

• Add --clone-target-ref flag to the atmos describe affected command for backwards compatibility. If the flag is passed, the command behaves as the old version (clones the target reference first from the remote origin)

breaking changes

If the atmos describe affected command was used in a GitHub Action (similar to https://github.com/cloudposse/github-action-atmos-affected-stacks), and the action performed a shallow Git clone (instead of a deep clone), it will break with an error that the target reference (branch) does not exist on the file system. There are a few ways to fix it:

• Use the flag --clone-target-ref=true to force the command to clone the target reference from the remote origin (this flag is addd for backwards compatibility)

  • atmos describe affected --clone-target-ref=true

• Update the GitHub Action to perform a deep-clone instead of a shallow-clone

    - uses: actions/checkout@v4
      with:
        fetch-depth: 0

• Perform a clone of the target branch into a separate directory and use the --repo-path=<dir> command line parameter to specify the path to the already cloned target repository (refer to https://atmos.tools/cli/commands/describe/affected#flags)

description

The atmos describe affected command uses two different Git commits to produce a list of affected Atmos components and stacks.

For the first commit, the command assumes that the current repo root is a Git checkout. An error will be thrown if the
current repo is not a Git repository (the .git/ folder does not exist or is configured incorrectly).

The second commit can be specified on the command line by using the --ref (Git References) or --sha (commit SHA) flags. The --sha takes precedence over the --ref flag.

How does it work?

The command performs the following:

• If the --repo-path flag is passed, the command uses it as the path to the already cloned target repo with which to compare the current working branch. I this case, the command will not clone and checkout the target reference, but instead will use the already cloned one to compare the current branch with. In this case, the --ref, --sha, --ssh-key and --ssh-key-password flags are not used, and an error will be thrown if the --repo-path flag and any of the --ref, --sha, --ssh-key or --ssh-key-password flags are provided at the same time

• Otherwise, if the --clone-target-ref=true flag is specified, the command clones (into a temp directory) the remote target with which to compare the current working branch. If the --ref flag or the commit SHA flag --sha are provided, the command uses them to clone and checkout the remote target. Otherwise, the HEAD of the remote origin is used (refs/remotes/origin/HEAD Git ref, usually the main branch)

• Otherwise, (if the --repo-path and --clone-target-ref=true flags are not passed), the command does not clone anything from the remote origin, but instead just copies the current repo into a temp directory and checks out the target reference with which to compare the current working branch. If the --ref flag or the commit SHA flag --sha are provided, the command uses them to check out. Otherwise, the HEAD of the remote origin is used (refs/remotes/origin/HEAD Git ref, usually the main branch). This requires that the target reference is already cloned by Git, and the information about it exists in the .git directory (in case of using a non-default branch as the target, Git deep clone needs to be executed instead of a shallow clone). This is the recommended way to execute the atmos describe affected command since it allows working with private repositories without providing the SSH credentials (--ssh-key and --ssh-key-password flags), since in this case Atmos does not access the remote origin and instead just checks out the target reference (which is already on the local file system)

• The command deep-merges all stack configurations from both sources: the current working branch and the target reference

• The command searches for changes in the component directories

• The command compares each stack manifest section of the stack configurations from both sources looking for differences

• And finally, the command outputs a JSON or YAML document consisting of a list of the affected components and stacks and what caused it to be affected

v1.70.0

Add `gomplate` datasources to `Go` templates in Atmos stack manifests. Update docs @aknysh (#582)

what

• Add gomplate datasources to Go templates in Atmos stack manifests
• Fix an issue with enabling/disabling Go templates in imports using the templates.settings.enabled section in atmos.yaml
• Update docs
  • https://atmos.tools/core-concepts/stacks/templating/

why

• Allow using Gomplate Datasources in Go templates in Atmos stack manifests
• The templates.settings.enabled section in atmos.yaml also affected templating in imports, but templating in imports should always be enabled regardless of enabling/disabling templating in Atmos stack manifests

description

Atmos supports Go templates in stack manifests.

Sprig Functions, Gomplate Functions and Gomplate Datasources are supported as well.

Configuration

Templating in Atmos stack manifests can be configured in the following places:

• In the templates.settings section in atmos.yaml CLI config file

• In the settings.templates.settings section in Atmos stack manifests. The settings.templates.settings section can be defined globally per organization, tenant, account, or per component. Atmos deep-merges the configurations from all scopes into the final result using inheritance.

Configuring templating in atmos.yaml CLI config file

Templating in Atmos stack manifests is configured in the atmos.yaml CLI config file in the templates.settings section:

# https://pkg.go.dev/text/template
templates:
  settings:
    enabled: true
    # https://masterminds.github.io/sprig
    sprig:
      enabled: true
    # https://docs.gomplate.ca
    # https://docs.gomplate.ca/functions
    gomplate:
      enabled: true
      # Timeout in seconds to execute the datasources
      timeout: 5
      # https://docs.gomplate.ca/datasources
      datasources:
        # 'http' datasource
        # https://docs.gomplate.ca/datasources/#using-file-datasources
        ip:
          url: "https://api.ipify.org?format=json"
          # https://docs.gomplate.ca/datasources/#sending-http-headers
          # https://docs.gomplate.ca/usage/#--datasource-header-h
          headers:
            accept:
              - "application/json"
        # 'file' datasources
        # https://docs.gomplate.ca/datasources/#using-file-datasources
        config-1:
          url: "./config1.json"
        config-2:
          url: "file:///config2.json"
        # `aws+smp` AWS Systems Manager Parameter Store datasource
        # https://docs.gomplate.ca/datasources/#using-awssmp-datasources
        secret-1:
          url: "aws+smp:///path/to/secret"
        # `aws+sm` AWS Secrets Manager datasource
        # https://docs.gomplate.ca/datasources/#using-awssm-datasource
        secret-2:
          url: "aws+sm:///path/to/secret"
        # `s3` datasource
        # https://docs.gomplate.ca/datasources/#using-s3-datasources
        s3-config:
          url: "s3://mybucket/config/config.json"

• templates.settings.enabled - a boolean flag to enable/disable the processing of Go templates in Atmos stack manifests. If set to false, Atmos will not process Go templates in stack manifests

• templates.settings.sprig.enabled - a boolean flag to enable/disable the Sprig Functions in Atmos stack manifests

• templates.settings.gomplate.enabled - a boolean flag to enable/disable the Gomplate Functions and Gomplate Datasources in Atmos stack manifests

• templates.settings.gomplate.timeout - timeout in seconds to execute Gomplate Datasources

• templates.settings.gomplate.datasources - a map of Gomplate Datasource definitions:

  • The keys of the map are the datasource names, which are used in Go templates in Atmos stack manifests.
    For example:

     terraform:
       vars:
         tags:
           provisioned_by_ip: '{{ (datasource "ip").ip }}'
           config1_tag: '{{ (datasource "config-1").tag }}'
           config2_service_name: '{{ (datasource "config-2").service.name }}'

  • The values of the map are the datasource definitions with the following schema:

    • url - the Datasource URL

    • headers - a map of HTTP request headers for the http datasource. The keys of the map are the header names. The values of the map are lists of values for the header.

      The following configuration will result in the accept: application/json HTTP header being sent with the HTTP request to the datasource:

      headers:
        accept:
          - "application/json"

Configuring templating in Atmos stack manifests

The settings.templates.settings section can be defined globally per organization, tenant, account, or per component. Atmos deep-merges the configurations from all scopes into the final result using inheritance.

For example, define Gomplate Datasources for the entire organization in the stacks/orgs/acme/_defaults.yaml stack manifest:

settings:
  templates:
    settings:
      gomplate:
        # 7 seconds timeout to execute the datasources
        timeout: 7
        # https://docs.gomplate.ca/datasources
        datasources:
          # 'file' datasources
          # https://docs.gomplate.ca/datasources/#using-file-datasources
          config-1:
            url: "./my-config1.json"
          config-3:
            url: "file:///config3.json"

Atmos deep-merges the configurations from the settings.templates.settings section in Atmos stack manifests with the templates.settings section in atmos.yaml CLI config file using inheritance.

The settings.templates.settings section in Atmos stack manifests takes precedence over the templates.settings section in atmos.yaml CLI config file, allowing you to define the global datasources in atmos.yaml and then add or override datasources in Atmos stack manifests for the entire organization, tenant, account, or per component.

For example, taking into account the configurations described above in atmos.yaml CLI config file and in the stacks/orgs/acme/_defaults.yaml stack manifest, the final datasources map will look like this:

gomplate:
  timeout: 7
  datasources:
    ip:
      url: "https://api.ipify.org?format=json"
      headers:
        accept:
          - "application/json"
    config-1:
      url: "./my-config1.json"
    config-2:
      url: "file:///config2.json"
    config-3:
      url: "file:///config3.json"

Note that the config-1 datasource from atmos.yaml was overridden with the config-1 datasource from the stacks/orgs/acme/_defaults.yaml stack manifest. The timeout attribute was overridden as well.

You can now use the datasources in Go templates in all Atmos sections that support Go templates.

Atmos sections supporting Go templates

You can use Go templates in the following Atmos sections to refer to values in the same or other sections:

• vars
• settings
• env
• metadata
• providers
• overrides
• backend
• backend_type

For example, let's say we have the following component configuration using Go templates:

component:
  terraform:
    vpc:
      settings:
        setting1: 1
        setting2: 2
        setting3: "{{ .vars.var3 }}"
        setting4: "{{ .settings.setting1 }}"
        component: vpc
        backend_type: s3
        region: "us-east-2"
        assume_role: "<role-arn>"
      backend_type: "{{ .settings.backend_type }}"
      metadata:
        component: "{{ .settings.component }}"
      providers:
        aws:
          region: "{{ .settings.region }}"
          assume_role: "{{ .settings.assume_role }}"
      env:
        ENV1: e1
        ENV2: "{{ .settings.setting1 }}-{{ .settings.setting2 }}"
      vars:
        var1: "{{ .settings.setting1 }}"
        var2: "{{ .settings.setting2 }}"
        var3: 3
        # Add the tags to all the resources provisioned by this Atmos component
        tags:
          atmos_component: "{{ .atmos_component }}"
          atmos_stack: "{{ .atmos_stack }}"
          atmos_manifest: "{{ .atmos_stack_file }}"
          region: "{{ .vars.region }}"
          terraform_workspace: "{{ .workspace }}"
          assumed_role: "{{ .providers.aws.assume_role }}"
          description: "{{ .atmos_component }} component provisioned in {{ .atmos_stack }} stack by assuming IAM role {{ .providers.aws.assume_role }}"
          # Examples of using the Sprig and Gom...

v1.69.0

In Atmos v1.55 (PR #515) we switched to using the TF_WORKSPACE environment variable for selecting Terraform workspaces when issuing Terraform commands. This had the unintended consequence that the workspace was no longer being set for use outside of Atmos as a side effect. This version reverses that change, once again setting the workspace via terraform workspace select so that it remains selected after Atmos finishes. Expand the details below for an extended explanation.

Use `terraform workspace select` to select workspace @Nuru (#580)

what

• Use terraform workspace select to select the workspace when needed
• Do not ask for Read permissions for output files
• Only attempt terraform workspace new when select returns with exit code 1; report any other kind of error

why

• Restores previous behavior, which includes fixing #574, and leaving the active workspace selected for use by terraform commands outside of Atmos
• Prevents users running under atmos terraform shell from having issues running terraform workspace commands or having the workspace remain selected after changing directories
• Atmos does not need Read permission on outputs, and redirected outputs may not be readable, causing unnecessary failures
• Avoid hiding unexpected errors

discussion

While Atmos is a powerful tool and for some there is a desire to make Atmos fully capable of replacing the terraform CLI entirely, we are not there yet. For now, there is a need to be able to use terraform alongside with atmos in some edge cases. Also, for some, there is a desire to bypass some of Atmos' overhead when running repeated tasks (especially failing tasks) by invoking terraform commands directly.

The issue of selecting a Terraform "workspace" highlights the kind of difficulty one runs into when designing a solution where Atmos and Terraform can work side by side. Terraform has the concept of, for each component, a "current workspace", which is a segregated Terraform state. Cloud Posse uses a different workspace for each deployment of a component (account and region and, where the component is deployed multiple times in the same account and region, each separate deployment).

The problem with a workspace selection is that it is stateful, modal, and mostly hidden. When you run a terraform command, the component it applies to is determined by the current working directory (one form of mode/state), and the workspace it uses is determined by either:

1. The most recent invocation of terraform workspace select in that directory
2. The value of the environment variable TF_WORKSPACE

with the environment variable taking precedence.

To further complicate matters, if you run terraform workspace select foo and the workspace foo has not been created, you get an error. Likewise, if you run TF_WORKSPACE=foo terraform init -reconfigure and the workspace foo does not exist, you get an error. But if you run TF_WORKSPACE=foo terraform plan and the workspace foo does not exist, it will silently and automatically be created for you (so beware of typos).

Whichever mechanism you choose, you are setting some kind of persistent state which affects what terraform commands affect, and it becomes easy to get lost as to what you are actually working on. Atmos handles this by always setting the state for its Terraform commands, which is admirable, and it can be more helpful to people using terraform directly if it helps those users manage this hidden state effectively.

Prior to Atmos version 1.55, its behavior was that if it ran any Terraform command that applied to a specific workspace (anything other than init), it selected the workspace, creating it if needed, and left the workspace selected. This means that if you working in the component directory, then any terraform commands like terraform state show would work on the component in the workspace of the stack you most recently used for that component. After running atmos terraform plan <component> -s <stack>, you could do everything else using plain Terraform commands if you wanted to (the only trick being that, for some commands, you would need to supply the -var-file that Atmos generated). While many users would never do things this way, in some use cases it can be particularly time saving. For example, you may want to run the same set of Terraform commands repeatedly on the same component, using different stacks, as can happen when migrating a component to a new version requires some manual steps.

As of Atmos 1.55, Atmos changed how it selects a workspace. It started using the environment variable TF_WORKSPACE instead of terraform workspace select. While this works for Atmos, it changes the behavior for running the bare terraform commands. No longer did Atmos set the workspace for you, nor would atmos terraform workspace <component> -s <stack> set the workspace for you, either. In fact, there was no longer any direct way to find out the correct workspace name to use. Your plain terraform commands worked on the default workspace, or whatever one was selected the last time you used an old version of Atmos.

So in summary, the change made using terraform harder to use, without appreciably making atmos easier to use. The change did eliminate a bug that was causing problems, and the consequences were not obvious (nor universally considered negative), so it was made. This PR fixes the bug a different way and restores the behavior of v1.54 and earlier.

In the long run, we are seeking ways to make these kinds of behaviors optional and easier to choose. For now, we are just reverting to the "tried-and-true" behavior to avoid further surprises.

references

• Reverts #515
• Fixes #574

v1.68.0

Add `gomplate` templating engine to Atmos stack manifests. Update docs @aknysh (#578)

Breaking changes

If you used Go templates in Atmos stack manifest before, they were enabled by default.
Starting with this release, you have to explicitly enable Go templating in atmos.yaml and enable either Sprig or Gomplate (or both).
See the description below.

what

• Add gomplate templating engine to Atmos stack manifests
• Update docs:
  • https://atmos.tools/core-concepts/stacks/templating/
  • https://atmos.tools/cli/configuration/

why

• Allow using the rich collection of the Gomplate Functions in Atmos stack manifest

• NOTE: The Gomplate Datasources, including configuring the datasources in Atmos stack manifests and using Inheritance for datasources, will be added in the follow up PR

Description

Configuration

Templating in Atmos stack manifests is configured in the atmos.yaml CLI config file in the templates section:

# https://pkg.go.dev/text/template
templates:
  settings:
    enabled: true
    # https://masterminds.github.io/sprig
    sprig:
      enabled: true
    # https://docs.gomplate.ca
    gomplate:
      enabled: true

• templates.settings.enabled - a boolean flag to enable/disable the processing of Go templates in Atmos stack manifests. If set
  to false, Atmos will not process Go templates in stack manifests

• templates.settings.sprig.enabled - a boolean flag to enable/disable the Sprig Functions
  in Atmos stack manifests

• templates.settings.gomplate.enabled - a boolean flag to enable/disable the Gomplate Functions
  in Atmos stack manifests

Atmos sections supporting Go templates

You can use Go templates in the following Atmos sections to refer to values in the same or other sections:

• vars
• settings
• env
• metadata
• providers
• overrides
• backend
• backend_type

For example, let's say we have the following component configuration using Go templates:

component:
  terraform:
    vpc:
      settings:
        setting1: 1
        setting2: 2
        setting3: "{{ .vars.var3 }}"
        setting4: "{{ .settings.setting1 }}"
        component: vpc
        backend_type: s3
        region: "us-east-2"
        assume_role: "<role-arn>"
      backend_type: "{{ .settings.backend_type }}"
      metadata:
        component: "{{ .settings.component }}"
      providers:
        aws:
          region: "{{ .settings.region }}"
          assume_role: "{{ .settings.assume_role }}"
      env:
        ENV1: e1
        ENV2: "{{ .settings.setting1 }}-{{ .settings.setting2 }}"
      vars:
        var1: "{{ .settings.setting1 }}"
        var2: "{{ .settings.setting2 }}"
        var3: 3
        # Add the tags to all the resources provisioned by this Atmos component
        tags:
          atmos_component: "{{ .atmos_component }}"
          atmos_stack: "{{ .atmos_stack }}"
          atmos_manifest: "{{ .atmos_stack_file }}"
          region: "{{ .vars.region }}"
          terraform_workspace: "{{ .workspace }}"
          assumed_role: "{{ .providers.aws.assume_role }}"
          description: "{{ .atmos_component }} component provisioned in {{ .atmos_stack }} stack by assuming IAM role {{ .providers.aws.assume_role }}"
          # Examples of using the Gomplate and Sprig functions
          # https://docs.gomplate.ca/functions/strings
          atmos_component_description: "{{ strings.Title .atmos_component }} component {{ .vars.name | strings.Quote }} provisioned in the stack {{ .atmos_stack | strings.Quote }}"
          # https://masterminds.github.io/sprig/os.html
          provisioned_by_user: '{{ env "USER" }}'

When executing Atmos commands like atmos describe component and atmos terraform plan/apply, Atmos processes all the template tokens in the manifest and generates the final configuration for the component in the stack:

settings:
  setting1: 1
  setting2: 2
  setting3: 3
  setting4: 1
  component: vpc
  backend_type: s3
  region: us-east-2
  assume_role: <role-arn>
backend_type: s3
metadata:
  component: vpc
providers:
  aws:
    region: us-east-2
    assume_role: <role-arn>
env:
  ENV1: e1
  ENV2: 1-2
vars:
  var1: 1
  var2: 2
  var3: 3
  tags:
    assumed_role: <role-arn>
    atmos_component: vpc
    atmos_component_description: Vpc component "common" provisioned in the stack "plat-ue2-dev"
    atmos_manifest: orgs/acme/plat/dev/us-east-2
    atmos_stack: plat-ue2-dev
    description: vpc component provisioned in plat-ue2-dev stack by assuming IAM role <role-arn>
    provisioned_by_user: <user>
    region: us-east-2
    terraform_workspace: plat-ue2-dev

Use-cases

While Go templates in Atmos stack manifests offer great flexibility for various use-cases, one of the obvious use-cases is to add a standard set of tags to all the resources in the infrastructure.

For example, by adding this configuration to the stacks/orgs/acme/_defaults.yaml Org-level stack manifest:

terraform:
  vars:
    tags:
      atmos_component: "{{ .atmos_component }}"
      atmos_stack: "{{ .atmos_stack }}"
      atmos_manifest: "{{ .atmos_stack_file }}"
      terraform_workspace: "{{ .workspace }}"
      # Examples of using the Gomplate and Sprig functions
      # https://docs.gomplate.ca/functions/strings
      atmos_component_description: "{{ strings.Title .atmos_component }} component {{ .vars.name | strings.Quote }} provisioned in the stack {{ .atmos_stack | strings.Quote }}"
      # https://masterminds.github.io/sprig/os.html
      provisioned_by_user: '{{ env "USER" }}'

The tags will be processed and automatically added to all the resources provisioned in the infrastructure.

Excluding templates from processing by Atmos

If you need to provide Go templates to external systems (e.g. ArgoCD or Datadog) verbatim and prevent Atmos from processing the templates, use double curly braces + backtick + double curly braces instead of just double curly braces:

{{`{{  instead of  {{

}}`}}  instead of  }}

For example:

components:
  terraform:

    eks/argocd:
      metadata:
        component: "eks/argocd"
      vars:
        enabled: true
        name: "argocd"
        chart_repository: "https://argoproj.github.io/argo-helm"
        chart_version: 5.46.0

        chart_values:
          template-github-commit-status:
            message: |
              Application {{`{{ .app.metadata.name }}`}} is now running new version.
            webhook:
              github-commit-status:
                method: POST
                path: "/repos/{{`{{ call .repo.FullNameByRepoURL .app.metadata.annotations.app_repository }}`}}/statuses/{{`{{ .app.metadata.annotations.app_commit }}`}}"
                body: |
                  {
                    {{`{{ if eq .app.status.operationState.phase "Running" }}`}} "state": "pending"{{`{{end}}`}}
                    {{`{{ if eq .app.status.operationState.phase "Succeeded" }}`}} "state": "success"{{`{{end}}`}}
                    {{`{{ if eq .app.status.operationState.phase "Error" }}`}} "state": "error"{{`{{end}}`}}
                    {{`{{ if eq .app.status.operationState.phase "Failed" }}`}} "state": "error"{{`{{end}}`}},
                    "description": "ArgoCD",
                    "target_url": "{{`{{ .context.argocdUrl }}`}}/applications/{{`{{ .app.metadata.name }}`}}",
                    "context": "continuous-delivery/{{`{{ .app.metadata.name }}`}}"
                  }

When Atmos processes the templates in the manifest shown above, it renders them as raw strings allowing sending the templates to the external system for processing:

chart_values:
  template-github-commit-status:
    message: |
      Application {{ .app.metadata.name }} is now running new version.
    webhook:
      github-commit-status:
        method: POST
        path: "/repos/{{ call .repo.FullNameByRepoURL .app.metadata.annotations.app_repository }}/statuses/{{ .app.metadata.annotations.app_commit }}"
        body: |
          {
            {{ if eq .app.status.operationState.phase "Running" }} "state": "pending"{{end}}
            {{ if eq .app.status.operationState.phase "Succeeded" }} "state": "success"{{end}}
            {{ if eq .app.status.operationState.phase "Error" }} "state": "error"{{end}}
            {{ if eq .app.status.operationState.phase "Failed" }} "state": "error"{{end}},
            "description": "ArgoCD",
            "target_url": "{{ .context.argocdUrl }}/applications/{{ .app.metadata.name }}",
            "context": "continuous-delivery/{{ .app.metadata.name }}"
          }

The printf template function is also supported and can be used instead of double curly braces + backtick + double curly braces.

The following examples produce the same result:

chart_values:
  template-github-commit-status:
    message: >-
      Application {{`{{ .app.metadata.name }}`}} is now running new version.

chart_values:
  template-github-commit-status:
    message: "Application {{`{{ .app.metadata.name }}`}} is now running new version."

chart_values:
  template-github-commit-status:
    message: >-
      {{ printf "Application {{ .app.metadata.name }} is now running new version." ...

v1.67.0

Add Terraform Cloud backend. Add/update docs @aknysh (#572)

what

• Add Terraform Cloud backend
• Add docs:
  • Terraform Workspaces in Atmos
  • Terraform Backends. Describes how to configure backends for AWS S3, Azure Blob Storage, Google Cloud Storage, and Terraform Cloud

why

• Allow configuring and using the Terraform Cloud backend
• Document existing and new Atmos features

docs

Terraform Cloud backend uses a cloud block to specify which organization and workspace(s) to use.

To configure the Terraform Cloud backend in Atmos, add the following config to an Atmos manifest in _defaults.yaml:

terraform:
  backend_type: cloud
  backend:
    cloud:
      organization: "my-org"
      hostname: "app.terraform.io"
      workspaces:
        # Parameters for workspaces

For each component, you can optionally specify the workspaces.name parameter similar to the following:

components:
  terraform:
    my-component:
      # Optional backend configuration for the component
      backend:
        cloud:
          workspaces:
            name: "my-component-workspace"

If auto_generate_backend_file is set to true in the atmos.yaml CLI config file in the components.terraform section,
Atmos will deep-merge the backend configurations from the _defaults.yaml manifests and from the component itself, and will generate a backend config JSON file backend.tf.json in the component's folder, similar to the following example:

{
  "terraform": {
    "cloud": {
      "hostname": "app.terraform.io",
      "organization": "my-org",
      "workspaces": {
        "name": "my-component-workspace"
      }
    }
  }
}

Instead of specifying the workspaces.name parameter for each component in the component manifests, you can use
the {terraform_workspace} token in the cloud backend config in the _defaults.yaml manifest.
The token {terraform_workspace} will be automatically replaced by Atmos with the Terraform workspace for each component. This will make the entire configuration DRY.

terraform:
  backend_type: cloud
  backend:
    cloud:
      organization: "my-org"
      hostname: "app.terraform.io"
      workspaces:
        # The token `{terraform_workspace}` will be automatically replaced with the
        # Terraform workspace for each Atmos component
        name: "{terraform_workspace}"

v1.66.0

Add `stacks.name_template` section to `atmos.yaml`. Add `Go` templating to Atmos stack manifests @aknysh (#560)

what

• Add stacks.name_template section to atmos.yaml
• Add Go templating to Atmos stack manifests
• Update docs
  • https://atmos.tools/cli/configuration/#stacks
  • https://atmos.tools/core-concepts/stacks/templating/

why

• Add stacks.name_template section to atmos.yaml - stacks.name_template section allows greater flexibility in defining the naming convention for the top-level Atmos stacks

  • stacks.name_pattern configures the name pattern for the top-level Atmos stacks using the context variables namespace, tenant, environment and stage as the tokens. Depending on the structure of your organization, OUs, accounts and regions, set stacks.name_pattern to the following:

    • name_pattern: {stage} - if you use just one region and a few accounts (stages) in just one organization and one OU. In this case, the top-level Atmos stacks will use just the stage (account) in their names, and to provision the Atmos components in the top-level stacks, you will be executing Atmos commands like atmos terraform apply <component> --stack dev, atmos terraform apply <component> --stack staging and atmos terraform apply <component> --stack prod

    • name_pattern: {environment}-{stage} - if you have multiple regions and accounts (stages) in just one organization and one OU. In this case, the top-level Atmos stacks will use the environment (region) and stage (account) in their names, and to provision the Atmos components in the top-level stacks, you will be executing Atmos commands like atmos terraform apply <component> --stack ue2-dev, atmos terraform apply <component> --stack uw2-staging and atmos terraform apply <component> --stack ue1-prod. Note that the name_pattern can also be defined as {stage}-{environment}, in which case the Atmos commands will look like atmos terraform apply <component> --stack dev-ue2

    • name_pattern: {tenant}-{environment}-{stage} - if you have multiple regions, OUs (tenants) and accounts (stages) in just one organization. In this case, the top-level Atmos stacks will use the tenant, environment (region) and stage (account) in their names, and to provision the Atmos components in the top-level stacks, you will be executing Atmos commands like atmos terraform apply <component> --stack plat-ue2-dev, atmos terraform apply <component> --stack core-uw2-staging and atmos terraform apply <component> --stack plat-ue1-prod, where plat and core are the OUs/tenants in your organization

    • name_pattern: {namespace}-{tenant}-{environment}-{stage} - if you have a multi-org, multi-tenant, multi-account and multi-region architecture. In this case, the top-level Atmos stacks will use the namespace, tenant, environment (region) and stage (account) in their names, and to provision the Atmos components in the top-level stacks, you will be executing Atmos commands like atmos terraform apply <component> --stack org1-plat-ue2-dev, atmos terraform apply <component> --stack org2-core-uw2-staging and atmos terraform apply <component> --stack org2-plat-ue1-prod, where org1 and org2 are the organization names (defined as namespace in the corresponding _defaults.yaml config files for the organizations)

  • stacks.name_template serves the same purpose as stacks.name_pattern (defines the naming convention for the top-level Atmos stacks), but provides much more functionality. Instead of using the predefined context variables as tokens, it uses Go templates. Sprig Functions are supported as well

    • For the Go template tokens, and you can use any Atmos sections (e.g. vars, providers, settings)
      that the Atmos command atmos describe component <component> -s <stack> generates for a component in a stack.

    • name_template: "{{.vars.tenant}}-{{.vars.environment}}-{{.vars.stage}}" defines the same name pattern for the top-level Atmos stacks as name_pattern: "{tenant}-{environment}-{stage}" does

    • Since stacks.name_template allows using any variables form the vars section (and other sections), you can define
      your own naming convention for your organization or for different clouds (AWS, Azure, GCP). For example, in the
      corresponding _defaults.yaml stack manifests, you can use the following variables:

      • org instead of namespace
      • division instead of tenant
      • region instead of environment
      • account instead of stage

      Then define the following stacks.name_template in atmos.yaml:

      stacks:
        name_template: "{{.vars.division}}-{{.vars.account}}-{{.vars.region}}"

      You will be able to execute all Atmos commands using the newly defined naming convention:

      atmos terraform plan <component> -s <division-account-region>
      atmos terraform apply <component> -s <division-account-region>
      atmos describe component <component> -s <division-account-region>

      NOTE:
      Use either stacks.name_pattern or stacks.name_template to define the naming convention for the top-level Atmos stacks. stacks.name_template has higher priority. If stacks.name_template is specified, stacks.name_pattern will be ignored.

• Add Go templating to Atmos stack manifests

  Atmos supports Go templates in stack manifests. Sprig Functions are supported as well.

  You can use Go templates in the following Atmos section to refer to values in the same or other sections:

  • vars
  • settings
  • env
  • metadata
  • providers
  • overrides
  • backend
  • backend_type
  
  

  NOTE:
  In the template tokens, you can refer to any value in any section that the Atmos command atmos describe component <component> -s <stack>generates.

  For example, let's say we have the following component configuration using Go templates:

  component:
    terraform:
      vpc:
        settings:
          setting1: 1
          setting2: 2
          setting3: "{{ .vars.var3 }}"
          setting4: "{{ .settings.setting1 }}"
          component: vpc
          backend_type: s3
          region: "us-east-2"
          assume_role: "<role-arn>"
        backend_type: "{{ .settings.backend_type }}"
        metadata:
          component: "{{ .settings.component }}"
        providers:
          aws:
            region: "{{ .settings.region }}"
            assume_role: "{{ .settings.assume_role }}"
        env:
          ENV1: e1
          ENV2: "{{ .settings.setting1 }}-{{ .settings.setting2 }}"
        vars:
          var1: "{{ .settings.setting1 }}"
          var2: "{{ .settings.setting2 }}"
          var3: 3
          # Add the tags to all the resources provisioned by this Atmos component
          tags:
            atmos_component: "{{ .atmos_component }}"
            atmos_stack: "{{ .atmos_stack }}"
            atmos_manifest: "{{ .atmos_stack_file }}"
            region: "{{ .vars.region }}"
            terraform_workspace: "{{ .workspace }}"
            assumed_role: "{{ .providers.aws.assume_role }}"
            description: "{{ .atmos_component }} component provisioned in {{ .atmos_stack }} stack by assuming IAM role {{ .providers.aws.assume_role }}"
            # `provisioned_at` uses the Sprig functions
            # https://masterminds.github.io/sprig/date.html
            # https://pkg.go.dev/time#pkg-constants
            provisioned_at: '{{ dateInZone "2006-01-02T15:04:05Z07:00" (now) "UTC" }}'

  When executing Atmos commands like atmos describe component and atmos terraform plan/apply, Atmos processes all the template tokens in the manifest and generates the final configuration for the component in the stack:

  settings:
    setting1: 1
    setting2: 2
    setting3: 3
    setting4: 1
    component: vpc
    backend_type: s3
    region: us-east-2
    assume_role: <role-arn>
  backend_type: s3
  metadata:
    component: vpc
  providers:
    aws:
      region: us-east-2
      assume_role: <role-arn>
  env:
    ENV1: e1
    ENV2: 1-2
  vars:
    var1: 1
    var2: 2
    var3: 3
    tags:
      assumed_role: <role-arn>
      atmos_component: vpc
      atmos_manifest: orgs/acme/plat/dev/us-east-2
      atmos_stack: plat-ue2-dev
      description: vpc component provisioned in plat-ue2-dev stack by assuming IAM role <role-arn>
      provisioned_at: "2024-03-12T16:18:24Z"
      region: us-east-2
      terraform_workspace: plat-ue2-dev

  
  

  While Go templates in Atmos stack manifests offer great flexibility for various use-cases, one of the obvious use-cases
  is to add a standard set of tags to all the resources in the infrastructure.

  For example, by adding this configuration to the stacks/orgs/acme/_defaults.yaml Org-level stack manifest:

  terraform:
    vars:
      tags:
        atmos_component: "{{ .atmos_component }}"
        atmos_stack: "{{ .atmos_stack }}"
        atmos_manifest: "{{ .atmos_stack_file }}"
        terraform_workspace: "{{ .workspace }}"
        provisioned_at: '{{ dateInZone "2006-01-02T15:04:05Z07:00" (now) "UTC" }}'

  the tags will be processed and automatically added to all the resources provisioned in the infrastructure.

v1.65.0

Add `providers` section to Atmos manifests. Update docs @aknysh (#555)

what

• Add providers section to Atmos manifests
• Auto-generate the prefix attribute for Atmos components for Terraform backend gcs for GCP
• Update docs (https://atmos.tools/core-concepts/components/terraform-providers/)

why

Terraform utilizes plugins known as providers for communication with cloud providers, SaaS providers, and various APIs.

In order for Terraform to install these providers, the corresponding Terraform configurations need to explicitly state what providers are required. Furthermore, certain providers require additional configuration, such as specifying endpoint URLs or cloud regions, before they can be used.

Provider Configuration in Terraform

When working with Terraform, you specify provider configurations in your Terraform code. This involves declaring which providers your infrastructure requires and providing any necessary configuration parameters. These parameters may include endpoint URLs, cloud regions, access credentials, or any other provider-specific configuration parameters.

To declare a provider in Terraform, use a provider block within your Terraform configuration files, usually in a providers.tf file in the component (a.k.a. root module) directory. The provider block specifies the provider type and all the necessary configuration parameters.

Here's an AWS provider configuration example for a vpc component. The provider config is defined in the components/terraform/vpc/providers.tf file:

  provider "aws" {
    region = "us-east-2"
    assume_role = "IAM Role ARN"
  }

In this example, the aws provider block includes the region and IAM role required for Terraform to communicate with the AWS services.

By correctly defining provider configurations in your Terraform code, you ensure that Terraform can seamlessly install, configure, and use the necessary plugins to manage your infrastructure across various cloud and services.

Provider Configuration and Overrides in Atmos Manifests

Atmos allows you to define and override provider configurations using the providers section in Atmos stack manifests.
The section can be defined globally for the entire organization, OU/tenant, account, region, or per component.

For example, the providers section at the global scope can look like this:

terraform:
  providers:
    aws:
      region: "us-east-2"
      assume_role: "IAM Role ARN"

Similarly, it can be defined (or overridden) at the OU/tenant, account and region scopes in the corresponding _defaults.yaml stack manifests.

If you want to override a provider configuration for a specific component, use the component.terraform.<component>.providers section. For example, the following config can be used to override the assume_role parameter just for the vpc component:

components:
  terraform:
    vpc:
      providers:
        aws:
         assume_role: "IAM Role ARN for VPC"

You can include the providers sections in any Atmos stack manifest at any level of inheritance. Atmos will process, deep-merge and override all the providers configurations for a component in the following order:

• Global scopes (terraform.providers sections for the Org, OUs, accounts and regions)
• Base component scope (component.terraform.<base_component>.providers section)
• Current component scope (component.terraform.<component>.providers section)

Refer to Atmos Component Inheritance for more information on all types of component inheritance supported by Atmos.

When you define the providers sections, Atmos processes the inheritance chain for a component and generates a
file providers_override.tf.json in the component's folder with the final values for all the defined providers.

For example:

> atmos terraform plan vpc -s plat-ue2-prod --logs-level=Trace

Variables for the component 'vpc' in the stack 'plat-ue2-prod':
  environment: ue2
  max_subnet_count: 3
  name: common
  namespace: cp
  region: us-east-2
  stage: prod
  tenant: plat

Writing the variables to file:
components/terraform/vpc/plat-ue2-prod.terraform.tfvars.json

Writing the provider overrides to file:
components/terraform/vpc/providers_override.tf.json

The generated providers_override.tf.json file would look like this:

{
    "provider": {
      "aws": {
        "assume_role": "IAM Role ARN for VPC"
      }
    }
}

Terraform then uses the values in the generated providers_override.tf.json to override the parameters for all the providers in the file.

References

• Terraform Providers
• Terraform Override Files

v1.64.3

🚀 Enhancements

include some protection on the azurerm backend when global does not exist @aochsner (#548)

what

When using the azurerm backend, the logic assumes a global key is set and prepends that to the component key. However, when it doesn't exist it causes atmos to crash. This checks if a global key is set and if not, then don't prepend anything.

why

• don't require a global key

v1.64.2

Add Atmos CLI command aliases @aknysh (#547)

what

• Add Atmos CLI command aliases
• Update docs https://atmos.tools/cli/configuration/#aliases

why

An alias lets you create a shortcut name for an existing CLI command. Any CLI command can be aliased, including the Atmos native commands like terraform apply or describe stacks, as well as Atmos Custom Commands.

CLI command aliases are configured in the aliases section in the atmos.yaml file.

For example:

aliases:
  # Aliases for Atmos native commands
  tf: terraform
  tp: terraform plan
  up: terraform apply
  down: terraform destroy
  ds: describe stacks
  dc: describe component
  # Aliases for Atmos custom commands
  ls: list stacks
  lc: list components

Execute an alias as you would any Atmos native or custom command:

> atmos ls

plat-ue2-dev
plat-ue2-prod
plat-ue2-staging
plat-uw2-dev
plat-uw2-prod
plat-uw2-staging

The aliases configured in the aliases section automatically appear in Atmos help, and are shown as
alias for '<command>'.

For example:

An alias automatically supports all command line arguments and flags that the aliased command accepts.

For example:

• atmos up <component> -s <stack> supports all the parameters from the aliased command atmos terraform apply <component> -s <stack>
• atmos dc <component> -s <stack> supports all the parameters from the aliased command atmos describe component <component> -s <stack>

references

• closes #506

Bump the website group in /website with 1 update @dependabot (#545)

Bumps the website group in /website with 1 update: @mdx-js/react.

Updates @mdx-js/react from 3.0.0 to 3.0.1

Release notes

Sourced from @​mdx-js/react's releases.

3.0.1

Fix

• 49fd094a Fix generated JSX pragmas for new babel by @​remcohaszing in mdx-js/mdx#2438
• d1fface1 Refactor to handle uncaught webpack errors by @​remcohaszing in mdx-js/mdx#2398

Types

• c03e16c6 Add declaration maps
• 85c5dd05 Fix webpack loader types by @​remcohaszing in mdx-js/mdx#2386

Site

• 38f7a5c9 Fix reset of playground render errors by @​ManasMakde in mdx-js/mdx#2392
• 680445df Add note on broken react runtime types by @​ky1ejs in mdx-js/mdx#2383

Full Changelog: mdx-js/mdx@3.0.0...3.0.1

Commits

• 78aee48 3.0.1
• c03e16c Add declaration maps
• 0e6dd7a Update dev-dependencies
• 038bb68 Refactor some version numbers in docs
• See full diff in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

---

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot merge will merge this PR after your CI passes on it
• @dependabot squash and merge will squash and merge this PR after your CI passes on it
• @dependabot cancel merge will cancel a previously requested merge and block automerging
• @dependabot reopen will reopen this PR if it is closed
• @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
• @dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency
• @dependabot ignore <dependency name> major version will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself)
• @dependabot ignore <dependency name> minor version will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself)
• @dependabot ignore <dependency name> will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself)
• @dependabot unignore <dependency name> will remove all of the ignore conditions of the specified dependency
• @dependabot unignore <dependency name> <ignore condition> will remove the ignore condition of the specified dependency and ignore conditions

🚀 Enhancements

Simplify install page @osterman (#544)

what

• Use tabs for each installation method
• Add new "installer" script installation method

why

• Make it easier to get started