Add support for a custom aws endpoint

.circleci/config.yml

@@ -316,6 +316,56 @@ jobs:
       - store_test_results:
           path: /tmp/logs
 
+  mock_aws_test:
+    <<: *env
+    docker:
+      - image: cimg/python:3.10.2
+    steps:
+      - attach_workspace:
+          at: /home/circleci
+
+      - run:
+          <<: *install_gruntwork_utils
+
+      # The weird way you have to set PATH in Circle 2.0
+      - run: |
+          echo 'export PATH=$HOME/.local/bin:$HOME/terraform:$HOME/packer:$PATH' >> $BASH_ENV
+      - run:
+          name: install mock aws server
+          command: |
+            pip install moto[server]
+      - run:
+          name: install aws cli
+          command: |
+            curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
+            unzip awscliv2.zip
+            sudo ./aws/install
+      - run:
+          name: run mock aws server
+          command: |
+            mkdir -p /tmp/logs
+            moto_server | tee /tmp/logs/moto_server_output.log
+          background: true
+      - run:
+          name: wait for mock aws server
+          command: |
+            # Disable AWS_PAGER because the CI system isn't an
+            # interactive terminal, and will throw warnings otherwise:
+            # https://stackoverflow.com/questions/60122188/how-to-turn-off-the-pager-for-aws-cli-return-value
+            while ! AWS_PAGER="" AWS_REGION=us-west-2 AWS_ACCESS_KEY_ID=dummy AWS_SECRET_ACCESS_KEY=dummy aws --endpoint http://localhost:5000 sts get-caller-identity; do
+              echo "Waiting for local mock AWS server"
+              sleep 1
+            done
+      - run:
+          command: |
+            mkdir -p /tmp/logs
+            run-go-tests --packages "-tags=mockaws -run TestTerraformAwsEndpointExample ./test/" | tee /tmp/logs/test_output.log
+
+      # Store test result and log artifacts for browsing purposes
+      - store_artifacts:
+          path: /tmp/logs
+      - store_test_results:
+          path: /tmp/logs
 
   deploy:
     <<: *defaults
@@ -386,6 +436,16 @@ workflows:
             tags:
               only: /^v.*/
 
+      - mock_aws_test:
+          context:
+            - AWS__PHXDEVOPS__circle-ci-test
+            - GITHUB__PAT__gruntwork-ci
+          requires:
+            - setup
+          filters:
+            tags:
+              only: /^v.*/
+
       - deploy:
           context:
             - AWS__PHXDEVOPS__circle-ci-test

docs/_docs/02_testing-best-practices/tools-and-plugins.md

@@ -16,7 +16,21 @@ This page contains a list of tools and plugins for Terratest to make integration
 
 - [Tools and Plugins](#tools-and-plugins)
   - [Terratest Maven Plugin](#terratest-maven-plugin)
+  - [Custom AWS Endpoints](#custom-aws-endpoints)
+  - [S3 Path Style Endpoints](#s3-path-style-endpoints)
 
 ### Terratest Maven Plugin
 
 The Terratest Maven Plugin aims to bring Terratest to the JVM world. Create your Go based tests beside your Java code with Maven and run them together. You can export the results into Json or an HTML page. As the plugin is MIT licensed, it is easy and painless to integrate into any Java+Maven combination. To learn more check out the website: [Terratest Maven Plugin](https://terratest-maven-plugin.github.io) and the [GitHub repository](https://github.com/terratest-maven-plugin/terratest-maven-plugin)
+
+### Custom AWS Endpoints
+
+Terratest can be used with a custom AWS endpoint. One of the use cases for this is testing your automation using a mock AWS server such as [Moto in standalone server mode](https://docs.getmoto.org/en/latest/docs/getting_started.html#stand-alone-server-mode). This allows you to test your AWS based automation without an AWS account.
+
+To configure this, set `TERRATEST_CUSTOM_AWS_ENDPOINT` to the endpoint that you want Terratest to use for accessing AWS resources. You must also configure your automation to use the same endpoint, for example [Terraform supports custom AWS endpoints as well](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/guides/custom-service-endpoints).
+
+### S3 Path Style Endpoints
+
+By default, the golang client will interact with s3 in a way that uses subdomains to specify information about the bucket. If you have issues with subdomains resolving properly, set `TERRATEST_AWS_S3_USE_PATH_STYLE_ENDPOINT` to `1` to tell Terratest to use [path style access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#path-style-access) rather than subdomains for this information.
+
+Note that AWS is [deprecating path style endpoints](https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/) someday, but it seems like enough users are pushing against it that it's not scheduled for a specific time.

examples/terraform-aws-endpoint-example/README.md

@@ -0,0 +1,28 @@
+# Terraform AWS Endpoint Example
+
+This folder contains a simple Terraform module to demonstrate using custom
+endpoints. It's deploying some AWS resources to `http://localhost:5000`, which
+is the default port for [moto running in server
+mode](http://docs.getmoto.org/en/latest/docs/server_mode.html). This allows for
+testing terraform modules locally with no connection to AWS.
+
+Check out
+[test/terraform_aws_endpoint_example_test.go](/test/terraform_aws_endpoint_example_test.go)
+to see how you can write automated tests for this module.
+
+## Running this module manually
+
+1. Run [Moto locally in server mode](http://docs.getmoto.org/en/latest/docs/server_mode.html)
+1. Install [Terraform](https://www.terraform.io/) and make sure it's on your `PATH`.
+1. Run `terraform init`.
+1. Run `terraform apply`.
+1. When you're done, run `terraform destroy`.
+
+## Running automated tests against this module
+
+1. Run [Moto locally in server mode](http://docs.getmoto.org/en/latest/docs/server_mode.html)
+1. Install [Terraform](https://www.terraform.io/) and make sure it's on your `PATH`.
+1. Install [Golang](https://golang.org/) and make sure this code is checked out into your `GOPATH`.
+1. `cd test`
+1. `dep ensure`
+1. `go test -v -run TestTerraformAwsEndpointExample`

examples/terraform-aws-endpoint-example/main.tf

@@ -0,0 +1,138 @@
+# ---------------------------------------------------------------------------------------------------------------------
+# PIN TERRAFORM VERSION TO >= 0.12
+# The examples have been upgraded to 0.12 syntax
+# ---------------------------------------------------------------------------------------------------------------------
+provider "aws" {
+  region     = var.region
+  access_key = "dummy"
+  secret_key = "dummy"
+
+  # By default, terraform will interact with s3 in a way that uses subdomains
+  # to specify information about the bucket. If you have issues with subdomains
+  # of "localhost" not resolving properly, set this to true to tell terraform
+  # to use url paths rather than subdomains for this information.
+  # See: https://registry.terraform.io/providers/hashicorp/aws/latest/docs#s3_use_path_style
+  s3_use_path_style = "true"
+
+  endpoints {
+    sts = "http://localhost:5000"
+    s3  = "http://localhost:5000"
+  }
+}
+
+terraform {
+  # This module is now only being tested with Terraform 0.13.x. However, to make upgrading easier, we are setting
+  # 0.12.26 as the minimum version, as that version added support for required_providers with source URLs, making it
+  # forwards compatible with 0.13.x code.
+  required_version = ">= 0.12.26"
+}
+
+# ---------------------------------------------------------------------------------------------------------------------
+# DEPLOY A S3 BUCKET WITH VERSIONING ENABLED INCLUDING TAGS TO A LOCAL ENDPOINT
+# See test/terraform_aws_endpoint_example_test.go for how to write automated tests for this code.
+# ---------------------------------------------------------------------------------------------------------------------
+
+# Deploy and configure test S3 bucket with versioning and access log
+resource "aws_s3_bucket" "test_bucket" {
+  bucket = "${local.aws_account_id}-${var.tag_bucket_name}"
+
+  tags = {
+    Name        = var.tag_bucket_name
+    Environment = var.tag_bucket_environment
+  }
+}
+
+resource "aws_s3_bucket_logging" "test_bucket" {
+  bucket        = aws_s3_bucket.test_bucket.id
+  target_bucket = aws_s3_bucket.test_bucket_logs.id
+  target_prefix = "TFStateLogs/"
+}
+
+resource "aws_s3_bucket_versioning" "test_bucket" {
+  bucket = aws_s3_bucket.test_bucket.id
+  versioning_configuration {
+    status = "Enabled"
+  }
+}
+
+resource "aws_s3_bucket_acl" "test_bucket" {
+  bucket = aws_s3_bucket.test_bucket.id
+  acl    = "private"
+}
+
+
+# Deploy S3 bucket to collect access logs for test bucket
+resource "aws_s3_bucket" "test_bucket_logs" {
+  bucket = "${local.aws_account_id}-${var.tag_bucket_name}-logs"
+
+  tags = {
+    Name        = "${local.aws_account_id}-${var.tag_bucket_name}-logs"
+    Environment = var.tag_bucket_environment
+  }
+
+  force_destroy = true
+}
+
+resource "aws_s3_bucket_acl" "test_bucket_logs" {
+  bucket = aws_s3_bucket.test_bucket_logs.id
+  acl    = "log-delivery-write"
+}
+
+# Configure bucket access policies
+
+resource "aws_s3_bucket_policy" "bucket_access_policy" {
+  count  = var.with_policy ? 1 : 0
+  bucket = aws_s3_bucket.test_bucket.id
+  policy = data.aws_iam_policy_document.s3_bucket_policy.json
+}
+
+data "aws_iam_policy_document" "s3_bucket_policy" {
+  statement {
+    effect = "Allow"
+    principals {
+      # TF-UPGRADE-TODO: In Terraform v0.10 and earlier, it was sometimes necessary to
+      # force an interpolation expression to be interpreted as a list by wrapping it
+      # in an extra set of list brackets. That form was supported for compatibility in
+      # v0.11, but is no longer supported in Terraform v0.12.
+      #
+      # If the expression in the following list itself returns a list, remove the
+      # brackets to avoid interpretation as a list of lists. If the expression
+      # returns a single list item then leave it as-is and remove this TODO comment.
+      identifiers = [local.aws_account_id]
+      type        = "AWS"
+    }
+    actions   = ["*"]
+    resources = ["${aws_s3_bucket.test_bucket.arn}/*"]
+  }
+
+  statement {
+    effect = "Deny"
+    principals {
+      identifiers = ["*"]
+      type        = "AWS"
+    }
+    actions   = ["*"]
+    resources = ["${aws_s3_bucket.test_bucket.arn}/*"]
+
+    condition {
+      test     = "Bool"
+      variable = "aws:SecureTransport"
+      values = [
+        "false",
+      ]
+    }
+  }
+}
+
+# ---------------------------------------------------------------------------------------------------------------------
+# LOCALS
+# Used to represent any data that requires complex expressions/interpolations
+# ---------------------------------------------------------------------------------------------------------------------
+
+data "aws_caller_identity" "current" {
+}
+
+locals {
+  aws_account_id = data.aws_caller_identity.current.account_id
+}
+

examples/terraform-aws-endpoint-example/outputs.tf

@@ -0,0 +1,15 @@
+output "bucket_id" {
+  value = aws_s3_bucket.test_bucket.id
+}
+
+output "bucket_arn" {
+  value = aws_s3_bucket.test_bucket.arn
+}
+
+output "logging_target_bucket" {
+  value = aws_s3_bucket_logging.test_bucket.target_bucket
+}
+
+output "logging_target_prefix" {
+  value = aws_s3_bucket_logging.test_bucket.target_prefix
+}

examples/terraform-aws-endpoint-example/variables.tf

@@ -0,0 +1,40 @@
+# ---------------------------------------------------------------------------------------------------------------------
+# ENVIRONMENT VARIABLES
+# Define these secrets as environment variables
+# ---------------------------------------------------------------------------------------------------------------------
+
+# AWS_ACCESS_KEY_ID
+# AWS_SECRET_ACCESS_KEY
+
+# ---------------------------------------------------------------------------------------------------------------------
+# REQUIRED PARAMETERS
+# You must provide a value for each of these parameters.
+# ---------------------------------------------------------------------------------------------------------------------
+variable "region" {
+  description = "The AWS region to deploy to"
+  type        = string
+}
+
+# ---------------------------------------------------------------------------------------------------------------------
+# OPTIONAL PARAMETERS
+# These parameters have reasonable defaults.
+# ---------------------------------------------------------------------------------------------------------------------
+
+variable "with_policy" {
+  description = "If set to `true`, the bucket will be created with a bucket policy."
+  type        = bool
+  default     = false
+}
+
+variable "tag_bucket_name" {
+  description = "The Name tag to set for the S3 Bucket."
+  type        = string
+  default     = "Test Bucket"
+}
+
+variable "tag_bucket_environment" {
+  description = "The Environment tag to set for the S3 Bucket."
+  type        = string
+  default     = "Test"
+}
+