Add support for a custom aws endpoint #1264

sverch · 2023-03-18T01:22:15Z

Description

Fixes #494.

This makes it possible to run terratest against a custom aws endpoint. This allows it to be used woth Moto's standalone server mode for example, to test AWS modules locally without needing an AWS account or any access to AWS.

Unfortunately these tests don't pass as is, because they would require setting up the moto server, and I'm not sure where that setup should be added. They do pass if the moto server is running.

TODOs

Read the Gruntwork contribution guidelines.

Update the docs (I don't see a relevant page for this, but I'm willing to do this).
Run the relevant tests successfully, including pre-commit checks (this passes, but requires a moto_server to be running).
Ensure any 3rd party code adheres with our license policy or delete this line if its not applicable. (moto is apache 2: https://github.com/getmoto/moto/blob/master/LICENSE)
Include release notes. If this PR is backward incompatible, include a migration guide.

Release Notes (draft)

Added support for aws custom endpoints with TERRATEST_CUSTOM_AWS_ENDPOINT environment variable.

Migration Guide

No migration needed. No changes if TERRATEST_CUSTOM_AWS_ENDPOINT is unset.

denis256 · 2023-03-19T13:58:35Z

test/terraform_aws_endpoint_example_test.go

+
+	// Set a custom endpoint for AWS, and set the keys to dummy keys to
+	// pass that check
+	os.Setenv("TERRATEST_CUSTOM_AWS_ENDPOINT", "http://localhost:5000")


I think should be better to use t.Setenv() since it has automatic cleanup of env variables

https://pkg.go.dev/testing#B.Setenv

I get this error:

panic: testing: t.Setenv called after t.Parallel; cannot set environment variables in parallel tests

If I make it a non parallel test it passes.

Oh, in the end I did change it to t.Setenv. I just wanted to let you know that had to change it to a non parallel test for that to work.

denis256 · 2023-03-24T19:49:24Z

Noticed that some of files aren't formatted:

goimports................................................................Failed
- hook id: goimports
- files were modified by this hook

modules/aws/auth.go
modules/aws/auth.go

Terraform fmt............................................................Failed
- hook id: terraform-fmt
- files were modified by this hook

main.tf

sverch · 2023-03-24T21:11:29Z

Ah, sorry about that @denis256. I hadn't installed the pre-commit hooks in this repo. It should be fixed now.

sverch · 2023-04-23T02:35:51Z

@denis256 It seems like the previous CI test failed because of a golang installation. I don't think I changed anything that would affect that.

I added a new CI task that installs moto server and runs the local endpoint test case against it. I used this to test the installation of the dependencies and verify that the local server works:

circleci local execute mock_aws_test

However, it looks like there are some private repos this depends on, so I can't test the whole thing as it would be in CI.

denis256 · 2023-04-23T21:16:47Z

Not sure if are required any changes in .circleci/config.yml, just rebase from the master branch where circle ci issue was fixed

Fixes gruntwork-io#494. This makes it possible to run terratest against a custom aws endpoint. This allows it to be used woth [Moto's standalone server mode](http://docs.getmoto.org/en/latest/docs/server_mode.html) for example, to test AWS modules locally without needing an AWS account or any access to AWS. Unfortunately these tests don't pass as is, because they would require setting up the moto server, and I'm not sure where that setup should be added. They do pass if the moto server is running.

It needs to be a separate group because moto server needs to be installed and running

sverch · 2023-04-23T23:04:03Z

Ok, done!

sverch · 2023-04-23T23:05:45Z

I'm happy to update the docs too, although I don't know where I would add something like this.

Specifically, I'd put something like this:

Terratest supports using a custom AWS endpoint by setting TERRATEST_CUSTOM_AWS_ENDPOINT to the URL you want to use for AWS. This is the same as setting --endpoint on the aws CLI.

denis256 · 2023-04-24T20:16:03Z

Documentation is located in the same repository, into docs directory

sverch · 2023-05-12T22:06:55Z

@denis256 Added the docs and attempted to fix the circleci build again. Thanks for your patience!

examples/terraform-aws-endpoint-example/README.md

Co-authored-by: Etiene Dalcol <Etiene@users.noreply.github.com>

sverch · 2023-05-23T20:12:40Z

I see an error in CI:

2023-05-23 15:47:14 [ERROR] [gruntwork-install] Required environment variable GITHUB_OAUTH_TOKEN not set.

Do you have any advice on how to fix that?

The other failures look unrelated to this change, so I'll merge in latest master.

.circleci/config.yml

Co-authored-by: Etiene Dalcol <Etiene@users.noreply.github.com>

…m-endpoint-494

sverch · 2023-06-03T14:07:59Z

Thanks @Etiene. I added your suggested change and merged in the latest master.

Etiene · 2023-06-12T11:55:03Z

Thank you @sverch. I think this looks good now! I've kicked off the tests again :)

sverch · 2023-06-13T02:06:43Z

@Etiene Thanks for running that again!

Fixed these issues:

localhost caused a DNS lookup that failed, changed to 127.0.0.1.
run-go-tests needs the same argument as go test to select the right test and not run everything (I thought the tags option was filtering tests).
Set AWS_PAGER="", because the aws cli complains about not having a complete terminal.

I can't say for sure it'll work now, but hopefully that's one step closer! Although, the test job also failed, so I might need to merge master again when a fix is in.

Etiene · 2023-06-13T09:06:05Z

No worries! Running them again

.circleci/config.yml

+      - run:
+          command: |
+            mkdir -p /tmp/logs
+            run-go-tests --packages "-tags=mockaws -run TestTerraformAwsEndpointExample ./test/" | tee /tmp/logs/test_output.log


sverch · 2023-06-16T01:11:00Z

@Etiene Thanks for rerunning. I also needed to remove "localhost" from the terraform backend configuration and use 127.0.0.1 instead.

Etiene · 2023-06-28T12:38:48Z

The tests are still failing though!

sverch · 2023-06-29T00:23:33Z

Error: creating Amazon S3 (Simple Storage) Bucket (123456789012-terratest-aws-endpoint-example-gyu2pw-logs): RequestError: send request failed
│ caused by: Put "http://123456789012-terratest-aws-endpoint-example-gyu2pw-logs.127.0.0.1:5000/": dial tcp: lookup 123456789012-terratest-aws-endpoint-example-gyu2pw-logs.127.0.0.1 on 127.0.0.11:53: no such host

Ah, I see. So using the IP to try to get around the DNS issue doesn't really work because S3 is using a subdomain.

I'll have to see if there's a circleci way to reference this local AWS server with a DNS name. But now I'm wondering why localhost didn't work before.

I'm looking at https://circleci.com/docs/postgres-config/ for an example, and it says that services are available with localhost, and then when I look at https://circleci.com/docs/configuration-reference/, I see:

name defines the the hostname for the container (the default is localhost), which is used for reaching secondary (service) containers. By default, all services are exposed directly on localhost. This field is useful if you would rather have a different hostname instead of localhost, for example, if you are starting multiple versions of the same service.

So I wonder if the reason it didn't work before is because localhost is meant to be used if you create a "service" container. The problem with that though is that I don't think I can run commands in the service container to run the mock aws server.

At this point, I'm just speculating. The tests pass for me locally, so I think this does come down to figuring out how to properly connect to localhost in circleci. Any chance you've done this for any other tests?

Etiene · 2023-06-29T14:43:46Z

I have not done this for other tests before, sorry :( I could try investigating this in more details, but Im not sure if I'll have the time to get to it.

This reverts commit 62c801d.

This reverts commit 76bc05e.

sverch · 2023-06-30T00:30:08Z

I understand that @Etiene, and thank you for your patience with all the back and forth.

I finally just created a custom circleci config on my own branch without the private gruntwork dependencies to get to the bottom of this, and was able to reproduce the issue.

It turns out that localhost resolves fine, but <subdomain>.localhost does not. I suspect that's because from circleci's perspective, that's the "service name" with localhost being the default.

Circleci doesn't expose any way to configure this as far as I can tell, so the only option from this end would be to run some custom DNS server. That's starting to get kind of heavyweight, and I wouldn't want to do anything that might be used by other tests or cause hard to debug issues later.

So I tried to get the calls to s3 to use the "path style" API, and it worked! See: https://app.circleci.com/pipelines/github/sverch/terratest/12/workflows/5a161e91-bd96-4951-b7b2-b31d50f2919d/jobs/12

The caveats are that I had to add support for that in terratest, so the scope of this change is a bit bigger, and AWS wants to deprecate that someday (I put that in the comments). Enough people seem to be pushing back on it that it's unclear when that will actually happen.

sverch requested review from zackproser and denis256 as code owners March 18, 2023 01:22

denis256 reviewed Mar 19, 2023

View reviewed changes

Shaun Verch added 4 commits April 23, 2023 19:02

Use t.Setenv so env vars are cleaned up

5c2e8e4

Fix formatter errors

67df0cd

Add CI task to run tests

6905e03

It needs to be a separate group because moto server needs to be installed and running

sverch force-pushed the feature/aws-custom-endpoint-494 branch from d18e6dd to 6905e03 Compare April 23, 2023 23:02

Shaun Verch added 3 commits May 12, 2023 23:57

Add documentation for custom aws endpoints

0606e8b

Fix section header

5da3b5a

Attempt to fix circleci configuration

55fc462

Etiene reviewed May 23, 2023

View reviewed changes

examples/terraform-aws-endpoint-example/README.md Outdated Show resolved Hide resolved

Update examples/terraform-aws-endpoint-example/README.md

4aa41b0

Co-authored-by: Etiene Dalcol <Etiene@users.noreply.github.com>

Merge branch 'master' into feature/aws-custom-endpoint-494

0636877

Etiene reviewed May 25, 2023

View reviewed changes

.circleci/config.yml Outdated Show resolved Hide resolved

sverch and others added 2 commits June 3, 2023 10:03

Update .circleci/config.yml

ab57be4

Co-authored-by: Etiene Dalcol <Etiene@users.noreply.github.com>

Merge remote-tracking branch 'upstream/master' into feature/aws-custo…

471e5c3

…m-endpoint-494

Shaun Verch added 2 commits June 12, 2023 21:56

Make log directory for mock aws server logs

212fbb4

Disable AWS_PAGER because of bad terminal warning

dcf6e5c

Shaun Verch added 2 commits June 12, 2023 21:56

Make sure to only run mock aws test

7aad127

Use local IP in attempt to not cause dns lookup

76bc05e

Etiene reviewed Jun 13, 2023

View reviewed changes

Use 127.0.0.1 in local endpoint example

62c801d

Shaun Verch added 4 commits June 29, 2023 18:06

Revert "Use 127.0.0.1 in local endpoint example"

cb1b7c5

This reverts commit 62c801d.

Revert "Use local IP in attempt to not cause dns lookup"

7bf168c

This reverts commit 76bc05e.

Use s3 path style endpoints

4303600

Merge branch 'master' into feature/aws-custom-endpoint-494

7faf973

jmoghisi mentioned this pull request Jul 9, 2023

Assert failing if you are using local configuration for testing (i.e: localstack locally) #494

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for a custom aws endpoint #1264

Add support for a custom aws endpoint #1264

sverch commented Mar 18, 2023 •

edited

denis256 Mar 19, 2023 •

edited

sverch Mar 21, 2023

sverch Apr 8, 2023

denis256 commented Mar 24, 2023

sverch commented Mar 24, 2023

sverch commented Apr 23, 2023

denis256 commented Apr 23, 2023

sverch commented Apr 23, 2023

sverch commented Apr 23, 2023

denis256 commented Apr 24, 2023

sverch commented May 12, 2023

sverch commented May 23, 2023

sverch commented Jun 3, 2023

Etiene commented Jun 12, 2023

sverch commented Jun 13, 2023

Etiene commented Jun 13, 2023

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

sverch commented Jun 16, 2023

Etiene commented Jun 28, 2023

sverch commented Jun 29, 2023

Etiene commented Jun 29, 2023

sverch commented Jun 30, 2023

Add support for a custom aws endpoint #1264

Are you sure you want to change the base?

Add support for a custom aws endpoint #1264

Conversation

sverch commented Mar 18, 2023 • edited

Description

TODOs

Release Notes (draft)

Migration Guide

denis256 Mar 19, 2023 • edited

Choose a reason for hiding this comment

sverch Mar 21, 2023

Choose a reason for hiding this comment

sverch Apr 8, 2023

Choose a reason for hiding this comment

denis256 commented Mar 24, 2023

sverch commented Mar 24, 2023

sverch commented Apr 23, 2023

denis256 commented Apr 23, 2023

sverch commented Apr 23, 2023

sverch commented Apr 23, 2023

denis256 commented Apr 24, 2023

sverch commented May 12, 2023

sverch commented May 23, 2023

sverch commented Jun 3, 2023

Etiene commented Jun 12, 2023

sverch commented Jun 13, 2023

Etiene commented Jun 13, 2023

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

sverch commented Jun 16, 2023

Etiene commented Jun 28, 2023

sverch commented Jun 29, 2023

Etiene commented Jun 29, 2023

sverch commented Jun 30, 2023

sverch commented Mar 18, 2023 •

edited

denis256 Mar 19, 2023 •

edited