Issues with infrastructure deployer, logs aren't much help

[umm0n]

Hey folks, similar issues to this discussion: https://github.com/orgs/gruntwork-io/discussions/767

We’re using the gruntworks ref arch with associated pipeline, in github actions. The pipe has begun to crash during some plans, mostly those that involves altering several _envcommon files. Altering each _envcommon file individually, in discreet PRs, works.

These plans are running in fargate tasks with 8vCPU and 16GB of memory, so it’s very unlikely that they’re resource constrained.

Terraform version: v1.2.0
Terragrunt ci version: v0.52.13
Terragrunt version: v0.57.5

The errors we're getting in the logs aren't very helpful and I'm honestly running out of ideas. I've noticed in some other threads we might be able to set a debug flag somewhere to get some additional insight, but it isn't super clear how to do it.

Any help would be greately appreciated!

Some of the logs:

ecs-deploy-runner][2024-04-23T11:06:52+0000] time=2024-04-23T11:06:52Z level=info msg=The stack at /tmp/tmpo20zc9ay/prod will be processed in the following order for command plan:
[ecs-deploy-runner][2024-04-23T11:06:52+0000] Group 1
[ecs-deploy-runner][2024-04-23T11:06:52+0000] - Module /tmp/tmpo20zc9ay/prod/sa-east-1/prod/services/backend-A
[ecs-deploy-runner][2024-04-23T11:06:52+0000] - Module /tmp/tmpo20zc9ay/prod/sa-east-1/prod/services/backend-B
[ecs-deploy-runner][2024-04-23T11:06:52+0000] - Module /tmp/tmpo20zc9ay/prod/sa-east-1/prod/services/backend-C
[ecs-deploy-runner][2024-04-23T11:06:52+0000] - Module /tmp/tmpo20zc9ay/prod/sa-east-1/prod/services/backend-D
[ecs-deploy-runner][2024-04-23T11:06:52+0000]  prefix=[/tmp/tmpo20zc9ay/prod] 
[ecs-deploy-runner][2024-04-23T11:06:54+0000] time=2024-04-23T11:06:54Z level=error msg=Module /tmp/tmpo20zc9ay/prod/sa-east-1/prod/services/backend-C has finished with an error: [/tmp/tmpo20zc9ay/prod/sa-east-1/prod/services/backend-C] exit status 126 prefix=[/tmp/tmpo20zc9ay/prod/sa-east-1/prod/services/backend-C] 

[....]

Traceback (most recent call last):
[ecs-deploy-runner][2024-04-23T10:26:22+0000]   File "/opt/ecs-deploy-runner/scripts/infrastructure-deploy-script", line 242, in <module>
[ecs-deploy-runner][2024-04-23T10:26:22+0000]     main(max_content_width=120, auto_envvar_prefix=ENVVAR_PREFIX)
[ecs-deploy-runner][2024-04-23T10:26:22+0000]   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1134, in __call__
[ecs-deploy-runner][2024-04-23T10:26:22+0000]     return self.main(*args, **kwargs)
[ecs-deploy-runner][2024-04-23T10:26:22+0000]   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1059, in main
[ecs-deploy-runner][2024-04-23T10:26:22+0000]     rv = self.invoke(ctx)
[ecs-deploy-runner][2024-04-23T10:26:22+0000]   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1401, in invoke
[ecs-deploy-runner][2024-04-23T10:26:22+0000]     return ctx.invoke(self.callback, **ctx.params)
[ecs-deploy-runner][2024-04-23T10:26:22+0000]   File "/usr/local/lib/python3.8/site-packages/click/core.py", line 767, in invoke
[ecs-deploy-runner][2024-04-23T10:26:22+0000]     return __callback(*args, **kwargs)
[ecs-deploy-runner][2024-04-23T10:26:22+0000]   File "/opt/ecs-deploy-runner/scripts/infrastructure-deploy-script", line 226, in main
[ecs-deploy-runner][2024-04-23T10:26:22+0000]     raise exc
[ecs-deploy-runner][2024-04-23T10:26:22+0000]   File "/opt/ecs-deploy-runner/scripts/infrastructure-deploy-script", line 207, in main
[ecs-deploy-runner][2024-04-23T10:26:22+0000]     deploy.deploy(
[ecs-deploy-runner][2024-04-23T10:26:22+0000]   File "/usr/local/lib/python3.8/site-packages/infrastructure_deploy_script/deploy.py", line 156, in deploy
[ecs-deploy-runner][2024-04-23T10:26:22+0000]     __run_deployment(options, deploy_dir)
[ecs-deploy-runner][2024-04-23T10:26:22+0000]   File "/usr/local/lib/python3.8/site-packages/infrastructure_deploy_script/deploy.py", line 205, in __run_deployment
raise DeployError(
[ecs-deploy-runner][2024-04-23T10:26:22+0000] infrastructure_deploy_script.exceptions.DeployError: Could not run "terragrunt plan-all". Error=time=2024-04-23T10:25:22Z level=warning msg='plan-all' is deprecated. Running 'terragrunt run-all plan -input=false' instead. Please update your workflows to use 'terragrunt run-all plan -input=false', as 'plan-all' may be removed in the future!
exit status 126
[ecs-deploy-runner][2024-04-23T10:26:22+0000] exit status 1
[ecs-deploy-runner][2024-04-23T10:26:22+0000] time="2024-04-23T10:26:22Z" level=error msg="exit status 1" binary=deploy-runner-entrypoint version=latest
ERROR: Deployment failed
xargs: bash: exited with status 255; aborting

---

Tracked in ticket #110893

[umm0n]

We've raised this issue previously, none of the suggestions in this thread have helped unfortunately.

[josh-padnick]

@denis256 Any thoughts on what might be happening here?

@umm0n Thanks for reporting this. The longer-term solution here is to get you upgraded to our latest version of Pipelines, which we're currently doing beta testing for. If this is something you'd be interested in testing with us in the next few weeks, let me know!

[umm0n]

Hello @josh-padnick . Yes, I've actually been trying to deploy the latest version of pipelines, by following this tutorial. I've faced a lot of errors in the process, as it seems the tutorial is pretty outdated, but I'm slowly pushing through. Stuck at the dispatch step for now.

[umm0n]

Well, after a lot of backs and forths I was able to get it running, albeit by setting the presign_token flag to false in the dispatch GHA. It's expecting a role that we don't have in our ref arch, but I'll figure out a way to deploy it later.

Feel free to close this issue! But I wouldn't necessarily mark it as solved.

If you guys want feedback into the ECS deploy runner migration docs, I'd be happy to share some.

[denis256]

Hi, feedback will be helpful to improve documentation and the entire deployment workflow 👀 ✍️

[josh-padnick]

@umm0n Thanks for your feedback on the tutorial, and I acknowledge your feedback that that the docs/UX was not great here. Fortunately, we've already been working on this!

• We’re a few weeks away from releasing a much-improved version of both Pipelines and docs.
• I've just added you to a list of users to notify once the newest version is available.
• I've also sent you an email with more details.

Please let us know if we can be of further help here in the meantime.

[umm0n]

Hi, feedback will be helpful to improve documentation and the entire deployment workflow 👀 ✍️

@umm0n Thanks for your feedback on the tutorial, and I acknowledge your feedback that that the docs/UX was not great here. Fortunately, we've already been working on this!

• We’re a few weeks away from releasing a much-improved version of both Pipelines and docs.
• I've just added you to a list of users to notify once the newest version is available.
• I've also sent you an email with more details.

Please let us know if we can be of further help here in the meantime.

Great, thanks Josh. Looking forward to trying out the new pipelines.

I'm still having issues with envcommon changes with pipelinesv2, but it's different error now, so I think it deserves it's own thread.

Any place where I could share feedback for pipelines? Off the top of my mind:

• Are envcommon changes supposed to trigger plans across all accounts, even those with no calls to that envcommon module? ECS deploy runner only triggered plans in the affected accounts, whereas pipelines v2 seems to run plan in all of them.
• For some modules, we use template .json files alongside the .hcl files to deploy infra, in the same directory. Pipelinesv2 doesn't seem to detect changes to these .json files, so no plan is run on these modules. ECS deploy runner did, however.

[ZachGoldberg]

Hi!

I'm still having issues with envcommon changes with pipelinesv2, but it's different error now, so I think it deserves it's own thread.

Any place where I could share feedback for pipelines? Off the top of my mind:

If you log in to app.gruntwork.io there is a Send Feedback button in the top right that would get your feedback to the right place!

• Are envcommon changes supposed to trigger plans across all accounts, even those with no calls to that envcommon module? ECS deploy runner only triggered plans in the affected accounts, whereas pipelines v2 seems to run plan in all of them.

Although not desirable in many situations, this is the expected behavior as of today for pipelines v2.

• For some modules, we use template .json files alongside the .hcl files to deploy infra, in the same directory. Pipelinesv2 doesn't seem to detect changes to these .json files, so no plan is run on these modules. ECS deploy runner did, however.

This is almost certainly a bug with pipelines, I'll file it internally to make sure this gets addressed.