chore: replace TF Cloud implementation with TF CLI

[DanielMSchmidt]

This will allow us to integrate better with TFC and TFE without additional work being spend on polishing this part.

TODO

• Check if this solves Auto Migrate State into Terraform Cloud  #490
• Check if this solves Handle Switching Backend from Remote to Something Else #489

[ansgarm]

Okay, so #490 does not quite work yet with this PR and requires a follow-up:

[michaellzc]

Hey, this seems like a better approach otherwise cdktf will always have to play catch-up with TFC/TFE, e.g. #2243.

but... when I tried out this branch, and got this error when running cdktf destroy

[2022-11-25T19:30:29.198] [ERROR] default - ╷
│ Error: Saving a generated plan is currently not supported
│
│ Terraform Cloud does not support saving the generated execution plan
output  ╷
        │ Error: Saving a generated plan is currently not supported
        │
        │ Terraform Cloud does not support saving the generated execution plan
        │ locally at this time.

Apparently, TFC/TFE does not support saving the plan locally through terraform cli when remote execution is enabled, and the raw plan is only exposed through their API. (TIL we just migrated to TFC not that long ago)

Therefore, it seems almost impossible to get to the state where cdktf has zero knowledge of how TFC/TFE works, and always requires some special handling.

Is such a huge discrepancy between normal terraform behaviour and TFC/TFE something on the terraform and cdktf team radar? What is the plan for cdktf to address it?

[ansgarm]

Hi @michaellzc 👋

Wow, you are quite the adventurer by daring to test this draft PR 😁👏

Yeah, I noticed that as well when I took over Daniel's work on this. In the long term, we're working with the team working on the Terraform CLI on a solution that allows us to close this gap (I don't know yet whether they'll support saving a TFC/E remote plan in a file that can be applied or whether a different solution will be preferred).

However, as we need to support current Terraform CLI versions as well, our current approach is to use the Terraform CLI in an interactive way (via node-pty). So I'm currently working on switching the implementation from cdktf deploy = terraform plan -out <plan> && terraform apply <plan> to cdktf deploy = terraform apply (and relaying the "Do you want to apply? yes / no"-question). This will work for all cases but still has uncertainties that we need to test.

[team-tf-cdk]

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

[team-tf-cdk]

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

[team-tf-cdk]

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

[team-tf-cdk]

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days. This helps our maintainers find and focus on the active issues. If you've found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.