Address s3 compatible remote state issues + logging

[cam72cam]

This PR started out addressing #821, but ended up being an investigation of the workflow itself instead of the not-adopted validation changes.

• Add in the relevant context hooks for aws-go-sdk-base logging. This is required to easily debug the rest of the changes.
• Missing proto:// backwards compatibility
  • BaseEndpoint -> EndpointResolver fixed many of the issues related to non-proto endpoints.
  • Unfortunately it did not cover all cases, the main one I identified was in the aws-go-sdk-base middleware.
  • A workaround was added such that any non-proto endpoints would automatically have the https:// prefix added.
  • This is hopefully a sane default, and comes with a debug message that should help track down if it ever causes issues.
• PutObject checksum calculation
  • skip_s3_checksum was added in https://github.com/opentofu/opentofu/pull/795/files to partially work around s3 compatible endpoints not supporting the default sha256 PARTIAL headers.
  • This can be turned off, but should be avoided if possible.
  • Since we already have the payload as a []byte in memory, it's simple to compute the hash ahead of time and bypass the streaming PARTIAL header issue.
  • In testing with DigitalOcean, providing the full sha256 instead of messing with PARTIAL fixed the issue. Issues here and on related repositories suggest that this should fix the issue for multiple providers.
  • More reading: Developer Guide: Document Support for Amazon S3 Object Integrity Features aws/aws-sdk-go-v2#1689
• Added a HeadObject check to state.get to work around a MinIO issue where the GetObject request would return the wrong error message and fail. I'm not sure that this is worth keeping in, though it might solve issues with other providers?

TODO:

• Add tests to detecting / adding proto://
• Decide if removing the new HeadObject call is the right call.

Resolves #821

Target Release

1.6.0

Draft CHANGELOG entry

BUG FIXES

• S3 backend endpoints without proto:// now default to https:// instead of failing.
• Most S3 compatible remote state backends should now work without checksum errors / 400s by default.
• Logging has been re-added to S3 remote state calls.

[github-actions]

Reminder for the PR assignee: If this is a user-visible change, please update the changelog as part of the PR.

[Yantrio]

I've converted this to draft whilst we still have items here in TODO

[Yantrio]

LGTM so far, just a couple nitpicks 👍

[Yantrio]

So far so good, I'll tentitively approve for now, please ping me again when the final TODO items are done and it's ready for review again. Nice work!

[kislerdm]

(question) Shall we also cover custom endpoint for STS and IAM as per the issue description:

The following behaviour happens for S3, IAM, DynamoDB and STS endpoints as well

For context, see the details:

• IAM endpoint configuration
• STS endpoint configuration

cc: @RLRabinowitz @Yantrio @cam72cam

[cam72cam]

(question) Shall we also cover custom endpoint for STS and IAM as per the issue description:

The following behaviour happens for S3, IAM, DynamoDB and STS endpoints as well

For context, see the details:

* [IAM endpoint configuration](https://github.com/opentofu/opentofu/blob/aa6b6a2fec9924dbc70e15e04aa1ccdd514e7e16/internal/backend/remote-state/s3/backend.go#L706)

* [STS endpoint configuration](https://github.com/opentofu/opentofu/blob/aa6b6a2fec9924dbc70e15e04aa1ccdd514e7e16/internal/backend/remote-state/s3/backend.go#L713)

cc: @RLRabinowitz @Yantrio @cam72cam

The https:// correction happens for all of the endpoints in the StringOk function.