AWS session-based SSO does not work with s3 backend

[UVduane]

What happened?

Attempting to pulumi login to an S3 backend while using AWS session-based SSO and get an error:

export AWS_PROFILE=childaccount
pulumi login s3://mah-bukkit
error: problem logging in: unable to open bucket s3://mah-bukkit: open bucket s3://mah-bukkit: couldn't create session profile "my-profile" is configured to use SSO but is missing required configuration: sso_region, sso_start_url

In $HOME/.aws/config we have:

[sso-session sso-session]                                                                                                                                                               
sso_start_url = https://d-1a2b3c4d5e6f.awsapps.com/start#                                                                                                                        
sso_region = us-west-2                                                                                                                                                         
sso_registration_scopes = sso:account:access     

[profile childaccount]
sso_session = sso-session
sso_account_id = 222222222222
sso_role_name = AdministratorAccess
region = us-west-2

This AWS config works with AWS CLI v2, like so:

$ AWS_PROFILE=childaccount aws sts get-caller-identity
{
    "UserId": "AR<key>:<email>",
    "Account": "222222222222",
    "Arn": "arn:aws:sts::222222222222:assumed-role/AWSReservedSSO_AdministratorAccess_<id>/<email>"
}

On our own testing, this is related to aws/aws-sdk-go#4649. Looking at the gocloud.dev blob library, it uses AWS SDK v1 by default, making SDKv2 available as an option. But, the version of SDKv2 required in gocloud.dev v0.27 does not work with session-based SSO either. I think a version bump in pulumi to gocloud.dev v0.29 would be a good start to a decent workaround on this.

https://github.com/UVduane/musical-octo-telegram/tree/main might be helpful as a minimized test case

Expected Behavior

pulumi login should log in without error

Steps to reproduce

• Configure IAM Identity Center (AWS SSO) in a multi-account organization
• Set up a pulumi project to use S3 state
• Attempt a pulumi login

Output of pulumi about

Version      3.70.0
Go Version   go1.20.4
Go Compiler  gc

Plugins
NAME            VERSION
aws             5.40.0
aws-apigateway  1.0.1
aws-native      0.64.0
command         0.7.2
python          unknown

Host
OS       fedora
Version  37
Arch     x86_64

This project is written in python: executable='/home/dwaddle/src/xxx/xxx/venv/bin/python3' version='3.11.3
'

Dependencies:
NAME                   VERSION
black                  23.3.0
mypy                   1.3.0
pip                    22.2.2
powerline-status       2.7.0
pulumi-aws-apigateway  1.0.1
pulumi-aws-native      0.64.0
pulumi-command         0.7.2
pylint                 2.17.4
pyright                1.1.313

Pulumi locates its logs in /tmp by default
warning: Could not access the backend: unable to open bucket s3://mahbukkit: open bucket s3://mahbukkit: couldn't create session profile "mahbukkit" is configured to use SSO but is missing required configuration: sso_region, sso_start_url

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[Frassle]

Just a note for when looking into this. The s3 backend and the awskms secret manager both use go.cloud, and we do have some tests for awskms at https://github.com/pulumi/pulumi/blob/master/pkg/secrets/cloud/manager_test.go.

We should see about getting some similar tests setup for the filestate backend against real aws buckets and see if we can verify this use case.

[mattjones753]

I believe this issue is caused by the fact that the AWS Go SDK didn't support SSO session auth (aws/aws-sdk-go#4649) this has now been fixed and released in version v1.44.298 https://github.com/aws/aws-sdk-go/releases/tag/v1.44.298

[ekini]

For more context around v2 of the AWS SDK it seems like support for sso-session was introduced in aws/aws-sdk-go-v2#1903

And the earliest it exist in is credentials/v1.12.15

We can see go-cloud 0.27 references v1.12.10 https://github.com/google/go-cloud/blob/v0.27.0/go.sum#L239
And it was updated in 0.28 https://github.com/google/go-cloud/blob/v0.28.0/go.sum#L546

So I guess 0.28 would work as well :)

[ekini]

Why not update gocloud.dev to the latest then? @UVduane

[TheFynx]

I have a PR open that resolves this #13619 but can't get anyone to acknowledge here or in Slack.

[ekini]

That PR doesn't update go-cloud though?

I would also add more information what this is about, because when I first read the description for the first time, it wasn't really clear what it would fix, as I use SSO and it does work for me.

So for information sake, here's the relevant doc about SSO https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-sso.html

Before one would have something like this

[profile my-dev-profile]
sso_start_url = https://my-sso-portal.awsapps.com/start
sso_region = us-east-1
sso_account_id = 123456789011
sso_role_name = readOnly

as per https://docs.aws.amazon.com/cli/latest/userguide/sso-configure-profile-legacy.html
But it seems like the new (and apparently better) way is to have an sso-session configured separately like

[profile my-dev-profile]
sso_session = my-sso
sso_account_id = 123456789011
sso_role_name = readOnly
region = us-west-2
output = json

[sso-session my-sso]
sso_region = us-east-1
sso_start_url = https://my-sso-portal.awsapps.com/start
sso_registration_scopes = sso:account:access

as per https://docs.aws.amazon.com/cli/latest/userguide/sso-configure-profile-token.html

Hence the error with older AWS SDKs

couldn't create session profile "my-profile" is configured to use SSO but is missing required configuration: sso_region, sso_start_url

[TheFynx]

I have the sso-session profiles set up like what you posted. Actually, we have a single session and about 14 different profiles referencing it.

we were actively running into the

couldn't create session profile "my-profile" is configured to use SSO but is missing required configuration: sso_region, sso_start_url

with profiles like

[profile AwsProfile]
sso_session = my-sso
sso_account_id = 123456789011
sso_role_name = readOnly
region = us-west-2
output = json

[sso-session my-sso]
sso_region = us-east-1
sso_start_url = https://my-sso-portal.awsapps.com/start
sso_registration_scopes = sso:account:access

and this works after the aws-sdk update (must change awssdk to v3)

❯ ~/.pulumi-dev/bin/pulumi login 's3://state-bucket?region=us-east-1&awssdk=v3&profile=AwsProfile'
Logged in to levi-framework as levi (s3://state-bucket?region=us-east-1&awssdk=v3&profile=AwsProfile)

I am actively using the binary built from my branch with just updating the aws-sdk in our setup with no issues using sso-session based profiles.