Replies: 16 comments 1 reply
-
|
@hjkatz -- This is ultimately a fault in server-side behavior. The IAM state change associated with retrieving credentials (via AssumeRole or whatever else) does not propagate immediately after credentials are returned. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @hjkatz , Just to chime in as well. This is not an SDK issue and is not unique to SSO. What you are experiencing is a propagation delay that is solved with a retry. This is not an issue with the SDK itself but just a nature of a distributed system. Thanks, |
Beta Was this translation helpful? Give feedback.
-
|
This issue is now closed. Comments on closed issues are hard for our team to see. |
Beta Was this translation helpful? Give feedback.
-
@RanVaknin Kindly review the linked discussion. Retrying 5 times with a gradual backoff from 500ms to 2s still reproduces the issue. (I can provide a mvp example if desired too.) I understand that a distributed system will require time to propogate the new token. So I'll start with my goal instead: I want to verify that the SSO session via the SDK is valid. How can I test that reliably? |
Beta Was this translation helpful? Give feedback.
-
|
@RanVaknin I realize that the discussion doesn't have as clear of an example as I'm suggesting. I'll upload a clear example later today. |
Beta Was this translation helpful? Give feedback.
-
// Wrapper for sts.GetCallerIdentity() that supports retries
//
// Use the context to set a maximum time that retries can be performed.
// When the context is canceled then the final response will be returned.
//
// See: https://github.com/aws/aws-sdk-go-v2/discussions/2093#discussioncomment-8455830
func StsGetCallerIdentity(ctx context.Context, client *sts.Client) (result *sts.GetCallerIdentityOutput, err error) {
if client == nil {
return nil, errs.New("cannot call StsGetCallerIdentity with nil client!")
}
attempt := 0
retries := 5
// internal lib that implements a backoff between start -> end, without jitter (false)
backoff := reliable.NewBackoff(100*time.Millisecond, 1*time.Second, false)
for attempt < retries {
attempt++
select {
case <-ctx.Done():
// ran out of time, return whatever we got last
return
default:
// continue below
}
result, err = client.GetCallerIdentity(ctx, &sts.GetCallerIdentityInput{})
if err == nil {
// success
return // final values
}
// See: https://github.com/aws/aws-sdk-go-v2/discussions/2093
if !strings.Contains(err.Error(), "api error InvalidClientTokenId") {
// non-retryable error
return // last result + error
}
// error, backoff and try again
be := backoff.Wait(ctx) // Wait() == time.Sleep(backoff.NextDuration())
if be != nil {
// context canceled
return // last values
}
}
return // last values
}I hope this helps reproduce what we're seeing. I'm also hopeful we can find a way via the SDK to verify the session credentials returned by the SSO credentials provider are valid. Some thoughts to get at this information:
Happy to discuss alternatives too! |
Beta Was this translation helpful? Give feedback.
-
|
@hjkatz On average, how long does it take for This really seems like it should be something that sts models as In case you're not familiar with waiters -- https://aws.github.io/aws-sdk-go-v2/docs/making-requests/#using-waiters. The most ubiquitous example in my mind would be tl;dr waiting for async state changes is a problem we've solved at large, if I understand correctly it's just a question of pushing for sts to add some additional modeling to solve this specific case |
Beta Was this translation helpful? Give feedback.
-
|
Followup - what is the delay between you provisioning the token (looks like through sso) and first calling If it's on the order of seconds, then what I said above generally stands. If this is like minutes or hours, that doesn't seem at all like acceptable behavior in the IAM sense and would warrant further investigation. |
Beta Was this translation helpful? Give feedback.
-
It's on the order of milliseconds. For context we have a shared developer CLI that everyone uses for various commands/tools/utilities. In that CLI we annotate some commands as needing SSO to work correctly. Many of our commands are annotated and interact with AWS in some required way. Our goal was to warn the user that they have not started their SSO session for the running command. To do this we need to check if the session is valid (not sure how to do this), and we came up with generating a token then trying to see if it works.
For our use case I think the order of seconds is too long. It feels like a delay for our users interacting with a CLI. I would prefer milliseconds or some approach that does the bare minimum for testing that the SSO session is valid for generating an STS token or something like that. |
Beta Was this translation helpful? Give feedback.
-
|
Sorry, my last question there was incomplete in wording. I'm trying to understand what the actual delay is you're observing between provisioning the token and then getting a successful call to |
Beta Was this translation helpful? Give feedback.
-
|
I gotcha. I was writing up a test case to get some real data. Here's the summary of 1 million attempts. It's looking much better today than when I originally opened the ticket. For whatever reason I'm not seeing anything take longer than ~100ms but in the past I would feel the delay more in the ~1-2s range, so maybe something's improved. (It could be my network as I'm at my parents' place atm) How about I run this test again daily and get back to you with more data? |
Beta Was this translation helpful? Give feedback.
-
|
Today's summary also seems fine: I'm suspecting that the way I'm using a backoff after getting the credentials should be a naive loop with |
Beta Was this translation helpful? Give feedback.
-
|
Today seems the same too. I'm going to add additional logging into our CLI and see if I can reproduce any inconsistency today, but my suspicion is that 5 attempts isn't enough and we just need to try more times. |
Beta Was this translation helpful? Give feedback.
-
|
My recommendation in general there would not be to fix the number of attempts and instead write your "waiting" construct to accept a timeout. This is how modeled waiters are written (well, generated, but obviously we wrote the code to do that). The waiter then just retries "infinitely" (with an increasing backoff) until it either hits the success case or exceeds the caller-provided deadline. |
Beta Was this translation helpful? Give feedback.
-
|
FYI I've converted this back to a discussion since there's no SDK-side behavioral issue at play here. To summarize so far:
In the absence of a modeled waiter for You should generally not implement waiting logic that polls for a set number of attempts, it's going to be wholly unreliable in practice. Note that if you do see frequent service degradation in the future, in terms of the amount of time it takes for the token to propagate, you'll want to use the AWS console to open a support case with STS. Be prepared to provide request IDs and timestamps if/when that happens. @hjkatz Let me know if you agree with this as written so far. If you do, we'll mark this as the "solution" to the problem at hand. |
Beta Was this translation helpful? Give feedback.
-
|
Yes I think this is correct. The only addition I would add is any nuance around STS regional endpoints. I'll post a follow-up comment below for any discussion. |
Beta Was this translation helpful? Give feedback.
-
|
@lucix-aws I was able to find a reproduction. I think it's dependent on region, specifically the In our CLI we randomly select a region from a list ~8 regions so we don't accidentally rate limit ourselves against the STS endpoints. I think I'll remove region Here's 100 attempts: And some exact timings: |
Beta Was this translation helpful? Give feedback.
-
Describe the bug
See: #2093
Expected Behavior
I expect the call to
sts.GetCallerIdentity()to succeed immediately after authenticating/receiving new sts credentials.Current Behavior
The calls to
sts.GetCallerIdentity()seem to be inconsistently failing.Reproduction Steps
See: #2093
Possible Solution
See: #2093
Additional Information/Context
No response
AWS Go SDK V2 Module Versions Used
Compiler and Version used
go version go1.21.6 linux/amd64
Operating System and version
ubuntu
Beta Was this translation helpful? Give feedback.
All reactions