Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retry Internal Server Error(the server is currently unable to handle the request) #12986

Open
tamayika opened this issue Apr 26, 2024 · 3 comments

Comments

@tamayika
Copy link

When I'm applying helm chart, sometimes K8s server returns Internal Server Error.

Error: Failed to render chart: exit status 1: Error: Unable to continue with install: could not get information about the resource RoleBinding "namespace-labeler" in namespace "ci-namespace": an error on the server ("Internal Server Error: \"/apis/rbac.authorization.k8s.io/v1/namespaces/ci-namespace/rolebindings/namespace-labeler\": the server is currently unable to handle the request") has prevented the request from succeeding (get rolebindings.rbac.authorization.k8s.io namespace-labeler)

Current retry logic is limited to several internal server errors.

// Matches messages like "etcdserver: leader changed"
if strings.HasSuffix(ke.Message, "etcdserver: leader changed") {
return rt.roundTrip(req, retry-1, resp)
}
// Matches messages like "rpc error: code = Unknown desc = raft proposal dropped"
if strings.HasSuffix(ke.Message, "raft proposal dropped") {
return rt.roundTrip(req, retry-1, resp)
}

Is it reasonable to add the server is currently unable to handle the request error to retryable errors?
If ok, I will make a PR.

@sabre1041
Copy link
Contributor

Possibly related #12976

@tamayika
Copy link
Author

tamayika commented May 1, 2024

Thanks for the PR information.
But PR adds retry for 502 and 503, not 500.

@gjenkins8
Copy link
Contributor

It might be okay to retry for GETs and HEADs on 500 (but not other http verbs). Might, because I can't help thinking your api-server might be overloaded in this case. And retying may be unhelpful to that situation. But with the exception of the additional load, retries are generally helpful, and can/should be added for idempotent requests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants