Do not gracefully drain the proxy on SIGTERM #2266
Draft
+66
−86
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Per the discussion in linkerd/linkerd2#10379, the proxy's shutdown behavior is overly aggressive and does not properly honor a pod's `terminationGracePeriodSeconds` configuration: while an application container may continue to run with a server for some period after the pod's delete process begins, the proxy will not permit any new connections in or out of the pod. This may also interfere with administrative probes during the shutdown process. To address this, the proxy is updated to only change its readiness probe state when it receives a SIGTERM. The proxy continues to run-- without closing any connections or rejecting any work--until it is terminated (i.e., by kubelet) with a SIGKILL. It is expected that, in most cases, clients will voluntarily move traffic as they process discovery updates. We need not do anything to interfere with the application's graceful termination behavior. The proxy continues its former graceful shutdown behavior when shutdown is initiated via the admin server `/shutdown` endpoint (e.g., by linkerd-await).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Per the discussion in linkerd/linkerd2#10379, the proxy's shutdown behavior is overly aggressive and does not properly honor a pod's
terminationGracePeriodSecondsconfiguration: while an application container may continue to run with a server for some period after the pod's delete process begins, the proxy will not permit any new connections in or out of the pod. This may also interfere with administrative probes during the shutdown process.To address this, the proxy is updated to only change its readiness probe state when it receives a SIGTERM. The proxy continues to run-- without closing any connections or rejecting any work--until it is terminated (i.e., by kubelet) with a SIGKILL.
It is expected that, in most cases, clients will voluntarily move traffic as they process discovery updates. We need not do anything to interfere with the application's graceful termination behavior.
The proxy continues its former graceful shutdown behavior when shutdown is initiated via the admin server
/shutdownendpoint (e.g., by linkerd-await).