New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[chttp2] fix stream leak with queued flow control update and absence of writes #30907
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
grpc-checks
bot
added
per-call-memory/neutral
per-channel-memory/neutral
bloat/none
labels
Sep 9, 2022
This reverts commit 57f78ee.
ctiller
approved these changes
Sep 14, 2022
apolcyn
added a commit
to apolcyn/grpc
that referenced
this pull request
Sep 14, 2022
…ate and absence of writes (grpc#30907)" (grpc#30991)" This reverts commit 0f2a0f5.
copybara-service
bot
added
the
imported
Specifies if the PR has been imported to the internal repository
label
Sep 14, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
bloat/none
imported
Specifies if the PR has been imported to the internal repository
lang/core
per-call-memory/neutral
per-channel-memory/neutral
release notes: yes
Indicates if PR needs to be in release notes
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As discovered with internal issue b/242712384, there's a leak that can happen as follows:
WritesDone()
Since writes are closed first, chttp2 will call grpc_chttp2_mark_stream_closed with
close_reads=1
, during 2). This can cause remove_stream to be called before updating flow control and taking a new ref on the stream.In my repro of b/242712384, I observed this last stream flow control update to always have queuing urgency. In this case, nothing is guaranteed to initiate further writes on the transport (and the RPC has completed as far as the application is concerned). So even after the application destroys the channel and fully unrefs the RPC, the ref held by the queued flow control update will keep the stream and connection objects alive indefinitely.
Edit:
Since #30550 submitted, the above description doesn't actually repro because of the change to
StreamFlowControl::UpdateAction
. But the same underlying bug can also repro in a different way, which is what the test is now based off of:server sends a message and status
client polls the connection, reads message and status off the wire and marks the stream closed, without having yet started a RECV_MESSAGE or RECV_STATUS op
client performs RECV_MESSAGE and RECV_STATUS ops, which complete immediately. But the stream flow control update is queued and not flushed out because we already had all the data and didn't need to send any pre-emptive updates.