Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix hibernatable websocket race between auto-response and regular messages #1130

Merged
merged 2 commits into from
Oct 2, 2023

Conversation

jqmmes
Copy link
Contributor

@jqmmes jqmmes commented Sep 6, 2023

This PR fixes possible ws.send races between auto-response messages and regular web sockets.

@jqmmes jqmmes force-pushed the joaquim/ws-autoresponse-race-fix branch from da9a98f to 275fca0 Compare September 6, 2023 12:39
src/workerd/io/hibernation-manager.c++ Outdated Show resolved Hide resolved
src/workerd/io/hibernation-manager.c++ Outdated Show resolved Hide resolved
src/workerd/api/web-socket.c++ Outdated Show resolved Hide resolved
src/workerd/io/hibernation-manager.c++ Outdated Show resolved Hide resolved
src/workerd/api/web-socket.c++ Outdated Show resolved Hide resolved
src/workerd/api/web-socket.h Outdated Show resolved Hide resolved
@jqmmes jqmmes force-pushed the joaquim/ws-autoresponse-race-fix branch 5 times, most recently from 0b11e54 to 03e0f58 Compare September 8, 2023 11:10
@jqmmes jqmmes force-pushed the joaquim/ws-autoresponse-race-fix branch 2 times, most recently from d9a307d to fad8ffe Compare September 11, 2023 10:56
@jqmmes jqmmes changed the title [WIP] Fix hibernatable websocket race between auto-response and regular messages Fix hibernatable websocket race between auto-response and regular messages Sep 11, 2023
@jqmmes jqmmes marked this pull request as ready for review September 11, 2023 11:10
src/workerd/io/hibernation-manager.h Outdated Show resolved Hide resolved
src/workerd/io/hibernation-manager.h Outdated Show resolved Hide resolved
src/workerd/io/hibernation-manager.c++ Outdated Show resolved Hide resolved
src/workerd/io/hibernation-manager.c++ Outdated Show resolved Hide resolved
src/workerd/api/web-socket.h Outdated Show resolved Hide resolved
src/workerd/api/web-socket.c++ Outdated Show resolved Hide resolved
src/workerd/api/web-socket.c++ Outdated Show resolved Hide resolved
src/workerd/api/web-socket.c++ Outdated Show resolved Hide resolved
src/workerd/api/web-socket.c++ Show resolved Hide resolved
@@ -531,7 +531,10 @@ void WebSocket::send(jsg::Lock& js, kj::OneOf<kj::Array<byte>, kj::String> messa

KJ_UNREACHABLE;
}();
outgoingMessages->insert(GatedMessage{kj::mv(maybeOutputLock), kj::mv(msg)});

auto n = autoResponseStatus.pendingAutoResponseQueue.size() - autoResponseStatus.queuedAutoResponses;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain the difference between GatedMessage::pendingAutoResponses, and AutoResponse::queuedAutoResponses?

Copy link
Contributor Author

@jqmmes jqmmes Sep 27, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GatedMessage::pendingAutoResponses stores the number of AutoResponse::queuedAutoResponses that will be pumped before sending the current GatedMessage, guaranteeing order. AutoResponse::queuedAutoResponses stores the total number of already queued pendingAutoResponses, so we can check how many are yet to be added during each GateMessage instantiation.

@jqmmes jqmmes force-pushed the joaquim/ws-autoresponse-race-fix branch 4 times, most recently from 7352e8c to 6c636ab Compare September 22, 2023 16:09
src/workerd/api/web-socket.c++ Show resolved Hide resolved
src/workerd/api/web-socket.c++ Outdated Show resolved Hide resolved
src/workerd/io/hibernation-manager.h Outdated Show resolved Hide resolved
src/workerd/api/web-socket.h Outdated Show resolved Hide resolved
src/workerd/io/hibernation-manager.h Outdated Show resolved Hide resolved
src/workerd/io/hibernation-manager.c++ Outdated Show resolved Hide resolved
@jqmmes jqmmes force-pushed the joaquim/ws-autoresponse-race-fix branch 2 times, most recently from b9ffe75 to 8917df7 Compare September 27, 2023 11:20
@jqmmes
Copy link
Contributor Author

jqmmes commented Sep 27, 2023

@jasnell can I get another pair of eyes reviewing this PR if you have time?
Thanks!

@jqmmes jqmmes force-pushed the joaquim/ws-autoresponse-race-fix branch from 8917df7 to 7778033 Compare September 27, 2023 15:04
// Keep track of current hibernatable websockets auto-response status to avoid racing
// between regular websocket messages, and auto-responses.
struct AutoResponse {
kj::Promise<void> ongoingAutoResponse = kj::READY_NOW;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahhhh, so I think what I was expecting was that outgoingAutoResponse would be a forked promise ala:

kj::Promise<void> ongoingAutoResponse = kj::Proimse<void>(kj::READY_NOW).fork();

I think what you have here works, it's just a touch unusual! I'm guessing it was hard to phrase sendAutoResponse() in that context.

Copy link
Contributor

@MellowYarker MellowYarker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Joaquim!

This PR fixes possible racing issues between regular websocket messages and auto-responses.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants