fix: Process webhook refresh in background to not block the request (#14269) #18173
Conversation
dhruvang1
force-pushed
the
process-webhook-in-background
branch
from
0db36bd
to
81d564b
Compare
…rgoproj#14269) Signed-off-by: dhruvang1 <dhruvang1@users.noreply.github.com>
dhruvang1
force-pushed
the
process-webhook-in-background
branch
from
81d564b
to
409a7c2
Compare
| h.Add(1) | ||
| go func() { | ||
| defer h.Done() | ||
| h.HandleEvent(payload) | ||
| }() |
There was a problem hiding this comment.
There is a chance that this can be spammed, we should keep the no. of possible routines to a bounded no. configurable by the user. We should follow a worker pool model here instead of spawning 1:1 go routines for every request.
There was a problem hiding this comment.
This is a problem with existing implementation as well. This changes simply terminates the go routine handling the request and creates a new one, instead of continuing the request go routine. IMO, this change is no worse at handling the spam than existing implementation (since go can create thousands of go routines without breaking a sweat). We are more likely to get OOM in case of spam, since each request independently pulls all Apps/AppSet in memory than noticing any problems with go routines.
There was a problem hiding this comment.
That's good point though the no. of simultaneous connections that can possibly be created by the webhook server is restricted by the no. of available sockets/fds for the container, which is bounded and as far as I know can be configured if required. With this change the active connection doesn't live for too long even though a background job has been spun up which is eating the resources behind the scene so we should have the option to limit the no. of active routines. The limit can be hundreds of thousands but that's dependent on the usage of the actual user and the resources they are willing to provide to the controller.
To better handle spam we can probably dedup requests that come in at a quick succession.
This PR moves the webhook processing for both Application and ApplicationSet to a goroutine. This would allow the server to quickly send HTTP 200 to the webhook request adhering to the quick response guidelines set by GitHub, GitLab, etc.
Checklist:
Fixes #14269