Prevent pprof from causing BPF verifier livelocks #805
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Timo Beckers <timo@isovalent.com>
lmb
requested changes
Sep 27, 2022
internal/unix/signals.go
Outdated
| @@ -0,0 +1,41 @@ | |||
| package unix | |||
There was a problem hiding this comment.
Can you move our own new types from internal/unix to internal/sys? The former should only ever contain wrappers to make macOS users happy.
- Makes it easier to get rid of
internal/unixonce I figure out how to get gopls to pass linux build tags when editing. - Avoids
signals_other.go - Allows unexporting API
There was a problem hiding this comment.
once I figure out how to get gopls to pass linux build tags when editing.
In VSCode, I use:
"gopls": {
"build.buildFlags": [
"-tags=linux,amd64,integration"
],
},
There was a problem hiding this comment.
Avoids
signals_other.go
How, though? Package sys also needs to build on macOS, so we'd still need a signals_other.go there.
Edit: you mean to fold signals_other.go into types_other.go instead?
There was a problem hiding this comment.
I had forgotten about SIG_BLOCK and friends, sorry. I'll try to submit another CL that add them to x/sys, but let's keep SIG_BLOCK in internal/unix for now.
brb
reviewed
Sep 27, 2022
ti-mo
force-pushed
the
tb/block-sigprof
branch
2 times, most recently
from
0bd797e
to
1caaad1
Compare
Signed-off-by: Timo Beckers <timo@isovalent.com>
This patch implements the sigsetAdd function to add a signal to a signal set. Care was taken to make this build and run correctly on different GOOS and GOARCH combinations. Signed-off-by: Timo Beckers <timo@isovalent.com>
sys.maskProfilerSignal() locks the calling goroutine to its underlying OS thread and blocks the SIGPROF signal from interrupting the thread. sys.unmaskProfilerSignal() reverses the operation and should be used in tandem to properly restore OS thread state afterwards. Add a wrapper around unix.PthreadSigmask added in a recent version of x/sys. Signed-off-by: Timo Beckers <timo@isovalent.com>
If a thread receives a signal while blocked in BPF_PROG_LOAD, the verifier can cooperatively interrupt itself by checking pending signals for the thread and return -EAGAIN from the syscall to request userspace to retry. When a Go program is built and run with pprof enabled, threads are routinely sent a SIGPROF to make them dump profiling information, which can lead to a runaway reaction if the program takes longer to verify than the interrupt frequency. To prevent this, mask SIGPROF before calling BPF_PROG_LOAD and remove it when done. Signed-off-by: Timo Beckers <timo@isovalent.com>
ti-mo
force-pushed
the
tb/block-sigprof
branch
from
1caaad1
to
ef08703
Compare
ti-mo
changed the title
lmb
reviewed
Sep 28, 2022
| unmaskProfilerSignal() | ||
|
|
||
| var old unix.Sigset_t | ||
| if err := unix.PthreadSigmask(0, nil, &old); err != nil { |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Read https://dxuuu.xyz/bpf-go-pprof.html for more context.
If a thread receives a signal while blocked in BPF_PROG_LOAD, the verifier can cooperatively interrupt itself by checking pending signals for the thread and return -EAGAIN from the syscall to request userspace to retry.
When a Go program is built and run with pprof enabled, threads are routinely sent a SIGPROF to make them dump profiling information, which can lead to a runaway reaction if the program takes longer to verify than the interrupt frequency. To prevent this, mask SIGPROF before calling BPF_PROG_LOAD and remove it when done.
@danobi