Reduce overhead of polling perf buffers

[javierhonduco]

In the profiler we develop we use perf buffers to communicate events with userspace. We use this to notify of new processes that we need to generate information for, among other things. We use the default timeout, 300ms. While in the future we might conditionally use ring buffers, we have to support perf buffers for older kernels (<5.8).

While analysing the performance of our own profiler we've noticed that almost 26% of the CPU cycles are spent polling the buffers. It's well known that the Go-C boundary crossing is not cheap (thanks to Go for not following C's ABI!!), which is already documented in this TODO:

libbpfgo/libbpfgo.go

Line 1897 in 0aa3396

// todo: consider writing the perf polling in go as c to go calls (callback) are expensive

Would be curious to know if this is something you've also experienced in Tracee. It would be fantastic to see if implementing this in Go would help here. I think it would!

In the meantime, I've opened #309 to configure the timeout, which is something we needed even if the overhead were lower, but that can help folks that are willing to reduce overhead despite the higher chances of lost events and higher latency.

[rafaeldtinoco]

Would be curious to know if this is something you've also experienced in Tracee. It would be fantastic to see if implementing this in Go would help here. I think it would!

This is something we have discussed historically indeed. There was an issue opened in Tracee for moving away from the CGO polling logic and implement that in Go (inside libbpfgo).

Maybe the time has come ? @yanivagman FYI

[yanivagman]

This is something we have discussed historically indeed. There was an issue opened in Tracee for moving away from the CGO polling logic and implement that in Go (inside libbpfgo).

Maybe the time has come ? @yanivagman FYI

Yes, indeed we discussed this in the past (#42).
Like I wrote in the other issue, cgo improved in recent versions of go and I didn't see any particular improvement moving to pure go. Yet, I didn't put too much effort into this back then, and it may be a good idea to explore it again if you see a performance impact related to cgo.

[javierhonduco]

FWIW we use Go 1.20