perf.NewReader just hangs if perCPUBuffer is too small

[psanford]

Describe the bug

Using the uretprobe example as my starting point, I tried to get file paths from a tracepoint using bpf_perf_event_output. My userspace go program just hung on perf.Reader.Read() call, because I had set the perCPUBuffer to os.Getpagesize() (which for me was 4096) and my events were larger than that.

To Reproduce

Send a payload larger than os.Getpagesize() via bpf_perf_event_output. E.g.:

#define PATH_MAX 4096
struct event_t {
	u32 pid;
	char str[PATH_MAX];
};

Expected behavior
Ideally perf.Reader.Read() would detect that the buffer is too small and error. I've not looked at how feasible that is. Alternatively, changing the example code to use the size of Event for perCPUBuffer instead of os.Getpagesize() would have helped me:

rd, err := perf.NewReader(objs.Events, int(unsafe.Sizeof(Event{})))

[lmb]

A perf ring has no "inherent sample size", you can submit samples of variable length to it if you want. Therefore the perf reader can't know whether 4096 is enough for your needs or not.

I think if you check the return code from bpf_perf_event_output you'll see that it returns an error because the sample was too big. You may also be able to check https://pkg.go.dev/github.com/cilium/ebpf/perf#Record LostSamples but I haven't validated that.

[florianl]

The error code -28 translates to ENOSPC - No space left on device. It sounds like, sizeof(struct fs_event_t) is too large and should be split or reduced. Can you provide a full working example, that reproduces your issue? From the code snippets it is hard to make assumptions.

[brycekahle]

BPF_MAP_TYPE_PERF_EVENT_ARRAY is special in that the actual map values contain file descriptors to mmap-ed regions. cilium/ebpf and other loaders take care of this detail for you. There is no need to specify the type of the object you are going to use in the perf buffer, since it can actually be variable in size. You need to handle unmarshaling when you call Read on the reader. If you are wanting to control the total size of the perf buffer, you use max_entries / MaxEntries for that.

The key size must be 4 bytes because this is an array, thus the key is an index. For both key size and value size, 0 will cause the loader to automatically set it to 4 bytes.

[brycekahle]

If you getting -28 or ENOSPC when using max entries of 1 << 24, then that means that value is too large. Please remember that a perf buffer will create a buffer per CPU, so the total memory usage is really approximately (1 << 24) * numCPUs.

[asdfsx]

Thx a lot @lmb @florianl @brycekahle
I create repo ebpf-test and upload my test code.
I tried to capture vfs_write event, and get the content.
The ringbuf version seems ok, but the perfevent version is stucked.
Is there any suggestion for improving it？

[florianl]

Just hat a quick look. For perfevents you are allocating os.Getpagesize() (perCPUBuffer) - on most systems os.Getpagesize() comes down to 4096. On the same time, you try to write an event with an element of u8 data[10240]. As 10240 >> 4096, it will not fit into the memory that is allocated for the perfevent.

Please try to shrink or split your element, that you will write into perfevent, so it can fit os.perCPUBuffer. Increasing perCPUBuffer for perf.NewReader() is also possible. But as @brycekahle, please keep in mind that perCPUBuffer is allocted per CPU.