Pyroscope: Provide an example for ebpf sidecar collection

[simonswine]

This shows how Alloy can be used to collect profiles as a sidecar. I am not too sure if this is something we should advertise too heavily, but valuable if you would only like to collect a very specific workloads, that is supported by eBPF.

Maybe we also want to have something like pyroscope exec was for this usecase.

The test workload launches some short-lived sub processes of varying length, the problem is that collection requires to have a quite low discovery.process.refresh_interval

[simonswine]

This is a hard requirement for this to work

[simonswine]

With this I see the 100ms long running processes very rarely. I guess something that we loose by not consuming the PID events from the ebpf.Session in alloy directly.

[korniltsev]

I dont like the idea to run discovery.process every 200ms, but idk how bad it is.
I also don't like using alloy as sidecar at all. The binary is huge, consuming tons of space and memory.

1. As you said we can notify discovery.process from discovery.ebpf about new PID
2. Another option could be to somehow revive (but maybe with a different naming), the targets_only = false && default_target={"service_name": "foo239"}

This could look like this

sidecar {
  enabled=true

  discover_config {
    cwd = true
    exe = true
...
  }
relabel {
  rule {
...
  }
... 
}
}

Can we get k8s labels somehow?

[simonswine]

@korniltsev if you got a moment, would like to understand how you feel about this