[agentbeat/linux] shirou/gopsutil causes high CPU usage

[florianl]

agentbeat uses shirou/gopsutil to fetch network information.

When testing a current release, the following profile from agentbeat was collected.

From this on CPU profile, agentbeat spends most of its CPU resources on traversing /proc/net and /proc/<PID>/net in order to read files to fetch networking information.

Long standing issues on shirou/gopsutil to address performance issues when collecting network information:

• net.Connections("all") causes high CPU usage on server with 120K connections shirou/gopsutil#784
• Use netlink API to get net connections for better performance shirou/gopsutil#695

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[pierrehilbert]

Thx @florianl for your feedback.
Did you observe a worse behavior than standalone Beats or are we having the same CPU consumption?
cc @blakerouse

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[florianl]

It is hard to compare the standalone agentbeat with a collection of multiple beats. Before the change to agentbeat each beat was more independent and could use its available resources. Now all resources are channeled and internals of agentbeat are competing with each other for these resources. From that perspective, I would say, the behavior is worse than before.

[cmacknz]

Interesting, we haven't changed the number of processes (and in theory the work done by those processes), only the executable used to implement those processes.

[blakerouse]

As @cmacknz please compare this to native to metricbeat, because the number of processes and the work each process does has not changed.