monitoring tcp_info - loss, rtt, retransmits, and ecn

[dtaht]

I have a dream that more projects will poke deeper into the underlying behaviors of the tcp stack, to observe what is going wrong there, also.

[ldemailly]

You can combine fortio with network monitoring and visualization through metrics exporter, prometheus and grafana; e.g

https://grafana.com/grafana/dashboards/11074-node-exporter-for-prometheus-dashboard-en-v20201010/

Would that cover what you meant?

[dtaht]

Very pretty dashboard. The key statistics I was looking for were tcp_rtt, retransmits, ecn marks. Not sure if you can plot those from where you are extracting these (snmp?).

You can also get this data from a tool like this:

https://www.measurementlab.net/tests/tcp-info/

[ldemailly]

https://github.com/m-lab/tcp-info#tcp-info interesting, thanks

[dtaht]

The other really cool toolset we hope more adopt is ebpf based in-band "passive ping" tcp monitoring. The pure ebpf version can do 15gbit/core. https://github.com/thebracket/cpumap-pping#cpumap-pping is targetted more at ISPs shaping traffic, but also points back to the research by the famed kathie nichols (codel, diffserv, and pping creator)

It is cheaper! btw, to just poll or collect TCP_INFO stats periodically during a connection or at close, inside the application.

[ldemailly]

There is this we're looking at https://grafana.com/blog/2022/10/24/grafana-and-cilium-deep-ebpf-powered-observability-for-kubernetes-and-cloud-native-infrastructure/ as well