Dynamically discover offsets for finding custom labels map #2748
Conversation
This reads the DWARF info of a Go program to discover the offsets of the map containing custom pprof labels, and passes them to the unwinder, rather than hardcoding them manually. As before, the unwinder walks this label map to discover the OTEL thread ID, if it has been saved. Future work will include extending this to Rust as well as making it work in more exotic situations (e.g. when the Go runtime has been loaded from a shared library rather than being part of the main executable)
|
Since for go there is only one compiler and these are runtime structs, I think we should actually just extract these once, store the results in runtime data and based on the Go version decide which offsets to load. The nice thing about that is that debugger-level debuginfo don't have to be present in order to do this, we just need to find which Go version a binary is (which we already do anyway). FWIW goroutine labels were entirely broken until Go 1.18, so we would only need offsets back to 1.18 anyway. |
Can a go binary be stripped of this information? I think it can but I'm not sure. I almost wonder if we want the best of both worlds, dynamic lookup when dwarf symbols are present to support new releases w/o having to re-release our software w/ runtime-data to fallback on when dwarf symbols aren't present. For reference luajit release binaries are fully stripped and dwarf lookup would be a non-starter if we needed to dynamically lookup field offsets (which so far I haven't needed but suspect might be necessary down the road). |
|
Yeah Go binaries don't even have the DWARF required for this enabled by default, you need to pass special flags for it. The only thing that cannot be stripped from Go binaries is the gopclntab section. I like the idea of a hybrid approach here, we use the pre-extracted offsets for known versions and for custom Go builds (which do exist) or those that we don't have offsets for yet, we fall back to trying to extract it from debuginfo. |
Wait a minute, if go binaries don't have this information then I think we are doing it wrong. The Go binary has to know where pointers live in the "g" struct or the garbage collector couldn't work. I think maybe we need to get this information from pclntab? This project seems interesting: https://github.com/mandiant/GoReSym I also have a faint recollection that dlv can debug stripped binaries because it also uses the pclntab to figure out this stuff. I think we should fully understand what the solution space is here before proceeding, right now my thinking is this information has to be in the go binary, we should always be able to get it and we have no need for ahead of time runtime-data for Go (rust/C/C++ may be different). I'm prepared to be wrong! |
|
Closing in favor of #2808 |
This reads the DWARF info of a Go program to discover the offsets of the map containing custom pprof labels, and passes them to the unwinder, rather than hardcoding them manually. As before, the unwinder walks this label map to discover the OTEL thread ID, if it has been saved.
Future work will include extending this to Rust as well as making it work in more exotic situations (e.g. when the Go runtime has been loaded from a shared library rather than being part of the main executable)