Native profiling on macOS

[xhochy]

It seems that there was at one time support for native profiling on macOS and nowadays it is disabled due to missing features. What is needed to get native profiling for macOS back again?

[benfred]

There was a dev release with OSX native profling: pip install py-spy==0.2.0.dev4 will install the last version.

It didn't work all that well though, so I removed it =(. It was using a custom unwinder I wrote, built on top of gimli for dwarf unwinding. This frequently failed to unwind stack traces that I could unwind with llvm. Rather than ship known broken code, I pulled it out and just shipped the windows/linux version.

The tricky thing here is unwinding the native stack of a thread in another process. To get this going we have a couple of options that I know of - and probably more that I don't:

1. Fork the libunwind distributed by apple, and add hooks for remote unwinding from another process.

2. Fix up the gimli based unwinder I wrote for that dev release. This is probably harder than just trying to use a different unwinder

3. Extract the unwinder from llvm and use that (haven't looked into how much work is here).

[huguesb]

@benfred I would be interested in taking a stab at this. Do you have more details on what went wrong with your custom unwinder? How were you testing the correctness of the unwinder against llvm?

[mstange]

The incorrect unwinding was likely because macOS largely doesn't use DWARF CFI; it's only used as a last resort if Apple's own "Compact Unwinding Format" is insufficient to describe the unwind rule for a function.
See this comment for some more context.

I'd like to encourage you to give framehop a try. I use it successfully in perfrecord to get native stacks on macOS.

[benfred]

For OSX - I had some code that attempted to read in the the compact unwinding format for OX, but I was seeing fairly high error rates and removed it entirely ages ago. I was verifying a paused process using my unwinder and using lldb (for the llvm unwinder), and wasn't able to get correct stack traces around 25% of the time iirc. Fwiw, the compact unwind code was here https://github.com/benfred/remoteprocess/blob/41e37740e4b7d4ea148bbf6abbe5fe674ef50cb9/src/osx/compact_unwind.rs

@mstange - framehop looks great! I definitely am interested in giving it a go, potentially even using for linux as well instead of libunwind-ptrace . Does this include support for symbolication on OSX, or just unwinding?

In terms of getting framehop integrated, from scanning the example https://github.com/mstange/framehop/#example, it looks to me like we need to:

• Get the module information for each shared library. We have some code in https://github.com/rbspy/proc-maps that does this, which we used in the previous attempt to read here https://github.com/benfred/remoteprocess/blob/41e37740e4b7d4ea148bbf6abbe5fe674ef50cb9/src/osx/unwinder.rs#L130-L132 . It looks like perfrecord also has the same functionality here https://github.com/mstange/perfrecord/blob/master/perfrecord/src/proc_maps.rs

• Get initial set of registers to unwind. We have that functionality available from the process handle: https://github.com/benfred/remoteprocess/blob/808d6fa89f00bcecc4664eb6792933ac4dcca110/src/osx/mod.rs#L194-L205 still.

• Provide a callback for reading memory - which we can also do get from the process handle https://github.com/benfred/remoteprocess/blob/808d6fa89f00bcecc4664eb6792933ac4dcca110/src/osx/mod.rs#L152-L157 , using the read_process_memory crate.

@huguesb is this something you're interested in taking on? I'd love a PR if so,

[mstange]

Fwiw, the compact unwind code was here https://github.com/benfred/remoteprocess/blob/41e37740e4b7d4ea148bbf6abbe5fe674ef50cb9/src/osx/compact_unwind.rs

Oh, very neat!

@mstange - framehop looks great! I definitely am interested in giving it a go, potentially even using for linux as well instead of libunwind-ptrace . Does this include support for symbolication on OSX, or just unwinding?

Thank you! It only does unwinding, no symbolication.

In terms of getting framehop integrated, from scanning the example https://github.com/mstange/framehop/#example, it looks to me like we need to: [...]

That all sounds right!

It looks like perfrecord also has the same functionality here https://github.com/mstange/perfrecord/blob/master/perfrecord/src/proc_maps.rs

Yup, I think that file even started with the proc-maps crate code originally.

[huguesb]

@benfred I am sufficiently interested in getting this feature that I will eventually take a stab at implementing it myself. It might take me a little while to get started though, since this would be my first time actually writing Rust...

[movermeyer]

I'm a complete n00b when it comes to this sort of thing, but I saw enough overlapping keywords and thought maybe this might be useful for this??? IDK, just sharing in the chance it is useful:

Python 3.12 is improving debugging for MacOS:

• Include debug symbols in MacOS for 3.9-3.11 pyenv/pyenv#2466
• Add an option to include debug information in the Python executable and shared libraries in MacOS python/cpython#95973

Even if it isn't directly helpful for stack unwinding, perhaps it is an interesting development for you folks.

[duburcqa]

Any progress on this ? It would be amazing to be able to profile code on Apple Silicon !