PyOpenSSL / CFFI freezes when running codesigned x64 binary on M1/Rosetta. #8099

mhils · 2023-11-12T22:14:21Z

Description of the issue

PyInstaller's codesigning causes hangs with PyOpenSSL/cffi code when running macOS x86_64 binaries on Apple Silicon (M1) using Rosetta.

Context information (for bug reports)

Output of pyinstaller --version: pyinstaller 6.2.0
Version of Python: 3.12
Platform: macOS arm64
How you installed Python: brew

A minimal example program which shows the error

Here is a full reproducers for arm-based Macs:

#!/usr/bin/env bash
set -xeou pipefail

git clone https://github.com/Homebrew/brew homebrew-x64
ditto homebrew-x64 homebrew-arm

arch -x86_64 homebrew-x64/bin/brew install python@3.12
             homebrew-arm/bin/brew install python@3.12

homebrew-x64/bin/python3.12 -m venv venv-x64
homebrew-arm/bin/python3.12 -m venv venv-arm

venv-x64/bin/pip install pyinstaller pyopenssl
venv-arm/bin/pip install pyinstaller pyopenssl

cat >repro.py <<EOL
from OpenSSL._util import ffi
print("creating callback")
ffi.callback(
	"int (*)(SSL *, unsigned char **, unsigned char *, "
	"const unsigned char *, unsigned int, void *)",
	lambda *_, **__: None
)
print("ok")
EOL

venv-x64/bin/pyinstaller --clean --distpath dist-x64 --codesign-identity "Developer ID Application" repro.py
venv-arm/bin/pyinstaller --clean --distpath dist-arm --codesign-identity "Developer ID Application" repro.py

echo "Running ARM version..."
dist-arm/repro/repro  # works
echo "Running x64 version..."
dist-x64/repro/repro  # hangs

Additional observations:

The x86_64 binary does work on an intel Mac, see https://github.com/mhils/cffi-repro/actions/runs/6843292296/job/18605724809 for an example.
venv-x64/bin/python repro.py works flawlessly.
If I skip --codesign-identity, the binary works locally (but cannot be distributed).
You can find an example binary here (mirror: dist-x64.zip), which exhibits this issue after (chmod +x repro/repro && /usr/bin/xattr -dr com.apple.quarantine repro && repro/repro)

Stacktrace / full error message

None. The program just hangs with high CPU utilisation.

The text was updated successfully, but these errors were encountered:

rokm · 2023-11-12T22:33:21Z

Doesn't CFFI compile stuff on the fly? In that case, the resulting on-the-fly-compiled binary it generates is very likely not signed (with your codesign identity), and I would expect it to cause issues?

But if it works on the corresponding native platform (i.e., signed x86_64 build running on x86_64 Mac and signed arm64 build running on Apple Silicon Mac), then perhaps that's a Rosetta issue?

mhils · 2023-11-12T22:41:17Z

Yeah – as you said, x86_64 on x86_64 and arm64 on arm64 both work. x86_64 python on arm64 (without pyinstaller) also works, despite Rosetta.

rokm · 2023-11-12T22:42:22Z

Are you sure x86_64 works on x86_64, though? On my test x86_64 system, it also seems to get stuck until I remove the signature from the executable...

mhils · 2023-11-12T22:44:01Z

Yes, both locally and in CI. The GitHub Actions run linked above is one example.

rokm · 2023-11-12T22:48:38Z

x86_64 python on arm64 (without pyinstaller) also works, despite Rosetta.

That's largely irrelevant, though, because you don't have signing in python (nor hardened run-time that comes with it, when you use PyInstaller's --codesign-identity).

rokm · 2023-11-12T22:51:43Z

Also as a side note, .zip file fails to preserve symbolic links in the generated frozen application bundle. This might not have been issue with older PyInstaller versions that had no notion of symbolic links, but it is with PyInstaller >= 6.0.

rokm · 2023-11-12T23:14:39Z

Well, like I said, it gets stuck on my x86_64 test system as well, and I can see the system log spamming the following:

error	00:01:02.711811+0100	kernel	Library Validation failed: Rejecting '/private/var/folders/rd/v4qf4xpj3slf6nvnrp0gb1xh0000gn/T/ffijB38iE' (Team ID: none, platform: no) for process 'repro(1348)' (Team ID: S8XHQB96PW, platform: no), reason: mapped file has no cdhash, completely unsigned? Code has to be at least ad-hoc signed.
default	00:01:02.712275+0100	kernel	ASP: Library load (/private/var/folders/rd/v4qf4xpj3slf6nvnrp0gb1xh0000gn/T/ffijB38iE) rejected: library load disallowed by system policy
error	00:01:02.712106+0100	syspolicyd	Unable (errno: 2) to read file at <private> for process path: <private> library path: <private>
error	00:01:02.712167+0100	syspolicyd	Disallowing load of <private> in 1348, <private>
default	00:01:02.714325+0100	amfid	UNIX error exception: 2
default	00:01:02.715764+0100	amfid	MacOS error: -67068
error	00:01:02.716572+0100	kernel	Library Validation failed: Rejecting '/private/tmp/ffi7Pi5CE' (Team ID: none, platform: no) for process 'repro(1348)' (Team ID: S8XHQB96PW, platform: no), reason: mapped file has no cdhash, completely unsigned? Code has to be at least ad-hoc signed.
default	00:01:02.717009+0100	kernel	ASP: Library load (/private/tmp/ffi7Pi5CE) rejected: library load disallowed by system policy
error	00:01:02.716834+0100	syspolicyd	Unable (errno: 2) to read file at <private> for process path: <private> library path: <private>
error	00:01:02.716893+0100	syspolicyd	Disallowing load of <private> in 1348, <private>
default	00:01:02.717932+0100	amfid	UNIX error exception: 2
default	00:01:02.718564+0100	amfid	MacOS error: -67068
error	00:01:02.723469+0100	syspolicyd	Unable (errno: 2) to read file at <private> for process path: <private> library path: <private>
error	00:01:02.723553+0100	syspolicyd	Disallowing load of <private> in 1348, <private>
default	00:01:02.724315+0100	amfid	UNIX error exception: 2
default	00:01:02.725126+0100	amfid	MacOS error: -67068
error	00:01:02.725919+0100	kernel	Library Validation failed: Rejecting '/private/var/folders/rd/v4qf4xpj3slf6nvnrp0gb1xh0000gn/T/ffiRA2xvQ' (Team ID: none, platform: no) for process 'repro(1348)' (Team ID: S8XHQB96PW, platform: no), reason: mapped file has no cdhash, completely unsigned? Code has to be at least ad-hoc signed.
error	00:01:02.726285+0100	syspolicyd	Unable (errno: 2) to read file at <private> for process path: <private> library path: <private>
error	00:01:02.726340+0100	syspolicyd	Disallowing load of <private> in 1348, <private>
default	00:01:02.726429+0100	kernel	ASP: Library load (/private/var/folders/rd/v4qf4xpj3slf6nvnrp0gb1xh0000gn/T/ffiRA2xvQ) rejected: library load disallowed by system policy
default	00:01:02.727067+0100	amfid	UNIX error exception: 2
default	00:01:02.727713+0100	amfid	MacOS error: -67068
error	00:01:02.729370+0100	kernel	Library Validation failed: Rejecting '/private/tmp/ffiDmI0q7' (Team ID: none, platform: no) for process 'repro(1348)' (Team ID: S8XHQB96PW, platform: no), reason: mapped file has no cdhash, completely unsigned? Code has to be at least ad-hoc signed.
error	00:01:02.729523+0100	syspolicyd	Unable (errno: 2) to read file at <private> for process path: <private> library path: <private>
error	00:01:02.729593+0100	syspolicyd	Disallowing load of <private> in 1348, <private>
default	00:01:02.729685+0100	kernel	ASP: Library load (/private/tmp/ffiDmI0q7) rejected: library load disallowed by system policy
default	00:01:02.731007+0100	amfid	UNIX error exception: 2
error	00:01:02.737989+0100	kernel	Library Validation failed: Rejecting '/Users/rok/ffiIVbLAl' (Team ID: none, platform: no) for process 'repro(1348)' (Team ID: S8XHQB96PW, platform: no), reason: mapped file has no cdhash, completely unsigned? Code has to be at least ad-hoc signed.
default	00:01:02.738394+0100	kernel	ASP: Library load (/Users/rok/ffiIVbLAl) rejected: library load disallowed by system policy
error	00:01:02.738151+0100	syspolicyd	Unable (errno: 2) to read file at <private> for process path: <private> library path: <private>
error	00:01:02.738204+0100	syspolicyd	Disallowing load of <private> in 1348, <private>
default	00:01:02.739351+0100	amfid	UNIX error exception: 2
default	00:01:02.740144+0100	amfid	MacOS error: -67068
error	00:01:02.742013+0100	kernel	Library Validation failed: Rejecting '/private/var/folders/rd/v4qf4xpj3slf6nvnrp0gb1xh0000gn/T/ffi6Dk9pp' (Team ID: none, platform: no) for process 'repro(1348)' (Team ID: S8XHQB96PW, platform: no), reason: mapped file has no cdhash, completely unsigned? Code has to be at least ad-hoc signed.
error	00:01:02.742208+0100	syspolicyd	Unable (errno: 2) to read file at <private> for process path: <private> library path: <private>
default	00:01:02.742367+0100	kernel	ASP: Library load (/private/var/folders/rd/v4qf4xpj3slf6nvnrp0gb1xh0000gn/T/ffi6Dk9pp) rejected: library load disallowed by system policy

So I'm still leaning towards this being an issue with CFFI-generated binary not being signed.

I'm actually surprised it works on arm64, although perhaps the catch is that on arm64 CFFI at least ad-hoc codesigns the binary (since on arm64, a valid signature, even if an ad-hoc one, has been a hard requirement since its introduction), but apparently on x86_64, it does not.

Can you check if you get similar errors in the system log with Rosetta as well?

rokm · 2023-11-12T23:21:41Z

Perhaps you need to look into adding proper entitlements when signing the application? E.g., com.apple.security.cs.allow-jit, com.apple.security.cs.allow-unsigned-executable-memory, com.apple.security.cs.disable-library-validation.

Or maybe omit --codesign-identity and manually sign the executable and shared libraries without hardened runtime enabled, if you can get away with that (or remove this part in your local copy of PyInstaller).

mhils · 2023-11-13T10:41:54Z

Can you check if you get similar errors in the system log with Rosetta as well?

Yes!

Perhaps you need to look into adding proper entitlements when signing the application?

And Yes! This fixes it, wonderful. I didn't expect the lack of entitlements to manifest in a hang, but I'll take it. 🎉

Thank you so much for the super super speedy and helpful replies, this has been fantastic! 🍰 ✨ 😃

petiatil · 2023-11-15T19:23:31Z

@mhils I've been struggling with the same sort of issue for a few weeks.

Can you please give any other details on your use case? For instance:

Is the reason you (presumably) aren't using inheritance-based entitlements, because the binary is the main executable of your app, and so is implementing its own entitlements?

When I test the PyInstaller-bundled app directly, using its own entitlements rather attempting to inherit (i.e. as required when launching a helper process from the Xcode-built parent app, but fails), it passes library validation and doesn't hang.

If you have any insight into possible causes or solutions for this issue, they would be much appreciated (even if to help rule something out).

mhils · 2023-11-15T19:28:25Z

@petiatil: We're not using inheritance-based entitlements because we don't have an [Xcode-built] parent app. It's the main app. Sorry, that's probably not too helpful for your case. 😶

mhils added the triage Please triage and relabel this issue label Nov 12, 2023

mhils mentioned this issue Nov 12, 2023

Apple Silicon (macOS M1) Freezes -> Update to 10.1.4 mitmproxy/mitmproxy#6474

Closed

mhils closed this as completed Nov 13, 2023

rokm mentioned this issue Nov 13, 2023

Using codesigning flags still results in "no cdhash, completely unsigned?" errors (with --onedir executable or app bundle executable) #8029

Open

github-actions bot locked as resolved and limited conversation to collaborators Jan 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PyOpenSSL / CFFI freezes when running codesigned x64 binary on M1/Rosetta. #8099

PyOpenSSL / CFFI freezes when running codesigned x64 binary on M1/Rosetta. #8099

mhils commented Nov 12, 2023

rokm commented Nov 12, 2023

mhils commented Nov 12, 2023

rokm commented Nov 12, 2023

mhils commented Nov 12, 2023

rokm commented Nov 12, 2023 •

edited

rokm commented Nov 12, 2023

rokm commented Nov 12, 2023

rokm commented Nov 12, 2023

mhils commented Nov 13, 2023

petiatil commented Nov 15, 2023

mhils commented Nov 15, 2023

PyOpenSSL / CFFI freezes when running codesigned x64 binary on M1/Rosetta. #8099

PyOpenSSL / CFFI freezes when running codesigned x64 binary on M1/Rosetta. #8099

Comments

mhils commented Nov 12, 2023

Description of the issue

Context information (for bug reports)

A minimal example program which shows the error

Stacktrace / full error message

rokm commented Nov 12, 2023

mhils commented Nov 12, 2023

rokm commented Nov 12, 2023

mhils commented Nov 12, 2023

rokm commented Nov 12, 2023 • edited

rokm commented Nov 12, 2023

rokm commented Nov 12, 2023

rokm commented Nov 12, 2023

mhils commented Nov 13, 2023

petiatil commented Nov 15, 2023

mhils commented Nov 15, 2023

rokm commented Nov 12, 2023 •

edited