Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add details on native packaging requirements exposed by mobile platforms #27

Open
wants to merge 31 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
eb2419a
Add details on native packaging requirements exposed by mobile platfo…
freakboy3742 Jan 9, 2023
8fef63e
Clarified the role/impact of cross-compilation on non-macOS platforms.
freakboy3742 Jan 10, 2023
d16035f
Grammar cleanup.
freakboy3742 Jan 10, 2023
84dbd5f
Add note about Windows platform support
freakboy3742 Jan 10, 2023
2a40f47
Moved a paragraph about the universal2 to current state.
freakboy3742 Jan 10, 2023
2563270
Clarified how Android deals with dependencies.
freakboy3742 Jan 10, 2023
b9b904c
Added an alternative approach for handling iOS multi-arch.
freakboy3742 Jan 10, 2023
45f748f
Modified comments to use common section structure, and include specif…
freakboy3742 Jan 16, 2023
373bb09
Apply suggestions from code review
freakboy3742 Jan 16, 2023
d8a2ca6
More updates stemming from review.
freakboy3742 Jan 16, 2023
f533395
Expand note about Linux support.
freakboy3742 Jan 17, 2023
8475360
Correct an it's typo.
freakboy3742 Jan 17, 2023
2886f2c
Add content to page on cross compilation
rgommers Feb 27, 2023
7556850
Resolve the last cross-compilation comment, on `pip --platform`
rgommers Mar 10, 2023
cb85652
Merge branch 'main' into mobile-details
rgommers Mar 10, 2023
49806e2
Put back link to "multiple architectures" page from cross compile page
rgommers Mar 10, 2023
ea1fb60
Remove the `cross_platform.md` file
rgommers Mar 10, 2023
d249af6
Fix some formatting and typo issues
rgommers Mar 10, 2023
50d8c26
Revisions to multi-architecture notes following review.
freakboy3742 Mar 20, 2023
a9776e0
Add foldout for pros and cons of `universal2` wheels
rgommers Mar 21, 2023
8d46e06
Add the 'for' arguments for universal2.
freakboy3742 Mar 21, 2023
5d06a56
Clarified 'end user' language; added note about merge problems.
freakboy3742 Mar 22, 2023
3e1fc05
Clarify the state of arm64 on github actions.
freakboy3742 Mar 22, 2023
74705d8
Add reference to pip issue about universal2 wheel installation.
freakboy3742 Mar 22, 2023
f46d2b0
Fixed typo.
freakboy3742 Mar 22, 2023
e1c278f
Removed subjective language.
freakboy3742 Mar 22, 2023
1a926eb
Apply textual/typo suggestions
rgommers Mar 22, 2023
7967383
Rephrase universal2 usage frequency/demand phrasing
rgommers Mar 22, 2023
1fb0ffb
Tone down the statement on "must provide thin wheels"
rgommers Mar 22, 2023
b44a322
Rephrase note on needed robustness improvements in delocate-fuse
rgommers Mar 22, 2023
dd93f1f
Add "first-class support for fusing thin wheels" as a potential solution
rgommers Mar 22, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
62 changes: 62 additions & 0 deletions docs/key-issues/multiple_architectures.md
Expand Up @@ -107,6 +107,68 @@ To support the transition to Apple Silicon/M1 (ARM64), Python has introduced a
`.dylib` files contained in the wheel are fat binaries containing both x86-64
and ARM64 slices.


??? question "What's the deal with `universal2` wheels?"

The `universal2` fat wheel format has generated quite a bit of discussion,
and isn't well-supported by either packaging tools (e.g., there is no way
to install a `universal2` wheel from PyPI if thin wheels are also present)
or package authors (most numerical, scientific and ML/AI package authors do
not provide them). There are some arguments for and against supporting the
format or even defaulting to it.

Arguments for (Russell to write):

- xxx

Arguments against:

- `universal2` wheels are never necessary for end users, they are only an
intermediate stage for workflows and tooling to build macOS apps (`.dmg`
downloadable installers or similar formats, produced by for example
[py2app](https://py2app.readthedocs.io) or
[briefcase](https://beeware.org/project/projects/tools/briefcase/)).
- The tradeoff between download size and disk space usage vs. the upside
for say a .dmg installer is bad - for a typical PyData stack it takes
rgommers marked this conversation as resolved.
Show resolved Hide resolved
hundreds of MBs per Python environment more than thin wheels, and users
are likely to have quite a few environments on their system at once.
Meaning that defaulting to `universal2` would use several GBs of disk
space more.

- Disk space on the base MacBook models is 128 GB, and up to half of that
can be taken up by the OS and system data itself. So a few GBs is
significant.
- Internet plans in many countries are not unlimited; almost doubling the
download size of wheels is a serious cost, and not desirable for any
user - but especially unfriendly to users in countries where network
infrastructure is less developed.

- In addition, it takes extra space on PyPI (examples: `universal2` wheels
cost an extra 81.5 MB for NumPy 1.21.4 and 175.5 MB for SciPy 1.9.1), and
projects with large wheels often run into total size limits on PyPI.
- It imposes an extra maintenance burden for each project, because separate
CI jobs are needed to build and test `universal2` wheels. Typically
projects make tradeoffs there, because they cannot support every
platform. And `universal2` doesn't meet the bar for usage frequency /
user demand here - it's well below the demand for `musllinux`,
`ppc64le`, PyPy, and other such platforms with still patchy support
rgommers marked this conversation as resolved.
Show resolved Hide resolved
(see [Expectations that projects provide ever more wheels](../../meta-topics/user_expectations_wheels.md)
for more on that).
- When a project provides thin wheels (which is a must-do for projects with
native code, because those are the better experience due to smaller
size), you cannot even install a `universal2` wheel with pip from PyPI at
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, subjective, use-case specific language. In my use case, universal2 wheels unequivocally provide a better experience.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"because those are the better experience with pip install into an active environment" perhaps?

Because it is unequivocally better for all use cases where you aren't going to transfer the results to another computer. And virtual environments are not relocatable. And I'd imagine that conda doesn't provide a universal2 interpreter so conda environments might be relocatable, but presumably not across machines.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And I'd imagine that conda doesn't provide a universal2 interpreter so conda environments might be relocatable, but presumably not across machines.

Conda environments should be more or less relocatable (not that I'd recommend it), but only within the same CPU architecture (more precisely, something closely resembling the target triple). Everything in conda is per-arch, there are no fat binaries or something like universal2.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"because those are the better experience with pip install into an active environment" perhaps?

That sounds fine to me. Maybe adding "as an end user"? I imagine briefcase and py2app also use venv's under the hood.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking that those would be inactive environments, but it's a subtle distinction and "as an end user" is probably a better one.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW, Briefcase doesn't use literal venvs, but it does do some venv-like tricks to ensure interpreter isolation. I can't speak to the current state of py2app.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I toned this down from "must do" to "should be done" and added the clarification that it's only better when installing for use on that machine. Please resolve this comment if that looks good.

all. Why upload artifacts you cannot install?
rgommers marked this conversation as resolved.
Show resolved Hide resolved
- It is straightforward to fuse two thin wheels with `delocate-fuse` (a
tool that comes with [delocate](https://pypi.org/project/delocate/)),
it's a one-liner: `delocate-fuse $x86-64_wheel $arm64_wheel -w .`
- Open source projects rely on freely available CI systems to support
particular hardware architectures. CI support for macOS `arm64` was a
problem at first, but is now available through Cirrus CI. And that
availability is expected to grow over time; GitHub Actions and other
providers will roll out support at some point. This allows building thin
rgommers marked this conversation as resolved.
Show resolved Hide resolved
wheels and run tests - which is nicer than building `universal2` wheels
on x86-64 and testing only the x86-64 part of those wheels.

iOS has an additional complication of requiring support for mutiple *ABIs* in
rgommers marked this conversation as resolved.
Show resolved Hide resolved
addition to multiple CPU architectures. The ABI for the iOS simulator and
physical iOS devices are different; however, ARM64 is a supported CPU
Expand Down