[Help Required] qemu 8.1-rc1 requires Python venv and pip #8009

heitbaum · 2023-07-27T09:20:39Z

Describe the bug

Build of qemu is failing due to need to pip and vent for python

Ref [Solution Required] python setuptools needs requires pip/pep517-compliant builder #5538

need a better solution than “downloading” during build.

need help from someone more familiar with python build environment.

To Reproduce

Steps to reproduce the behavior:

Update to qemu-8.1-rc1
Run build.LibreELEC-Generic.x86_64-12.0-devel/toolchain/bin/python3 -m ensurepip --default-pip to fix problem one
Problem 2 is qemu-8.1.0-rc1/.x86_64-linux-gnu/pyvenv/bin/meson: not found

Informations

LE Version: [e.g. 9.2.1] 11.80.1
Hardware Platform: [e.g. RPi3] all

Log file

Additional context

diff for the change to qemu-8.0rc1

diff --git a/packages/tools/qemu/package.mk b/packages/tools/qemu/package.mk
index a248d60313..c74d2dc78c 100644
--- a/packages/tools/qemu/package.mk
+++ b/packages/tools/qemu/package.mk
@@ -2,8 +2,8 @@
 # Copyright (C) 2016-present Team LibreELEC (https://libreelec.tv)
 
 PKG_NAME="qemu"
-PKG_VERSION="8.0.3"
-PKG_SHA256="ecf4d32cbef9d397bfc8cc50e4d1e92a1b30253bf32e8ee73c7a8dcf9a232b09"
+PKG_VERSION="8.1.0-rc1"
+PKG_SHA256="98b032d46d23075a3011222b3642ccb9c327222bd4c67c33945fb9304a1e3f08"
 PKG_LICENSE="GPL"
 PKG_SITE="https://www.qemu.org"
 PKG_URL="https://download.qemu.org/qemu-${PKG_VERSION}.tar.xz"

this patch gets past (3) problem 2. But there must be a better way to do this.

--- a/configure 2023-07-27 09:58:05.337334830 +0000
+++ b/configure 2023-07-27 09:57:58.300630393 +0000
@@ -950,6 +950,8 @@
�����mkvenv_flags="--online"
 fi

+ln -s ${source_path}/../../toolchain/bin/meson pyvenv/bin/meson
+
 if ! $mkvenv ensure \
������$mkvenv_flags \
������--dir "${source_path}/python/wheels" \

The text was updated successfully, but these errors were encountered:

vpeter4 · 2023-07-30T12:58:23Z

What exactly is the problem here and what need to get fixed/investigated?

heitbaum · 2023-07-30T13:32:50Z

I created a draft pr - it does seem to cross compile the .so files of simplejson, pycryptodone and Pillow as aarch64 when using a target of A64.

To experience the issue first hand - cherry pick f9eb28c and try scripts/build qemu:host

the changes to the cross-compile and setuptools/pip may/probably have other unforeseen items to be worked through.

I tried a number of different ways to create a “bootstrap” pyvenv including ensurepip - with limited/no/inconsistent results - and configured qemu to use this….

This draft pr at least is producing consistent results when building

unknown run test results on x86 —> arm/aarch64 or aarch64 —> all.
run results x86 —> x86 seem ok.

If you had a better way - that would be great. Otherwise if this is the right track … would be good to understand why the 85a018611be9a28e87ff5af2e6fd4d4fdbcd50c5 workaround is currently necessary.

there is also the #7361 to be addressed

heitbaum · 2023-07-30T13:44:47Z

Output of .so files (build host is x86_64)

build.LibreELEC-A64.aarch64-12.0-devel/install_pkg$ file */usr/lib/python3.11/site-packages/*/*/*.so */usr/lib/python3.11/site-packages/*/*.so 
pycryptodome-3.18.0/usr/lib/python3.11/site-packages/Crypto/Cipher/_ARC4.abi3.so:            ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
pycryptodome-3.18.0/usr/lib/python3.11/site-packages/Crypto/Cipher/_chacha20.abi3.so:        ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
pycryptodome-3.18.0/usr/lib/python3.11/site-packages/Crypto/Cipher/_pkcs1_decode.abi3.so:    ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
pycryptodome-3.18.0/usr/lib/python3.11/site-packages/Crypto/Cipher/_raw_aes.abi3.so:         ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
pycryptodome-3.18.0/usr/lib/python3.11/site-packages/Crypto/Cipher/_raw_arc2.abi3.so:        ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
pycryptodome-3.18.0/usr/lib/python3.11/site-packages/Crypto/Cipher/_raw_blowfish.abi3.so:    ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
pycryptodome-3.18.0/usr/lib/python3.11/site-packages/Crypto/Cipher/_raw_cast.abi3.so:        ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
pycryptodome-3.18.0/usr/lib/python3.11/site-packages/Crypto/Cipher/_raw_cbc.abi3.so:         ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
pycryptodome-3.18.0/usr/lib/python3.11/site-packages/Crypto/Cipher/_raw_cfb.abi3.so:         ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
pycryptodome-3.18.0/usr/lib/python3.11/site-packages/Crypto/Cipher/_raw_ctr.abi3.so:         ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
pycryptodome-3.18.0/usr/lib/python3.11/site-packages/Crypto/Cipher/_raw_des3.abi3.so:        ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
pycryptodome-3.18.0/usr/lib/python3.11/site-packages/Crypto/Cipher/_raw_des.abi3.so:         ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
pycryptodome-3.18.0/usr/lib/python3.11/site-packages/Crypto/Cipher/_raw_ecb.abi3.so:         ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
pycryptodome-3.18.0/usr/lib/python3.11/site-packages/Crypto/Cipher/_raw_eksblowfish.abi3.so: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
pycryptodome-3.18.0/usr/lib/python3.11/site-packages/Crypto/Cipher/_raw_ocb.abi3.so:         ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
pycryptodome-3.18.0/usr/lib/python3.11/site-packages/Crypto/Cipher/_raw_ofb.abi3.so:         ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
pycryptodome-3.18.0/usr/lib/python3.11/site-packages/Crypto/Cipher/_Salsa20.abi3.so:         ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
pycryptodome-3.18.0/usr/lib/python3.11/site-packages/Crypto/Hash/_BLAKE2b.abi3.so:           ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
pycryptodome-3.18.0/usr/lib/python3.11/site-packages/Crypto/Hash/_BLAKE2s.abi3.so:           ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
pycryptodome-3.18.0/usr/lib/python3.11/site-packages/Crypto/Hash/_ghash_portable.abi3.so:    ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
pycryptodome-3.18.0/usr/lib/python3.11/site-packages/Crypto/Hash/_keccak.abi3.so:            ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
pycryptodome-3.18.0/usr/lib/python3.11/site-packages/Crypto/Hash/_MD2.abi3.so:               ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
pycryptodome-3.18.0/usr/lib/python3.11/site-packages/Crypto/Hash/_MD4.abi3.so:               ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
pycryptodome-3.18.0/usr/lib/python3.11/site-packages/Crypto/Hash/_MD5.abi3.so:               ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
pycryptodome-3.18.0/usr/lib/python3.11/site-packages/Crypto/Hash/_poly1305.abi3.so:          ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
pycryptodome-3.18.0/usr/lib/python3.11/site-packages/Crypto/Hash/_RIPEMD160.abi3.so:         ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
pycryptodome-3.18.0/usr/lib/python3.11/site-packages/Crypto/Hash/_SHA1.abi3.so:              ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
pycryptodome-3.18.0/usr/lib/python3.11/site-packages/Crypto/Hash/_SHA224.abi3.so:            ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
pycryptodome-3.18.0/usr/lib/python3.11/site-packages/Crypto/Hash/_SHA256.abi3.so:            ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
pycryptodome-3.18.0/usr/lib/python3.11/site-packages/Crypto/Hash/_SHA384.abi3.so:            ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
pycryptodome-3.18.0/usr/lib/python3.11/site-packages/Crypto/Hash/_SHA512.abi3.so:            ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
pycryptodome-3.18.0/usr/lib/python3.11/site-packages/Crypto/Math/_modexp.abi3.so:            ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
pycryptodome-3.18.0/usr/lib/python3.11/site-packages/Crypto/Protocol/_scrypt.abi3.so:        ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
pycryptodome-3.18.0/usr/lib/python3.11/site-packages/Crypto/PublicKey/_ec_ws.abi3.so:        ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
pycryptodome-3.18.0/usr/lib/python3.11/site-packages/Crypto/PublicKey/_ed25519.abi3.so:      ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
pycryptodome-3.18.0/usr/lib/python3.11/site-packages/Crypto/PublicKey/_ed448.abi3.so:        ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
pycryptodome-3.18.0/usr/lib/python3.11/site-packages/Crypto/PublicKey/_x25519.abi3.so:       ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
pycryptodome-3.18.0/usr/lib/python3.11/site-packages/Crypto/Util/_cpuid_c.abi3.so:           ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
pycryptodome-3.18.0/usr/lib/python3.11/site-packages/Crypto/Util/_strxor.abi3.so:            ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
Pillow-10.0.0/usr/lib/python3.11/site-packages/PIL/_imagingft.so:                            ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
Pillow-10.0.0/usr/lib/python3.11/site-packages/PIL/_imagingmath.so:                          ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
Pillow-10.0.0/usr/lib/python3.11/site-packages/PIL/_imagingmorph.so:                         ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
Pillow-10.0.0/usr/lib/python3.11/site-packages/PIL/_imaging.so:                              ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
Pillow-10.0.0/usr/lib/python3.11/site-packages/PIL/_imagingtk.so:                            ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped
simplejson-3.19.1/usr/lib/python3.11/site-packages/simplejson/_speedups.so:                  ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, stripped

vpeter4 · 2023-07-30T15:54:10Z

For those 3 python calls in a row: if you remove patch file https://github.com/LibreELEC/LibreELEC.tv/blob/aec466da8aaf5f79b8f18b8474d0a6e6fbaf5127/packages/lang/Python3/patches/Python3-0300-generate-legacy-pyc-bytecode.patch then python:host builds with just one call.
Tip comes from here: https://bugs.python.org/issue42446
Not sure if this is correct solution or not. Maybe patch is needed only for target?

heitbaum · 2023-07-30T19:33:17Z

For those 3 python calls in a row: if you remove patch file https://github.com/LibreELEC/LibreELEC.tv/blob/aec466da8aaf5f79b8f18b8474d0a6e6fbaf5127/packages/lang/Python3/patches/Python3-0300-generate-legacy-pyc-bytecode.patch then python:host builds with just one call. Tip comes from here: https://bugs.python.org/issue42446 Not sure if this is correct solution or not. Maybe patch is needed only for target?

d8ddcfad154d453dca2c30554b49c4904e9a47ae does indeed allow the successful compile. The does it work right is still the ultimate question :-)

vpeter4 · 2023-07-31T11:18:13Z

The does it work right is still the ultimate question :-)

Looks like not because at least my Generic legacy image doesn't really work with pr 8016.

heitbaum · 2023-08-01T12:07:31Z

The does it work right is still the ultimate question :-)

Looks like not because at least my Generic legacy image doesn't really work with pr 8016.

Backing out the move of the bytecode patch to target only and reintroduction of ensurepip * 3 returns my LE back to a working state.

Looks like we need to fix the builds of the target pyc (by the host Python3) to match the required bytecode in target + drop the patch in the host build - https://peps.python.org/pep-3147/#case-4-legacy-pyc-files-and-source-less-imports

heitbaum · 2023-08-01T13:35:12Z

Putting a ENV controlled use of the patch seems to have addressed this - 8d9376694898b699a2bb34b8cca54291399c49c3 - I’m now using a cleanly built LE with the updated 8016 on Generic nuc12.

should the packages/python/devel builds use export DONT_BUILD_LEGACY_PYC=1?
should export DONT_BUILD_LEGACY_PYC=1 be the default for all :host builds?
using the modified patch (and environment variable) returns Python3 back its standard pyc behaviour - which is probably more correct when not targeting the embedded device
run logs look good

heitbaum · 2023-08-27T13:12:53Z

Pr complete. Closing

heitbaum · 2023-09-04T11:57:51Z

PR was reverted Partial revert: Python cross-compile changes and qemu #8102

heitbaum added UPDATE PACKAGE TOOLCHAIN LE 12.0 HELP WANTED labels Jul 27, 2023

heitbaum mentioned this issue Jul 30, 2023

Python cross-compile changes and qemu #8016

Merged

heitbaum closed this as completed Aug 27, 2023

heitbaum linked a pull request Aug 27, 2023 that will close this issue

Python cross-compile changes and qemu #8016

Merged

heitbaum reopened this Sep 4, 2023

heitbaum added LE 13.0 and removed LE 12.0 labels Jan 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Help Required] qemu 8.1-rc1 requires Python venv and pip #8009

[Help Required] qemu 8.1-rc1 requires Python venv and pip #8009

heitbaum commented Jul 27, 2023 •

edited

vpeter4 commented Jul 30, 2023

heitbaum commented Jul 30, 2023 •

edited

heitbaum commented Jul 30, 2023

vpeter4 commented Jul 30, 2023 •

edited

heitbaum commented Jul 30, 2023

vpeter4 commented Jul 31, 2023

heitbaum commented Aug 1, 2023

heitbaum commented Aug 1, 2023

heitbaum commented Aug 27, 2023

heitbaum commented Sep 4, 2023

[Help Required] qemu 8.1-rc1 requires Python venv and pip #8009

[Help Required] qemu 8.1-rc1 requires Python venv and pip #8009

Comments

heitbaum commented Jul 27, 2023 • edited

Describe the bug

To Reproduce

Informations

Log file

Additional context

vpeter4 commented Jul 30, 2023

heitbaum commented Jul 30, 2023 • edited

heitbaum commented Jul 30, 2023

vpeter4 commented Jul 30, 2023 • edited

heitbaum commented Jul 30, 2023

vpeter4 commented Jul 31, 2023

heitbaum commented Aug 1, 2023

heitbaum commented Aug 1, 2023

heitbaum commented Aug 27, 2023

heitbaum commented Sep 4, 2023

heitbaum commented Jul 27, 2023 •

edited

heitbaum commented Jul 30, 2023 •

edited

vpeter4 commented Jul 30, 2023 •

edited