V4.5 possible regression: "version `GLIBC_2.35' not found" on Debian Bullseye (Stable/11) (ESPTOOL-623)

[jaimet]

Operating System

Debian Bullseye (Stable/11)

Esptool Version

4.5

Python Version

Python 3.9.2

Full Esptool Command Line that Was Run

./esptool version

Esptool Output

[6359] Error loading Python lib '/tmp/_MEIWcpyFV/libpython3.8.so.1.0': dlopen: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.35' not found (required by /tmp/_MEIWcpyFV/libpython3.8.so.1.0)

What is the Expected Behaviour?

The previous version (4.4) on the same machine outputs:

esptool.py v4.4
4.4

I expected something "similar".

More Information

I have been able to reproduce this on other Debain Bullseye machines,
so I do not think that it is a problem with this particular (linux) host.

I have searched esptool's github repo for references to glibc but I have not found anything.

Unfortunately my python packaging-fu is not great, so I don't know where to go from here.

Other Steps to Reproduce

No response

[fhrbata]

Hello @jaimet ,

yes this looks like a packaging problem. The problem is that the libpython was linked with glibc 2.35 on ubuntu-22.04, but the code is run on debian bullseye with glibc 2.31. Glibc of course does not guarantee backward compatibility, so the dynamic loader fails because new symbol versions in 2.35 are not available in 2.31. The problem is that esptool github build workflow uses ubuntu-latest(22.04) runner and pyinstaller does not pack glibc libraries.

This is easily reproducible

1. docker run -it --rm python:3.8-bullseye bash
2. wget https://github.com/espressif/esptool/releases/download/v4.5/esptool-v4.5-linux-amd64.zip
3. unzip esptool-v4.5-linux-amd64.zip
4. esptool-v4.5-linux-amd64/esptool

[12] Error loading Python lib '/tmp/_MEI265mQ2/libpython3.8.so.1.0': dlopen: /lib/x86_64-linux-gnu/libm.so.6: version 'GLIBC_2.35' not found (required by /tmp/_MEI265mQ2/libpython3.8.so.1.0)

One solution would be to somehow force the pyinstaller to pack also libc libraries for esptool, but I'm not sure if this could be easily done. Other possibility is to use some older ubuntu runner and make it clear that the pyinstaller one-file package requires at least a specific glibc version installed on the system where it's used.

There is a github ubuntu-20.04 runner which has glibc 2.31 and using it seems to solve this problem. At least for the current debian stable distro.

I quickly tried this in my esptool fork with the current version on master branch and the generated pyinstaller package is working for me on debian 11

root@82bef8e11ffe:/# cat /etc/debian_version 
11.6
root@82bef8e11ffe:/# ./esptool version
esptool.py v4.6-dev
4.6-dev

The following patch should hopefully fix this problem, but maybe there is a better solution.

diff --git a/.github/workflows/build_esptool.yml b/.github/workflows/build_esptool.yml
index 380732915f2c..ca4d7eaca49d 100644
--- a/.github/workflows/build_esptool.yml
+++ b/.github/workflows/build_esptool.yml
@@ -8,12 +8,12 @@ jobs:
     runs-on: ${{ matrix.os }}
     strategy:
       matrix:
-        os: [macos-latest, ubuntu-latest, windows-latest]
+        os: [macos-latest, ubuntu-20.04, windows-latest]
         include:
         - os: macos-latest
           TARGET: macos
           SEPARATOR: ':'
-        - os: ubuntu-latest
+        - os: ubuntu-20.04
           TARGET: linux-amd64
           SEPARATOR: ':'
         - os: windows-latest

Thank you very much for reporting this.

[jaimet]

Hello @jaimet ,

@fhrbata Hi Frantisek

Other possibility is to use some older ubuntu runner ...

I read the documentation that you linked to, and I noticed the phrase:

The solution is to always build your app on the oldest version of GNU/Linux you mean to support.

but I don't know whether the "decision makers" want to support Debian stable or not. If they do then your patch seems to be the correct solution.

However, I do not know if there is a specific reason why the currently-chosen linux runner is ubuntu-latest (as opposed to an earlier version).

I quickly tried this in my esptool fork with the current version on master branch and the generated pyinstaller package is working for me on debian 11

Your package works for me too. 👍

Thank you very much for reporting this.

Thank you for finding the cause and a possible solution.

[fhrbata]

Hello @jaimet ,

thank you for such a quick response and confirmation, it's very much appreciated. Let's wait and see what others think . Personally I cannot think of any problem that this approach could cause, but I for sure may be missing something.

Again thank you very much!

[iscooter]

I read the documentation that you linked to, and I noticed the phrase:

The solution is to always build your app on the oldest version of GNU/Linux you mean to support.

Once it clicked that GNU Linux libs were backwards compatible only...this issue was fairly issue to resolve. I needed a binary (pyinstaller) to work on ubuntu 22, amzn2 and amzn2023 - so i built with glibc225 on amzn2 and it worked on all versions.