tables 3.9.2 problem on macos CI #1093
Comments
marcofavoritobi
added a commit
to bancaditalia/black-it
that referenced
this issue
Dec 7, 2023
This change was needed because of a nondeterminism in how the macos-12 (=macos-latest at the time of writing) runner image was instantiated by GitHub Actions. In particular, for runner version 2.311.0, the operating system was nondeterministically chosen between 12.6.9 and 12.7.1, as it can be seen in these two GH runs: - https://github.com/bancaditalia/black-it/actions/runs/7128997647/job/19412070642 - https://github.com/bancaditalia/black-it/actions/runs/7128997647/job/19412070868 This is a known issue in the community: see actions/runner-images#8642 Furthermore, at the time of writing, it is suspected that Python code that uses pytables is not stable over macos at version 12.7.1 (but this problem does not occur on v12.6.9!), as witnessed by this issue: PyTables/PyTables#1093 What happened in our CI workflow is that tests resulted in a flaky outcome, but not because of our tests, but due to the nondeterminism in how the macos runner was instantiated by GitHub Actions. This was not our unique attempt. In fact, we first tried to improve the preliminary configuration steps for the testing environment, as follows: - if: matrix.os == 'macos-latest' name: Install dependencies (macos-latest) run: | # for some reason this seems to be a necessary step for MacOS images, but not for Ubuntu and Windows brew install hdf5 brew install c-blosc # https://stackoverflow.com/questions/73029883/could-not-find-hdf5-installation-for-pytables-on-m1-mac echo "HDF5_DIR=/usr/local/" >> $GITHUB_ENV echo "BLOSC_DIR=/usr/local/" >> $GITHUB_ENV This attempt consisted in installing HDF5 and C-blosc explicitly in the preliminary steps on CI for MacOS images. By the way, this step was part of the installation instructions of PyTables v3.9.2: https://github.com/PyTables/PyTables/blob/v3.9.2/README.rst#installation , although it seems this step is not actually needed. Moreover, as explained here: https://stackoverflow.com/questions/73029883/could-not-find-hdf5-installation-for-pytables-on-m1-mac, sometimes must be explicitly set the environment path variables where HDF5 ahd C-blosc are installed For some reason this is not needed on Ubuntu and Windows runners. However, despite our efforts in this direction, the issue with pytables persisted. From the logs of the CI: (https://github.com/bancaditalia/black-it/actions/runs/7121188681/job/19389914846?pr=78#step:5:62) ``` Fatal Python error: Illegal instruction Current thread 0x000000011783c600 (most recent call first): File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/tables/node.py", line 246 in __init__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/tables/leaf.py", line 264 in __init__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/tables/carray.py", line 200 in __init__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/tables/earray.py", line 143 in __init__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/tables/file.py", line 1345 in create_earray File "/Users/runner/work/black-it/black-it/black_it/utils/json_pandas_checkpointing.py", line 211 in save_calibrator_state File "/Users/runner/work/black-it/black-it/tests/test_utils/test_pandas_json_checkpointing.py", line 58 in test_save_and_load_calibrator_state File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/python.py", line 194 in pytest_pyfunc_call File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/python.py", line 1792 in runtest File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/runner.py", line 262 in <lambda> File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/runner.py", line 341 in from_call File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/runner.py", line 261 in call_runtest_hook File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/runner.py", line 222 in call_and_report File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/runner.py", line 133 in runtestprotocol File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/main.py", line 350 in pytest_runtestloop File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/main.py", line 325 in _main File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/main.py", line 271 in wrap_session File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/main.py", line 318 in pytest_cmdline_main File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/config/__init__.py", line 169 in main File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/config/__init__.py", line 192 in console_main File "/Users/runner/work/black-it/black-it/.tox/py310/bin/pytest", line 8 in <module> ``` Therefore, the current workaround we adopted was to downgrade the version of the macos runner to the previous version, 11.
marcofavoritobi
added a commit
to bancaditalia/black-it
that referenced
this issue
Dec 7, 2023
This change was needed because of a nondeterminism in how the macos-12 (=macos-latest at the time of writing) runner image was instantiated by GitHub Actions. In particular, for runner version 2.311.0, the operating system was nondeterministically chosen between 12.6.9 and 12.7.1, as it can be seen in these two GH runs: - https://github.com/bancaditalia/black-it/actions/runs/7128997647/job/19412070642 - https://github.com/bancaditalia/black-it/actions/runs/7128997647/job/19412070868 This is a known issue in the community: see actions/runner-images#8642 Furthermore, at the time of writing, it is suspected that Python code that uses pytables is not stable over macos at version 12.7.1 (but this problem does not occur on v12.6.9!), as witnessed by this issue: PyTables/PyTables#1093 What happened in our CI workflow is that tests resulted in a flaky outcome, but not because of our tests, but due to the nondeterminism in how the macos runner was instantiated by GitHub Actions. This was not our unique attempt. In fact, we first tried to improve the preliminary configuration steps for the testing environment, as follows: - if: matrix.os == 'macos-latest' name: Install dependencies (macos-latest) run: | # for some reason this seems to be a necessary step for MacOS images, but not for Ubuntu and Windows brew install hdf5 brew install c-blosc # https://stackoverflow.com/questions/73029883/could-not-find-hdf5-installation-for-pytables-on-m1-mac echo "HDF5_DIR=/usr/local/" >> $GITHUB_ENV echo "BLOSC_DIR=/usr/local/" >> $GITHUB_ENV This attempt consisted in installing HDF5 and C-blosc explicitly in the preliminary steps on CI for MacOS images. By the way, this step was part of the installation instructions of PyTables v3.9.2: https://github.com/PyTables/PyTables/blob/v3.9.2/README.rst#installation , although it seems this step is not actually needed. Moreover, as explained here: https://stackoverflow.com/questions/73029883/could-not-find-hdf5-installation-for-pytables-on-m1-mac, sometimes must be explicitly set the environment path variables where HDF5 ahd C-blosc are installed For some reason this is not needed on Ubuntu and Windows runners. However, despite our efforts in this direction, the issue with pytables persisted. From the logs of the CI: (https://github.com/bancaditalia/black-it/actions/runs/7121188681/job/19389914846?pr=78#step:5:62) ``` Fatal Python error: Illegal instruction Current thread 0x000000011783c600 (most recent call first): File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/tables/node.py", line 246 in __init__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/tables/leaf.py", line 264 in __init__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/tables/carray.py", line 200 in __init__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/tables/earray.py", line 143 in __init__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/tables/file.py", line 1345 in create_earray File "/Users/runner/work/black-it/black-it/black_it/utils/json_pandas_checkpointing.py", line 211 in save_calibrator_state File "/Users/runner/work/black-it/black-it/tests/test_utils/test_pandas_json_checkpointing.py", line 58 in test_save_and_load_calibrator_state File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/python.py", line 194 in pytest_pyfunc_call File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/python.py", line 1792 in runtest File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/runner.py", line 262 in <lambda> File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/runner.py", line 341 in from_call File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/runner.py", line 261 in call_runtest_hook File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/runner.py", line 222 in call_and_report File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/runner.py", line 133 in runtestprotocol File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/main.py", line 350 in pytest_runtestloop File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/main.py", line 325 in _main File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/main.py", line 271 in wrap_session File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/main.py", line 318 in pytest_cmdline_main File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/config/__init__.py", line 169 in main File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/config/__init__.py", line 192 in console_main File "/Users/runner/work/black-it/black-it/.tox/py310/bin/pytest", line 8 in <module> ``` Therefore, the current workaround we adopted was to downgrade the version of the macos runner to the previous version, 11.
marcofavoritobi
added a commit
to bancaditalia/black-it
that referenced
this issue
Dec 7, 2023
This change was needed because of a nondeterminism in how the macos-12 (=macos-latest at the time of writing) runner image was instantiated by GitHub Actions. In particular, for runner version 2.311.0, the operating system was nondeterministically chosen between 12.6.9 and 12.7.1, as it can be seen in these two GH runs: - https://github.com/bancaditalia/black-it/actions/runs/7128997647/job/19412070642 - https://github.com/bancaditalia/black-it/actions/runs/7128997647/job/19412070868 This is a known issue in the community: see actions/runner-images#8642 Furthermore, at the time of writing, it is suspected that Python code that uses pytables is not stable over macos at version 12.7.1 (but this problem does not occur on v12.6.9!), as witnessed by this issue: PyTables/PyTables#1093 What happened in our CI workflow is that tests resulted in a flaky outcome, but not because of our tests, but due to the nondeterminism in how the macos runner was instantiated by GitHub Actions. This was not our unique attempt. In fact, we first tried to improve the preliminary configuration steps for the testing environment, as follows: - if: matrix.os == 'macos-latest' name: Install dependencies (macos-latest) run: | # for some reason this seems to be a necessary step for MacOS images, but not for Ubuntu and Windows brew install hdf5 brew install c-blosc # https://stackoverflow.com/questions/73029883/could-not-find-hdf5-installation-for-pytables-on-m1-mac echo "HDF5_DIR=/usr/local/" >> $GITHUB_ENV echo "BLOSC_DIR=/usr/local/" >> $GITHUB_ENV This attempt consisted in installing HDF5 and C-blosc explicitly in the preliminary steps on CI for MacOS images. By the way, this step was part of the installation instructions of PyTables v3.9.2: https://github.com/PyTables/PyTables/blob/v3.9.2/README.rst#installation , although it seems this step is not actually needed. Moreover, as explained here: https://stackoverflow.com/questions/73029883/could-not-find-hdf5-installation-for-pytables-on-m1-mac, sometimes must be explicitly set the environment path variables where HDF5 ahd C-blosc are installed For some reason this is not needed on Ubuntu and Windows runners. However, despite our efforts in this direction, the issue with pytables persisted. From the logs of the CI: (https://github.com/bancaditalia/black-it/actions/runs/7121188681/job/19389914846?pr=78#step:5:62) ``` Fatal Python error: Illegal instruction Current thread 0x000000011783c600 (most recent call first): File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/tables/node.py", line 246 in __init__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/tables/leaf.py", line 264 in __init__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/tables/carray.py", line 200 in __init__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/tables/earray.py", line 143 in __init__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/tables/file.py", line 1345 in create_earray File "/Users/runner/work/black-it/black-it/black_it/utils/json_pandas_checkpointing.py", line 211 in save_calibrator_state File "/Users/runner/work/black-it/black-it/tests/test_utils/test_pandas_json_checkpointing.py", line 58 in test_save_and_load_calibrator_state File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/python.py", line 194 in pytest_pyfunc_call File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/python.py", line 1792 in runtest File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/runner.py", line 262 in <lambda> File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/runner.py", line 341 in from_call File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/runner.py", line 261 in call_runtest_hook File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/runner.py", line 222 in call_and_report File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/runner.py", line 133 in runtestprotocol File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/main.py", line 350 in pytest_runtestloop File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/main.py", line 325 in _main File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/main.py", line 271 in wrap_session File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/main.py", line 318 in pytest_cmdline_main File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/config/__init__.py", line 169 in main File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/config/__init__.py", line 192 in console_main File "/Users/runner/work/black-it/black-it/.tox/py310/bin/pytest", line 8 in <module> ``` Therefore, the current workaround we adopted was to downgrade the version of the macos runner to the previous version, 11.
sanssecours
added a commit
to MyTooliT/ICOlyzer
that referenced
this issue
Dec 12, 2023
See also: - #5 - PyTables/PyTables#1093
muxator
pushed a commit
to bancaditalia/black-it
that referenced
this issue
Feb 23, 2024
This change was needed because of a nondeterminism in how the macos-12 (=macos-latest at the time of writing) runner image was instantiated by GitHub Actions. In particular, for runner version 2.311.0, the operating system was nondeterministically chosen between 12.6.9 and 12.7.1, as it can be seen in these two GH runs: - https://github.com/bancaditalia/black-it/actions/runs/7128997647/job/19412070642 - https://github.com/bancaditalia/black-it/actions/runs/7128997647/job/19412070868 This is a known issue in the community: see actions/runner-images#8642 Furthermore, at the time of writing, it is suspected that Python code that uses pytables is not stable over macos at version 12.7.1 (but this problem does not occur on v12.6.9!), as witnessed by this issue: PyTables/PyTables#1093 What happened in our CI workflow is that tests resulted in a flaky outcome, but not because of our tests, but due to the nondeterminism in how the macos runner was instantiated by GitHub Actions. This was not our unique attempt. In fact, we first tried to improve the preliminary configuration steps for the testing environment, as follows: - if: matrix.os == 'macos-latest' name: Install dependencies (macos-latest) run: | # for some reason this seems to be a necessary step for MacOS images, but not for Ubuntu and Windows brew install hdf5 brew install c-blosc # https://stackoverflow.com/questions/73029883/could-not-find-hdf5-installation-for-pytables-on-m1-mac echo "HDF5_DIR=/usr/local/" >> $GITHUB_ENV echo "BLOSC_DIR=/usr/local/" >> $GITHUB_ENV This attempt consisted in installing HDF5 and C-blosc explicitly in the preliminary steps on CI for MacOS images. By the way, this step was part of the installation instructions of PyTables v3.9.2: https://github.com/PyTables/PyTables/blob/v3.9.2/README.rst#installation , although it seems this step is not actually needed. Moreover, as explained here: https://stackoverflow.com/questions/73029883/could-not-find-hdf5-installation-for-pytables-on-m1-mac, sometimes must be explicitly set the environment path variables where HDF5 ahd C-blosc are installed For some reason this is not needed on Ubuntu and Windows runners. However, despite our efforts in this direction, the issue with pytables persisted. From the logs of the CI: (https://github.com/bancaditalia/black-it/actions/runs/7121188681/job/19389914846?pr=78#step:5:62) ``` Fatal Python error: Illegal instruction Current thread 0x000000011783c600 (most recent call first): File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/tables/node.py", line 246 in __init__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/tables/leaf.py", line 264 in __init__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/tables/carray.py", line 200 in __init__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/tables/earray.py", line 143 in __init__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/tables/file.py", line 1345 in create_earray File "/Users/runner/work/black-it/black-it/black_it/utils/json_pandas_checkpointing.py", line 211 in save_calibrator_state File "/Users/runner/work/black-it/black-it/tests/test_utils/test_pandas_json_checkpointing.py", line 58 in test_save_and_load_calibrator_state File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/python.py", line 194 in pytest_pyfunc_call File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/python.py", line 1792 in runtest File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/runner.py", line 262 in <lambda> File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/runner.py", line 341 in from_call File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/runner.py", line 261 in call_runtest_hook File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/runner.py", line 222 in call_and_report File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/runner.py", line 133 in runtestprotocol File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/main.py", line 350 in pytest_runtestloop File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/main.py", line 325 in _main File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/main.py", line 271 in wrap_session File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/main.py", line 318 in pytest_cmdline_main File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__ File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/config/__init__.py", line 169 in main File "/Users/runner/work/black-it/black-it/.tox/py310/lib/python3.10/site-packages/_pytest/config/__init__.py", line 192 in console_main File "/Users/runner/work/black-it/black-it/.tox/py310/bin/pytest", line 8 in <module> ``` Therefore, the current workaround we adopted was to downgrade the version of the macos runner to the previous version, 11.
juarezr
added a commit
to juarezr/petl
that referenced
this issue
Mar 14, 2024
juarezr
added a commit
to petl-developers/petl
that referenced
this issue
Mar 14, 2024
7 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We're encountering a very strange issue in github Action runs on macos ... one i've not "really" been able to isolate further for now (so excuse the lacking information)
Due to a problem in the Github Actions infrasturcture, the "macos-latest" is not providing "stable" images.
As such, a run might get macos 12.7.1, while the next run of the same CI pipeline get's 12.6.9.
Now with 3.9.2 - i've observed a pytest failure of
Fatal Python error: Illegal instruction, followed by a rather long stacktrace (the most relevant section is the following:(ending with
Illegal instruction: 4).(examples of this can be found here: https://github.com/freqtrade/freqtrade/actions/runs/7090859904/job/19316103127).
Now unfortunately, it's only happening in CI, and i can't really reproduce this locally (no macos ...).
I've created a test run here https://github.com/xmatthias/freqtrade/actions/runs/7096962713/job/19334129858 - which i was able to ssh into - and was able to drill down the error to tests that do hdf5 loading.
these affected tests work fine with 3.9.1 (on both MacOS runner versions) - but fail on 12.7.1 with 3.9.2 (I've manually updated and downgraded tables within that CI environment to test this).
(a screenshot trying to capture this behavior, first 12 tests are with 3.9.1, then i update tables, and run the same tests again)
The CI runner version that's impacting this can be found here in the github action runs
Fails: Macos Runner 12.7.1
works: MacOS Runner 12.6.9
Unfortunately, i'm unable to provide too much additional information - also because i lack a proper test-system where i don't have to rely on CI which times out after X minutes to actually test this.
The text was updated successfully, but these errors were encountered: