Skip to content

Now that I enabled c-ares tests on all Linux builds, there is one get… #496

Now that I enabled c-ares tests on all Linux builds, there is one get…

Now that I enabled c-ares tests on all Linux builds, there is one get… #496

Workflow file for this run

###
# Initially copied from
# https://github.com/actions/starter-workflows/blob/main/ci/python-package.yml
#
# Original comment follows.
###
###
# This workflow will install Python dependencies, run tests and lint with a variety of Python versions
# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions
###
###
# Important notes on GitHub actions:
#
# - We only get 2,000 free minutes a month
# - We only get 500MB of artifact storage
# - Cache storage is limited to 7 days and 5GB.
# - macOS minutes are 10x as expensive as Linux minutes
# - windows minutes are twice as expensive.
#
# So keep those workflows light.
#
# In December 2020, github only supports x86/64. If we wanted to test
# gevent on other architectures, we might be able to use docker
# emulation, but there's no native support.
#
# Another major downside: You can't just re-run the job for one part
# of the matrix. So if there's a transient test failure that hit, say, 3.8,
# to get a clean run every version of Python runs again. That's bad.
# https://github.community/t/ability-to-rerun-just-a-single-job-in-a-workflow/17234/65
name: gevent testing
# Triggers the workflow on push or pull request events
on: [push, pull_request]
# Limiting to particular branches might be helpful to conserve minutes.
#on:
# push:
# branches: [ $default-branch ]
# pull_request:
# branches: [ $default-branch ]
env:
# Weirdly, this has to be a top-level key, not ``defaults.env``
PYTHONHASHSEED: 8675309
PYTHONUNBUFFERED: 1
PYTHONDONTWRITEBYTECODE: 1
PIP_UPGRADE_STRATEGY: eager
# Don't get warnings about Python 2 support being deprecated. We
# know. The env var works for pip 20.
PIP_NO_PYTHON_VERSION_WARNING: 1
PIP_NO_WARN_SCRIPT_LOCATION: 1
GEVENTSETUP_EV_VERIFY: 1
# Disable some warnings produced by libev especially and also some Cython generated code.
# These are shared between GCC and clang so it must be a minimal set.
# TODO: Figure out how to set env vars per platform without resorting to inline scripting.
# Note that changing the value of these variables invalidates configure caches
CFLAGS: -O3 -pipe -Wno-strict-aliasing -Wno-comment
CPPFLAGS: -DEV_VERIFY=1
# Uploading built wheels for releases.
# TWINE_PASSWORD is encrypted and stored directly in the
# travis repo settings.
TWINE_USERNAME: __token__
###
# caching
###
# CCACHE_DIR: ~/.ccache # Using ~ here makes it not find its cache.
CC: "ccache gcc"
CCACHE_NOCPP2: true
CCACHE_SLOPPINESS: file_macro,time_macros,include_file_ctime,include_file_mtime
CCACHE_NOHASHDIR: true
#
jobs:
test:
runs-on: ${{ matrix.os }}
strategy:
matrix:
# 3.10+ needs more work: dnspython for example doesn't work
# with it. That means for the bulk of our testing we need to
# stick to 3.9.
#
# PyPy 7.3.13 started crashing for unknown reasons. 7.3.15
# still crashes. The crash is somewhere in
# ``gevent.tests.test__queue gevent.tests.test__real_greenlet
# gevent.tests.test__refcount_core
# gevent.tests.test__resolver_dnspython``
#
# CAREFUL: Some of the tests are only run on specific versions of Python,
# as dictated by the conditions found below. So when you change a version,
# for example to force a specific patch release, don't forget to change conditions!
# XXX: We could probably make this easier on ourself by adding a specific
# key to the matrix in the version we care about and checking for that matrix key.
python-version: ["3.12.2", "pypy-3.10-v7.3.12", 3.8, 3.9, '3.10', '3.11.8']
os: [macos-latest, ubuntu-latest]
exclude:
# The bulk of the testing is on Linux and Windows (appveyor).
# Experience shows that it's sufficient to only test the latest
# version on macOS. However, that does mean you need to
# manually upload macOS wheels for those versions.
# - os: macos-latest
# python-version: 3.8
# - os: macos-latest
# python-version: 3.9
# - os: macos-latest
# python-version: 3.10
- os: macos-latest
python-version: "pypy-3.10-v7.3.15"
steps:
- name: checkout
uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: setup.py
- name: Install ccache (ubuntu)
if: startsWith(runner.os, 'Linux')
run: |
sudo apt-get install -y ccache sed gcc
echo CCACHE_DIR=$HOME/.ccache >>$GITHUB_ENV
mkdir -p $HOME/.ccache
- name: Install ccache (macos)
if: startsWith(runner.os, 'macOS')
run: |
brew install ccache
echo CFLAGS=$CFLAGS -Wno-parentheses-equality >>$GITHUB_ENV
echo CCACHE_DIR=$HOME/.ccache >>$GITHUB_ENV
mkdir -p $HOME/.ccache
- name: Set coverage status
# coverage is too slow on PyPy. We can't submit it from macOS (see that action),
# so don't bother taking the speed hit there either.
# Coverage can't run the test_interpreters.py tests because greenlet can't
# run there and that's how coverage is configured. Right now, we only have that
# on Python 3.12, so take the quick way out and nix that version too.
# Remember this condition needs to be synced with the coveralls/report step.
if: ${{ !startsWith(matrix.python-version, 'pypy') && !startsWith(matrix.python-version, '3.12') && startsWith(runner.os, 'Linux') }}
run: |
echo G_USE_COV=--coverage >> $GITHUB_ENV
###
# Caching.
# This actually *restores* a cache and schedules a cleanup action
# to save the cache. So it must come before the thing we want to use
# the cache.
###
- name: Cache ~/.ccache
uses: actions/cache@v4
# This is repeated in an explicit save always step below
# because normally it won't save anything if there's a cache hit!
# Which is silly, because things in the cache might have (will have)
# been changed.
with:
path: ~/.ccache/**
key: ${{ runner.os }}-ccache2-${{ matrix.python-version }}
- name: Cache config.cache
# Store the configure caches. Having a cache can speed up c-ares
# configure from 2-3 minutes to 20 seconds.
uses: actions/cache@v4
with:
path: deps/*/config.cache
# XXX: This should probably include a hash of each configure
# script (which is possible with hashFiles()). We don't have a restore-keys that doesn't include
# the CFLAGS becouse the scripts fail to run if they get
# different CFLAGS, CC, CPPFLAGS, etc, and GHA offers no way
# to manually clear the cache. At one time, we had a
# restore-key configured, and it still seems to be used even
# without that setting here. The whole thing is being
# matched even without the CFLAGS matching. Perhaps the - is
# a generic search separator?
key: ${{ runner.os }}-${{ matrix.os }}-configcache3-${{ matrix.python-version }}-${{ env.CFLAGS }}
# Install gevent. Yes, this will create different files each time,
# leading to a fresh cache. But because of CCache stats, we had already been doing
# that (before we learned about CCACHE_NOSTATS).
# We don't install using the requirements file for speed (reduced deps) and because an editable
# install doesn't work in the cache.
# First, the build dependencies (see setup.cfg)
# so that we don't have to use build isolation and can better use the cache;
# Note that we can't use -U for cffi and greenlet on PyPy.
# The -q is because Pypy-2.7 sometimes started raising
# UnicodeEncodeError: 'ascii' codec can't encode character u'\u2588' in position 6: ordinal not in range(128)
# when downloading files. This started sometime in mid 2020. It's from
# pip's vendored progress.bar class.
- name: Install dependencies
run: |
pip install -U pip
pip install -U -q setuptools wheel twine
pip install -q -U 'cffi;platform_python_implementation=="CPython"'
pip install -q -U 'cython>=3.0.2'
# Use a debug version of greenlet to help catch any errors earlier.
CFLAGS="$CFLAGS -Og -g -UNDEBUG" pip install -v --no-binary :all: 'greenlet>=2.0.0;platform_python_implementation=="CPython" and python_version < "3.11"'
CFLAGS="$CFLAGS -Og -g -UNDEBUG" pip install -v --no-binary :all: 'greenlet>=3.0rc3;platform_python_implementation=="CPython" and python_version >= "3.11"'
- name: Build gevent (non-Mac)
if: ${{ ! startsWith(runner.os, 'Mac') }}
run: |
# Next, build the wheel *in place*. This helps ccache, and also lets us cache the configure
# output (pip install uses a random temporary directory, making this difficult)
python setup.py build_ext -i
python setup.py bdist_wheel
env:
# Ensure we test with assertions enabled.
# As opposed to the manylinux builds, which we distribute and
# thus only use O3 (because Ofast enables fast-math, which has
# process-wide effects), we test with Ofast here, because we
# expect that some people will compile it themselves with that setting.
CPPFLAGS: "-Ofast -UNDEBUG"
- name: Build gevent (Mac)
if: startsWith(runner.os, 'Mac')
run: |
# Next, build the wheel *in place*. This helps ccache, and also lets us cache the configure
# output (pip install uses a random temporary directory, making this difficult)
python setup.py build_ext -i
python setup.py bdist_wheel
# Something in the build system isn't detecting that we're building for both,
# so we're getting tagged with just x86_64. Force the universal2 tag.
# (I've verified that the .so files are in fact universal, with both architectures.)
echo 'Done building'
ls -l dist
# (wheel tags --abi-tag universal2 dist/*x86_64.whl && ((rm dist/*universal2*universal2.whl || rm dist/*universal2*x86_86.whl) || rm dist/*x86_64.whl)) || true
# XXX: That can produce invalid filenames, for some reason. 3.11 came up with
# gevent-23.7.1.dev0-cp311-universal2-macosx_10_9_universal2.whl, which is not valid.
# gevent-23.9.1.dev0-cp38-universal2-macosx_11_0_x86_64.whl has also shown up.
# It's not clear why, because greenlet didn't do that. Maybe because it was already universal?
# So we attempt to only do this for non-universal wheels.
ls -l dist
env:
# Unlike the above, we are actually distributing these
# wheels, so they need to be built for production use.
CPPFLAGS: "-O3"
# Build for both architectures
ARCHFLAGS: "-arch x86_64 -arch arm64"
# Force the wheel tag.
_PYTHON_HOST_PLATFORM: "macosx-11.0-universal2"
- name: Check gevent build
run: |
ls -l dist
twine check dist/*whl
- name: Cache ~/.ccache
uses: actions/cache/save@v4
if: always()
with:
path: ~/.ccache/**
key: ${{ runner.os }}-ccache2-${{ matrix.python-version }}
- name: Upload gevent wheel
uses: actions/upload-artifact@v4
with:
name: gevent-${{ runner.os }}-${{ matrix.python-version }}.whl
path: dist/*whl
- name: Publish package to PyPI (mac)
# We cannot 'uses: pypa/gh-action-pypi-publish@v1.4.1' because
# that's apparently a container action, and those don't run on
# the Mac.
if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags') && startsWith(runner.os, 'Mac')
env:
TWINE_PASSWORD: ${{ secrets.TWINE_PASSWORD }}
run: |
twine upload --skip-existing dist/*
- name: Install gevent
run: |
WHL=$(ls dist/*whl)
pip install -U "$WHL[test]"
- name: Report environment details
run: |
python --version
python -c 'import greenlet; print(greenlet, greenlet.__version__)'
python -c 'import gevent; print(gevent.__version__)'
python -c 'from gevent._compat import get_clock_info; print(get_clock_info("perf_counter"))'
python -c 'import gevent.core; print(gevent.core.loop)'
python -c 'import gevent.ares; print(gevent.ares)'
echo CCache stats
ccache --version
ccache -s -v
- name: "Tests: Basic"
run: |
python -m gevent.tests --second-chance $G_USE_COV
# For the CPython interpreters, unless we have reason to expect
# different behaviour across the versions (e.g., as measured by coverage)
# it's sufficient to run the full suite on the current version
# and oldest version.
- name: "Tests: subproccess and FileObjectThread"
if: startsWith(runner.os, 'Linux') || (startsWith(runner.os, 'Mac') && matrix.python-version == '3.12.2')
# Now, the non-default threaded file object.
# In the past, we included all test files that had a reference to 'subprocess'' somewhere in their
# text. The monkey-patched stdlib tests were specifically included here.
# However, we now always also test on AppVeyor (Windows) which only has GEVENT_FILE=thread,
# so we can save a lot of CI time by reducing the set and excluding the stdlib tests without
# losing any coverage.
env:
GEVENT_FILE: thread
run: |
python -m gevent.tests --second-chance $G_USE_COV `(cd src/gevent/tests >/dev/null && ls test__*subprocess*.py)`
- name: "Tests: c-ares resolver"
# This sometimes fails on mac. # && (matrix.python-version == '3.11.8')
if: startsWith(runner.os, 'Linux')
env:
GEVENT_RESOLVER: ares
run: |
python -mgevent.tests --second-chance $G_USE_COV --ignore tests_that_dont_use_resolver.txt
- name: "Tests: dnspython resolver"
# This has known issues on Pypy-3.6. dnspython resolver not
# supported under anything newer than 3.10, so far.
if: (matrix.python-version == '3.9') && startsWith(runner.os, 'Linux')
env:
GEVENT_RESOLVER: dnspython
run: |
python -mgevent.tests --second-chance $G_USE_COV --ignore tests_that_dont_use_resolver.txt
- name: "Tests: leakchecks"
# Run the leaktests;
# This is incredibly important and we MUST have an environment that successfully passes
# these tests.
if: (startsWith(matrix.python-version, '3.11.8')) && startsWith(runner.os, 'Linux')
env:
GEVENTTEST_LEAKCHECK: 1
run: |
python -m gevent.tests --second-chance --ignore tests_that_dont_do_leakchecks.txt
- name: "Tests: PURE_PYTHON"
# No compiled cython modules on CPython, using the default backend. Get coverage here.
# We should only need to run this for a single version.
if: (matrix.python-version == '3.11.8') && startsWith(runner.os, 'Linux')
env:
PURE_PYTHON: 1
run: |
python -mgevent.tests --second-chance --coverage
- name: "Tests: libuv"
if: (startsWith(matrix.python-version, '3.11.8'))
env:
GEVENT_LOOP: libuv
run: |
python -m gevent.tests --second-chance $G_USE_COV
- name: "Tests: libev-cffi"
if: (matrix.python-version == '3.11.8') && startsWith(runner.os, 'Linux')
env:
GEVENT_LOOP: libev-cffi
run: |
python -m gevent.tests --second-chance $G_USE_COV
- name: Report coverage
if: ${{ !startsWith(matrix.python-version, 'pypy') }}
run: |
python -m coverage combine || true
python -m coverage report -i || true
- name: Coveralls Parallel
uses: coverallsapp/github-action@v2
# 20230707: On macOS, this installs coveralls from homebrew.
# It then runs ``coveralls report``. But that is producing
# a usage error from ``coveralls`` (report is not recognized) Presumably the
# brew and action versions are out of sync?
if: ${{ !startsWith(matrix.python-version, 'pypy') && !startsWith(matrix.python-version, '3.12') && startsWith(runner.os, 'Linux') }}
with:
flag-name: run-${{ join(matrix.*, '-') }}
parallel: true
- name: Lint
if: matrix.python-version == '3.10' && startsWith(runner.os, 'Linux')
# We only need to do this on one version.
# We do this here rather than a separate job to avoid the compilation overhead.
# 20230707: Python 3.11 crashes inside pylint/astroid on _ssl3.py;
# reverting to Python 3.10 solved that.
# TODO: Revisit this when we have caching of that part.
run: |
pip install -U pylint
python -m pylint --rcfile=.pylintrc gevent
coveralls_finish:
needs: test
runs-on: ubuntu-latest
steps:
- name: Coveralls Finished
uses: coverallsapp/github-action@v2
with:
parallel-finished: true
test_no_embed:
runs-on: ${{ matrix.os }}
strategy:
matrix:
python-version: ['3.11']
os: [ubuntu-latest]
steps:
- name: checkout
uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: setup.py
- name: Install ccache (ubuntu)
if: startsWith(runner.os, 'Linux')
run: |
sudo apt-get install -y ccache sed gcc
echo CCACHE_DIR=$HOME/.ccache >>$GITHUB_ENV
mkdir -p $HOME/.ccache
- name: Cache ~/.ccache
uses: actions/cache@v4
with:
path: ~/.ccache/**
key: ${{ runner.os }}-ccache2_embed-${{ matrix.python-version }}
- name: Cache config.cache
# Store the configure caches. Having a cache can speed up c-ares
# configure from 2-3 minutes to 20 seconds.
uses: actions/cache@v4
with:
path: deps/*/config.cache
# XXX: This should probably include a hash of each configure
# script We don't have a restore-keys that doesn't include
# the CFLAGS becouse the scripts fail to run if they get
# different CFLAGS, CC, CPPFLAGS, etc, and GHA offers no way
# to manually clear the cache. At one time, we had a
# restore-key configured, and it still seems to be used even
# without that setting here. The whole thing is being
# matched even without the CFLAGS matching. Perhaps the - is
# a generic search separator?
key: ${{ runner.os }}-${{ matrix.os }}-configcache_embed-${{ matrix.python-version }}-${{ env.CFLAGS }}
- name: Install dependencies
run: |
pip install -U pip
pip install -U -q setuptools wheel twine
pip install -q -U 'cffi;platform_python_implementation=="CPython"'
pip install -q -U 'cython>=3.0'
pip install 'greenlet>=2.0.0; platform_python_implementation=="CPython"'
- name: build libs and gevent
env:
GEVENTSETUP_EMBED: 0
GEVENTSETUP_EV_VERIFY: 1
run: |
# These need to be absolute paths
export BUILD_LIBS="$HOME/.libs/"
mkdir -p $BUILD_LIBS
export LDFLAGS=-L$BUILD_LIBS/lib
export CPPFLAGS="-I$BUILD_LIBS/include"
env | sort
echo which sed? `which sed`
echo LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$BUILD_LIBS/lib >>$GITHUB_ENV
(pushd deps/libev && sh ./configure -C --prefix=$BUILD_LIBS && make install && popd)
(pushd deps/c-ares && sh ./configure -C --prefix=$BUILD_LIBS && make -j4 install && popd)
(pushd deps/libuv && ./autogen.sh && sh ./configure -C --disable-static --prefix=$BUILD_LIBS && make -j4 install && popd)
# libev builds a manpage each time, and it includes today's date, so it frequently changes.
# delete to avoid repacking the archive
rm -rf $BUILD_LIBS/share/man/
ls -l $BUILD_LIBS $BUILD_LIBS/lib $BUILD_LIBS/include
python setup.py bdist_wheel
pip uninstall -y gevent
pip install -U `ls dist/*whl`[test]
# Test that we're actually linking
# to the .so file.
objdump -p build/lib*/gevent/libev/_corecffi*so | grep "NEEDED.*libev.so"
objdump -p build/lib*/gevent/libev/corecext*so | grep "NEEDED.*libev.so"
objdump -p build/lib*/gevent/libuv/_corecffi*so | grep "NEEDED.*libuv.so"
objdump -p build/lib*/gevent/resolver/cares*so | grep "NEEDED.*libcares.so"
- name: test non-embedded
run: |
# Verify that we got non-embedded builds
python -c 'import gevent.libev.corecffi as CF; assert not CF.LIBEV_EMBED'
python -c 'import gevent.libuv.loop as CF; assert not CF.libuv.LIBUV_EMBED'
python -mgevent.tests --second-chance
manylinux:
runs-on: ubuntu-latest
# If we have 'needs: test', then these wait to start running until
# all the test matrix passes. That's good, because these take a
# long time, and they take a long time to kill if something goes
# wrong. OTOH, if one of the tests fail, and this is a release tag,
# we have to notice that and try restarting things so that the
# wheels get built and uploaded. For that reason, it's simplest to
# remove this for release branches.
needs: test
strategy:
matrix:
python-version: [3.9]
image:
- manylinux2014_aarch64
- manylinux2014_ppc64le
- manylinux2014_s390x
- manylinux_2_28_x86_64
- musllinux_1_1_x86_64
- musllinux_1_1_aarch64
- manylinux2014_x86_64
name: ${{ matrix.image }}
steps:
- name: checkout
uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Cache ~/.ccache
uses: actions/cache@v4
with:
path: ~/.ccache/**
key: ${{ runner.os }}-ccache_${{ matrix.config[2] }}-${{ matrix.config[0] }}
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
with:
platforms: all
- name: Build and test gevent
env:
DOCKER_IMAGE: quay.io/pypa/${{ matrix.image }}
GEVENT_MANYLINUX_NAME: ${{ matrix.image }}
run: scripts/releases/make-manylinux
- name: Upload gevent wheels
uses: actions/upload-artifact@v4
with:
path: wheelhouse/*whl
name: ${{ matrix.config[2] }}_x86_64_wheels.zip
- name: Publish package to PyPI
uses: pypa/gh-action-pypi-publish@v1.4.1
if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags')
with:
user: __token__
password: ${{ secrets.TWINE_PASSWORD }}
skip_existing: true
packages_dir: wheelhouse/
# TODO:
# * Use YAML syntax to share snippets, like the old .travis.yml did