Reference Count Testing

(Please update or delete the page when it becomes outdated.)

It is possible to test numpy reference counts using pytest-leaks which as of now (July 2019) runs mostly clean through with NumPy.

To do this you need to:

1. Have a python debug build available (I use the python3-dbg as provided by debian)
2. Install pytest-leaks:
   • pip install pytest-leaks, note that you need version 3 for newer python version support.
3. Make sure you have an up to date Cython (as of July 2019 the dev version), since that fixes cython related leaks seen in numpy (which may mean the development version).
4. Run git clean -xdf in the numpy directory (make sure not to delete any new work related files)
5. Use NumPy 1.18.dev or newer (newer than 2019-07-31), to avoid false positives with object arrays.

You can now run: python3-dbg runtests.py -v -- -R 2:3 -s -k "not leaks_references" (runs each test 2 times for "warm up" and then checks that the reference count does not change for another 3 runs), or similar incarnations. Note that -s is necessary to disable the pytest output capture, which causes false positives (as of Aug. 2019). -k "not leaks_references" filters out some tests which are known to leak (most of which are impossible to fix).

The numpy tests in master run cleanly (with the exception of distutils and a few marked tests most of which cannot be fixed; most likely there are still reference count errors for code with low test coverage), so using a debug install of NumPy, the same process can be used for example to test SciPy, with hopefully no, or few false positives due to errors inside of NumPy. (Some tests fail, becaues they cannot be run multiple times correctly)