Make TSAN tests pass with the GIL disabled in free-threaded builds #117657

mpage · 2024-04-08T22:13:53Z

Feature or enhancement

We need to make the TSAN tests pass with the GIL disabled before we can disable the GIL by default in free-threaded builds.

This should proceed in two phases:

Add suppressions for existing TSAN warnings.
Triage, fix, and remove the suppressions for the warnings enumerated in (1).

How to run the TSAN tests

Build with TSAN:
env CC=clang CXX=clang++ ./configure --disable-gil --with-thread-sanitizer --with-pydebug && make -j
Run tests:
env TSAN_OPTIONS=suppressions=<repo_root>/Tools/tsan/suppressions_free_threading.txt ./python -m test --tsan -j4

Working on a race

Verify that the TSAN tests are passing using the steps from above.
Pick a suppression from the section below and assign it to yourself (edit it and add your username next to it). Some of them may be related. If you find that other suppressions are related to the race you're working on please assign them to yourself or contact their owner if they're already assigned.
Optionally, head over to the race index to see examples and which tests reproduce the race corresponding to the suppression.
Delete the suppression from <repo_root>/Tools/tsan/suppressions_free_threading.txt , run the TSAN tests, and verify that the race is reported by TSAN. You may need to comment out unrelated functions (notably, _PyEval_EvalFrameDefault) in order to reproduce the race.
Fix the race.

Suppressions

Linked PRs

The text was updated successfully, but these errors were encountered:

…tegration

Additionally, reduce the iterations for a few weakref tests that would otherwise take a prohibitively long amount of time (> 1 hour) when TSAN is enabled and the GIL is disabled.

…rld()` and `tstate_try_attach()` (#117828) TSAN erroneously reports a data race between the `_Py_atomic_compare_exchange_int` on `tstate->state` in `tstate_try_attach()` and the non-atomic load of `tstate->state` in `start_the_world`. The `_Py_atomic_compare_exchange_int` fails, but TSAN erroneously treats it as a store.

…are/exchange (#117830)

…ython#117736) Additionally, reduce the iterations for a few weakref tests that would otherwise take a prohibitively long amount of time (> 1 hour) when TSAN is enabled and the GIL is disabled.

…the_world()` and `tstate_try_attach()` (python#117828) TSAN erroneously reports a data race between the `_Py_atomic_compare_exchange_int` on `tstate->state` in `tstate_try_attach()` and the non-atomic load of `tstate->state` in `start_the_world`. The `_Py_atomic_compare_exchange_int` fails, but TSAN erroneously treats it as a store.

…f compare/exchange (python#117830)

…#117954) Fix data races in the method cache in free-threaded builds These are technically data races, but I think they're benign (to the extent that that is actually possible). We update cache entries non-atomically but read them atomically from another thread, and there's nothing that establishes a happens-before relationship between the reads and writes that I can see.

…#117955) Quiet erroneous TSAN reports of data races in `_PySeqLock` TSAN reports a couple of data races between the compare/exchange in `_PySeqLock_LockWrite` and the non-atomic loads in `_PySeqLock_{Abandon,Unlock}Write`. This is another instance of TSAN incorrectly modeling failed compare/exchange as a write instead of a load.

) Use relaxed load to check if dictkeys are immortal

…th strong enough semantics (#118111) Use acquire for load of ob_ref_shared

… `tstate->state` (#118165) Quiet TSAN warnings about remaining non-atomic accesses of `tstate->state`

…19268) Fix itertools.count in free-threading mode (cherry picked from commit 87939bd) Co-authored-by: Arnon Yaari <wiggin15@yahoo.com>

#120005) The free-threaded build currently immortalizes objects that use deferred reference counting (see gh-117783). This typically happens once the first non-main thread is created, but the behavior can be suppressed for tests, in subinterpreters, or during a compile() call. This fixes a race condition involving the tracking of whether the behavior is suppressed. (cherry picked from commit 47fb432)

) Fix itertools.count in free-threading mode (cherry picked from commit 87939bd) Co-authored-by: Arnon Yaari <wiggin15@yahoo.com>

…ython#119921) The GIL may be disabled concurrently with this call so we need to use a relaxed atomic load.

…-119907)

The `sem_clockwait` function is not currently instrumented, which leads to false positives.

) The free-threaded build currently immortalizes objects that use deferred reference counting (see pythongh-117783). This typically happens once the first non-main thread is created, but the behavior can be suppressed for tests, in subinterpreters, or during a compile() call. This fixes a race condition involving the tracking of whether the behavior is suppressed.

The `_PyThreadState_Bind()` function is called before the first `PyEval_AcquireThread()` so it's not synchronized with the stop the world GC. We had a race where `gc_visit_heaps()` might visit a thread's heap while it's being initialized. Use a simple atomic int to avoid visiting heaps for threads that are not yet fully initialized (i.e., before `tstate_mimalloc_bind()` is called). The race was reproducible by running: `python Lib/test/test_importlib/partial/pool_in_threads.py`.

…nGH-119923) The `_PyThreadState_Bind()` function is called before the first `PyEval_AcquireThread()` so it's not synchronized with the stop the world GC. We had a race where `gc_visit_heaps()` might visit a thread's heap while it's being initialized. Use a simple atomic int to avoid visiting heaps for threads that are not yet fully initialized (i.e., before `tstate_mimalloc_bind()` is called). The race was reproducible by running: `python Lib/test/test_importlib/partial/pool_in_threads.py`. (cherry picked from commit e69d068) Co-authored-by: Sam Gross <colesbury@gmail.com>

…19923) (#120038) The `_PyThreadState_Bind()` function is called before the first `PyEval_AcquireThread()` so it's not synchronized with the stop the world GC. We had a race where `gc_visit_heaps()` might visit a thread's heap while it's being initialized. Use a simple atomic int to avoid visiting heaps for threads that are not yet fully initialized (i.e., before `tstate_mimalloc_bind()` is called). The race was reproducible by running: `python Lib/test/test_importlib/partial/pool_in_threads.py`. (cherry picked from commit e69d068) Co-authored-by: Sam Gross <colesbury@gmail.com>

Only call `gc_restore_tid()` from stop-the-world contexts. `worklist_pop()` can be called while other threads are running, so use a relaxed atomic to modify `ob_tid`.

Due to a limitation in TSAN, all reads from `PyThreadState.state` must be atomic to avoid reported races.

…#119908) Seen in CI occasionally when running `test_weakref`.

…ython#119921) The GIL may be disabled concurrently with this call so we need to use a relaxed atomic load.

…-119907)

The `sem_clockwait` function is not currently instrumented, which leads to false positives.

) The free-threaded build currently immortalizes objects that use deferred reference counting (see pythongh-117783). This typically happens once the first non-main thread is created, but the behavior can be suppressed for tests, in subinterpreters, or during a compile() call. This fixes a race condition involving the tracking of whether the behavior is suppressed.

…n#119923) The `_PyThreadState_Bind()` function is called before the first `PyEval_AcquireThread()` so it's not synchronized with the stop the world GC. We had a race where `gc_visit_heaps()` might visit a thread's heap while it's being initialized. Use a simple atomic int to avoid visiting heaps for threads that are not yet fully initialized (i.e., before `tstate_mimalloc_bind()` is called). The race was reproducible by running: `python Lib/test/test_importlib/partial/pool_in_threads.py`.

This adds a `_PyRecursiveMutex` type based on `PyMutex` and uses that for the import lock. This fixes some data races in the free-threaded build and generally simplifies the import lock code.

This adds a `_PyRecursiveMutex` type based on `PyMutex` and uses that for the import lock. This fixes some data races in the free-threaded build and generally simplifies the import lock code. (cherry picked from commit e21057b) Co-authored-by: Sam Gross <colesbury@gmail.com>

…20169) This adds a `_PyRecursiveMutex` type based on `PyMutex` and uses that for the import lock. This fixes some data races in the free-threaded build and generally simplifies the import lock code. (cherry picked from commit e21057b) Co-authored-by: Sam Gross <colesbury@gmail.com>

mpage added type-feature A feature request or enhancement topic-free-threading labels Apr 8, 2024

mpage self-assigned this Apr 8, 2024

mpage mentioned this issue Apr 10, 2024

PEP 703 -- Making the Global Interpreter Lock Optional in CPython #108219

Open

bedevere-app bot mentioned this issue Apr 10, 2024

gh-117657: Add TSAN suppressions for the free-threaded build #117736

Merged

colesbury added a commit to colesbury/cpython that referenced this issue Apr 12, 2024

Merge branch 'pythongh-117657-add-ft-tsan-suppressions' into nogil-in…

73c8d7c

…tegration

This was referenced Apr 12, 2024

gh-117657: Quiet TSAN warning about a data race between start_the_world() and tstate_try_attach() #117828

Merged

gh-117657: Quiet more TSAN warnings due to incorrect modeling of compare/exchange #117830

Merged

colesbury pushed a commit that referenced this issue Apr 15, 2024

gh-117657: Quiet more TSAN warnings due to incorrect modeling of comp…

0d52383

…are/exchange (#117830)

This was referenced Apr 17, 2024

gh-117657: Fix data races in the method cache in free-threaded builds #117954

Merged

gh-117657: Quiet erroneous TSAN reports of data races in _PySeqLock #117955

Merged

diegorusso pushed a commit to diegorusso/cpython that referenced this issue Apr 17, 2024

pythongh-117657: Quiet more TSAN warnings due to incorrect modeling o…

f8e2d81

…f compare/exchange (python#117830)

DinoV added a commit to mpage/cpython that referenced this issue Apr 17, 2024

Merge branch 'main' into pythongh-117657-seq-lock

778de8e

This was referenced Apr 18, 2024

gh-117657: Fix small issues with instrumentation and TSAN #118064

Merged

gh-117657: use relaxed loads for checking dict keys immortality #118067

Merged

DinoV added a commit that referenced this issue Apr 19, 2024

gh-117657: use relaxed loads for checking dict keys immortality (#118067

1e4a4c4

) Use relaxed load to check if dictkeys are immortal

bedevere-app bot mentioned this issue Apr 19, 2024

[gh-117657] _Py_MergeZeroLocalRefcount isn't loading ob_ref_shared with strong enough semantics #118111

Merged

DinoV added a commit that referenced this issue Apr 19, 2024

[gh-117657] _Py_MergeZeroLocalRefcount isn't loading ob_ref_shared wi…

b45af00

…th strong enough semantics (#118111) Use acquire for load of ob_ref_shared

bedevere-app bot mentioned this issue Apr 23, 2024

gh-117657: Quiet TSAN warnings about remaining non-atomic accesses of tstate->state #118165

Merged

DinoV pushed a commit that referenced this issue Apr 23, 2024

gh-117657: Quiet TSAN warnings about remaining non-atomic accesses of…

2e7771a

… `tstate->state` (#118165) Quiet TSAN warnings about remaining non-atomic accesses of `tstate->state`

bedevere-app bot mentioned this issue Jun 3, 2024

[3.13] gh-117657: Fix itertools.count thread safety (GH-119268) #120007

Merged

colesbury added a commit that referenced this issue Jun 3, 2024

[3.13] gh-117657: Fix itertools.count thread safety (GH-119268) (#120007

79fae3b

) Fix itertools.count in free-threading mode (cherry picked from commit 87939bd) Co-authored-by: Arnon Yaari <wiggin15@yahoo.com>

mliezun pushed a commit to mliezun/cpython that referenced this issue Jun 3, 2024

pythongh-117657: Fix TSAN reported race in _PyEval_IsGILEnabled. (p…

ad1c71a

…ython#119921) The GIL may be disabled concurrently with this call so we need to use a relaxed atomic load.

mliezun pushed a commit to mliezun/cpython that referenced this issue Jun 3, 2024

pythongh-117657: Fix data races report by TSAN unicode-hash (pythongh…

7c006a3

…-119907)

mliezun pushed a commit to mliezun/cpython that referenced this issue Jun 3, 2024

pythongh-117657: Avoid sem_clockwait in TSAN (python#119915)

93d38fc

The `sem_clockwait` function is not currently instrumented, which leads to false positives.

bedevere-app bot mentioned this issue Jun 4, 2024

[3.13] gh-117657: Fix race involving GC and heap initialization (GH-119923) #120038

Merged

colesbury added a commit to colesbury/cpython that referenced this issue Jun 4, 2024

Merge branch 'main' into pythongh-117657-import-lock

225e5b9

bedevere-app bot mentioned this issue Jun 5, 2024

gh-117657: Fix some simple races in instrumentation.c #120118

Open

barneygale pushed a commit to barneygale/cpython that referenced this issue Jun 5, 2024

pythongh-117657: Fix TSAN race in QSBR assertion (python#119887)

d60317e

Due to a limitation in TSAN, all reads from `PyThreadState.state` must be atomic to avoid reported races.

barneygale pushed a commit to barneygale/cpython that referenced this issue Jun 5, 2024

pythongh-117657: Add TSAN suppression for set_discard_entry (python…

50f9637

…#119908) Seen in CI occasionally when running `test_weakref`.

barneygale pushed a commit to barneygale/cpython that referenced this issue Jun 5, 2024

pythongh-117657: Fix TSAN reported race in _PyEval_IsGILEnabled. (p…

e2633ae

…ython#119921) The GIL may be disabled concurrently with this call so we need to use a relaxed atomic load.

barneygale pushed a commit to barneygale/cpython that referenced this issue Jun 5, 2024

pythongh-117657: Fix data races report by TSAN unicode-hash (pythongh…

ad9cff9

…-119907)

barneygale pushed a commit to barneygale/cpython that referenced this issue Jun 5, 2024

pythongh-117657: Avoid sem_clockwait in TSAN (python#119915)

a232d05

The `sem_clockwait` function is not currently instrumented, which leads to false positives.

bedevere-app bot mentioned this issue Jun 6, 2024

gh-117657: Make Py_TYPE and Py_SET_TYPE thread safe #120165

Open

bedevere-app bot mentioned this issue Jun 6, 2024

[3.13] gh-117657: Fix TSAN race involving import lock (GH-118523) #120169

Merged

colesbury added a commit to colesbury/cpython that referenced this issue Jun 6, 2024

pythongh-117657: Fix TSan reported data race on ioctl_works

b5ebee7

This was referenced Jun 6, 2024

gh-117657: Fix TSan reported data race on ioctl_works #120175

Open

gh-117657: Make PyType_HasFeature atomic #120210

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make TSAN tests pass with the GIL disabled in free-threaded builds #117657

Make TSAN tests pass with the GIL disabled in free-threaded builds #117657

mpage commented Apr 8, 2024 •

edited by Fidget-Spinner

Tasks

Make TSAN tests pass with the GIL disabled in free-threaded builds #117657

Make TSAN tests pass with the GIL disabled in free-threaded builds #117657

Comments

mpage commented Apr 8, 2024 • edited by Fidget-Spinner

Feature or enhancement

How to run the TSAN tests

Working on a race

Suppressions

Tasks

Linked PRs

mpage commented Apr 8, 2024 •

edited by Fidget-Spinner