Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Avoid hanging forever after a parallel job was killed #7834

Merged
merged 10 commits into from Dec 12, 2022
Merged

Avoid hanging forever after a parallel job was killed #7834

merged 10 commits into from Dec 12, 2022

Commits on Nov 23, 2022

  1. Replace multiprocessing.pool with concurrent.futures.ProcessPoolExecu… 

    …tor to avoid deadlocks.

In a multiprocessing.pool, if a process terminates in a non-clean fashion
(for example, due to OOM or a segmentation fault), the pool will silently
replace said process, but the work that the process was supposed to do
will never be done, causing pylint to hang indefinitely.
The concurrent.futures.ProcessPoolExecutor will raise a
BrokenProcessPool exception in that case, avoiding the hang.
    @daniel-wer
    daniel-wer committed Nov 23, 2022
    Copy the full SHA
    fc1baf7 View commit details
    Browse the repository at this point in the history

Commits on Nov 24, 2022

  1. Fix initargs parameter to be a tuple.

    @daniel-wer
    daniel-wer committed Nov 24, 2022
    Copy the full SHA
    d5e17d8 View commit details
    Browse the repository at this point in the history

  2. add whatsnew fragment

    @daniel-wer
    daniel-wer committed Nov 24, 2022
    Copy the full SHA
    2657772 View commit details
    Browse the repository at this point in the history

Commits on Nov 28, 2022

  1. Add a test that an error in the initializer for the parallel jobs, do… 

    …esn't

lead to a deadlock.
    @daniel-wer
    daniel-wer committed Nov 28, 2022
    Copy the full SHA
    2e47049 View commit details
    Browse the repository at this point in the history

Commits on Dec 5, 2022

  1. Increase timeout for parallel initializer test to avoid flakiness.

    @daniel-wer
    daniel-wer committed Dec 5, 2022
    Copy the full SHA
    236e5e9 View commit details
    Browse the repository at this point in the history

Commits on Dec 6, 2022

  1. Guard concurrent.futures imports, because they are not available in E… 

    …mscripten and WASI.
    @daniel-wer
    daniel-wer committed Dec 6, 2022
    Copy the full SHA
    61b8342 View commit details
    Browse the repository at this point in the history

Commits on Dec 8, 2022

  1. Check availability of concurrent.futures module instead of multiproce… 

    …ssing for single process fallback.
    @daniel-wer
    daniel-wer committed Dec 8, 2022
    Copy the full SHA
    a52cb6c View commit details
    Browse the repository at this point in the history

Commits on Dec 10, 2022

  1. Small fixes

    @DanielNoord
    DanielNoord committed Dec 10, 2022
    Copy the full SHA
    30af13a View commit details
    Browse the repository at this point in the history

Commits on Dec 12, 2022

  1. Update tests/test_check_parallel.py 

    Co-authored-by: Daniël van Noord <13665637+DanielNoord@users.noreply.github.com>
    @daniel-wer @DanielNoord
    daniel-wer and DanielNoord committed Dec 12, 2022
    Copy the full SHA
    e164bea View commit details
    Browse the repository at this point in the history

  2. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Dec 12, 2022
    Copy the full SHA
    69729db View commit details
    Browse the repository at this point in the history