Hard-to-diagnose failures in sktime CI/CD

[fkiraly]

First of all, really great package! Excellent to speed up testing and/or distribute it.

I wanted to report some hard-to-diagnose issues on sktime (AI for time series package)
https://github.com/alan-turing-institute/sktime
that ultimately made us remove it from the CI/CD.
Summary here:
sktime/sktime#2012

The individual problems in the remote CI/CD are as follows:

• in combination with a test that tests parallelization of algorithm, it would lead to stochastic (not deterministic) hangs of individual workers. Problem described here: [BUG] Identify cause and fix CI/CD stuck in loop on unix, mac - reproduction & diagnostics appreciated sktime/sktime#1941
  Precise condition and stochastic behaviour identified here: DRAFT for diagnosing CI/CD: smaller set of class tests test_all_estimators, instance tests all removed sktime/sktime#1949
• stochastic worker crahses when using pytest test classes vs not, the precise condition is presence of the same tests in those classes, as opposed to functions. Problem described here: [ENH] test framework: test class for forecasting module sktime/sktime#1992

I know this may be hard to track down, but it seems to touch very general useability of pytest-xdist.
Happy to help in further identifying the problems, since we really would like to use pytest-xdist.

[fkiraly]

update: pytest-xdist may not actually be the root cause, after some digging.

Presence/absence of it does impact presence/absence of hangs and crashes - but that does not mean it is the cause.

A suspicion I have is high memory usage caused by sth, and pytest-xdist may or may not push it over the edge by partitioning memory allocations etc. That would also explain the stochastic behaviour.

Any good ideas, e.g., how to do memory profiling of pytest and/or pytest-xdist on the CI/CD?
sktime/sktime#2030

My current best guess is that

• [BUG] Identify cause and fix CI/CD stuck in loop on unix, mac - reproduction & diagnostics appreciated sktime/sktime#1941 is a direct interaction of two parallelization frameworks, possibly a bug in either
• observations in [#1992]([ENH] test framework: test class for forecasting module sktime/sktime#1992 are memory usage related, pytest-xdist is not the direct cause. See some more discussions at the end of [MNT] removing pytest-xdist from CI/CD? sktime/sktime#2012