Race condition when building with Meson

[lesteve]

Opening this to track it in an issue rather than seeing it appear in spread-out PRs.

The error looks like this:

[61/249] Compiling Cython source sklearn/linear_model/_sgd_fast.pyx
FAILED: sklearn/linear_model/_sgd_fast.cpython-312-darwin.so.p/sklearn/linear_model/_sgd_fast.pyx.c
cython -M --fast-fail -3 '-X language_level=3' '-X boundscheck=False' '-X wraparound=False' '-X initializedcheck=False' '-X nonecheck=False' '-X cdivision=True' '-X profile=False' --include-dir /Users/runner/work/1/s/build/cp312 sklearn/linear_model/_sgd_fast.pyx -o sklearn/linear_model/_sgd_fast.cpython-312-darwin.so.p/sklearn/linear_model/_sgd_fast.pyx.c


from cython cimport floating
from libc.math cimport exp, fabs, isfinite, log, pow, INFINITY

from ..utils._typedefs cimport uint32_t
^
------------------------------------------------------------

sklearn/linear_model/_sgd_fast.pyx:9:0: relative cimport from non-package directory is not allowed

or like this:

[60/249] Compiling Cython source /Users/runner/work/1/s/sklearn/cluster/_hdbscan/_linkage.pyx
  FAILED: sklearn/cluster/_hdbscan/_linkage.cpython-312-darwin.so.p/sklearn/cluster/_hdbscan/_linkage.pyx.c
  cython -M --fast-fail -3 '-X language_level=3' '-X boundscheck=False' '-X wraparound=False' '-X initializedcheck=False' '-X nonecheck=False' '-X cdivision=True' '-X profile=False' --include-dir /Users/runner/work/1/s/build/cp312 /Users/runner/work/1/s/sklearn/cluster/_hdbscan/_linkage.pyx -o sklearn/cluster/_hdbscan/_linkage.cpython-312-darwin.so.p/sklearn/cluster/_hdbscan/_linkage.pyx.c

  Error compiling Cython file:
  ------------------------------------------------------------
  ...

  cimport numpy as cnp
  from libc.float cimport DBL_MAX

  import numpy as np
  from ...metrics._dist_metrics cimport DistanceMetric64
  ^
  ------------------------------------------------------------

  /Users/runner/work/1/s/sklearn/cluster/_hdbscan/_linkage.pyx:38:0: 'sklearn/metrics/_dist_metrics.pxd' not found

Some files __init__.py of .pxd need to be copied to the Meson build dir before the extension using cimport can be built. Since the Meson (or rather ninja) build order is based on dependencies (there is no way to tweak the build order ninja-build/ninja#232), when you are unlucky it does not happen in the right order and the file is not found. Not clear to me right now whether this is a Meson issue or not.

For some unknown reason, until now this has always happened in the no-OpenMP build. I tried to reproduce on OSX with the no-OpenMP lock-file but I did not manage to reproduce.

Occurences of this:

• CI Use check-sdist to check sdist rather than check-manifest #28757 (comment) fixed by BLD Fix build dependency in meson build #28789
• DOC Add 1.4.2 to News #28797 (comment) and MAINT Fix badge links and underscores in README.rst file #28815 (comment), should be fixed by BLD Fix more build dependencies in meson build #28821
• ⚠️ CI failed on macOS.pylatest_conda_mkl_no_openmp (last failure: Apr 20, 2024) ⚠️ #28863

[ngoldbaum]

I think this just happened on #28980