Meson does not fully build the project in one go and need to be run twice ?

[lesteve]

To reproduce (I use Meson commands directly below to show that is is not related to meson-python):

meson setup build/test
# Start from a built project
ninja -C build/test

# ninja is timestamp-based to touching this pxd will cause things to rebuild
touch sklearn/utils/_typedefs.pxd
# 124 targets need to be rebuilt, this is expected
ninja -C build/test

# I expected "no work to do here"
# Actually 36 targets need to be rebuilt, which is NOT expected
ninja -C build/test

cc @eli-schwartz in case you have any suggestions on this.

I don't quite understand why the files need to be rebuilt, ninja -d explain does not shed too much light on it. Here is the -d explain output on the second build, it says _dist_metrics.pyx.c needs to be rebuilt although the first build did not think it needed to be rebuilt for some reason ...

"ninja -d explain" output for the second build

ninja: Entering directory `build/cp312'
ninja explain: output meson-test-prereq of phony edge with no inputs doesn't exist
ninja explain: meson-test-prereq is dirty
ninja explain: output meson-benchmark-prereq of phony edge with no inputs doesn't exist
ninja explain: meson-benchmark-prereq is dirty
ninja explain: restat of output sklearn/metrics/_dist_metrics.cpython-312-x86_64-linux-gnu.so.p/sklearn/metrics/_dist_metrics.pyx.c older than most recent input /home/lesteve/dev/scikit-learn/build/cp312/sklearn/utils/_typedefs.pxd (1713170316072356503 vs 1713170351662119783)
ninja explain: sklearn/metrics/_dist_metrics.cpython-312-x86_64-linux-gnu.so.p/sklearn/metrics/_dist_metrics.pyx.c is dirty
ninja explain: sklearn/metrics/_dist_metrics.cpython-312-x86_64-linux-gnu.so.p/sklearn/metrics/_dist_metrics.pyx.c is dirty
ninja explain: sklearn/metrics/_dist_metrics.cpython-312-x86_64-linux-gnu.so.p/meson-generated_sklearn_metrics__dist_metrics.pyx.c.o is dirty
ninja explain: sklearn/metrics/_dist_metrics.cpython-312-x86_64-linux-gnu.so is dirty
ninja explain: restat of output sklearn/metrics/_pairwise_distances_reduction/_datasets_pair.cpython-312-x86_64-linux-gnu.so.p/sklearn/metrics/_pairwise_distances_reduction/_datasets_pair.pyx.cpp older than most recent input /home/lesteve/dev/scikit-learn/build/cp312/sklearn/utils/_typedefs.pxd (1713170315105696264 vs 1713170351662119783)
ninja explain: sklearn/metrics/_pairwise_distances_reduction/_datasets_pair.cpython-312-x86_64-linux-gnu.so.p/sklearn/metrics/_pairwise_distances_reduction/_datasets_pair.pyx.cpp is dirty
ninja explain: sklearn/metrics/_pairwise_distances_reduction/_datasets_pair.cpython-312-x86_64-linux-gnu.so.p/sklearn/metrics/_pairwise_distances_reduction/_datasets_pair.pyx.cpp is dirty
ninja explain: sklearn/metrics/_pairwise_distances_reduction/_datasets_pair.cpython-312-x86_64-linux-gnu.so.p/meson-generated_sklearn_metrics__pairwise_distances_reduction__datasets_pair.pyx.cpp.o is dirty
ninja explain: sklearn/metrics/_pairwise_distances_reduction/_datasets_pair.cpython-312-x86_64-linux-gnu.so is dirty
ninja explain: restat of output sklearn/metrics/_pairwise_distances_reduction/_base.cpython-312-x86_64-linux-gnu.so.p/sklearn/metrics/_pairwise_distances_reduction/_base.pyx.cpp older than most recent input /home/lesteve/dev/scikit-learn/build/cp312/sklearn/utils/_typedefs.pxd (1713170315109029574 vs 1713170351662119783)
ninja explain: sklearn/metrics/_pairwise_distances_reduction/_base.cpython-312-x86_64-linux-gnu.so.p/sklearn/metrics/_pairwise_distances_reduction/_base.pyx.cpp is dirty
ninja explain: sklearn/metrics/_pairwise_distances_reduction/_base.cpython-312-x86_64-linux-gnu.so.p/sklearn/metrics/_pairwise_distances_reduction/_base.pyx.cpp is dirty
ninja explain: sklearn/metrics/_pairwise_distances_reduction/_base.cpython-312-x86_64-linux-gnu.so.p/meson-generated_sklearn_metrics__pairwise_distances_reduction__base.pyx.cpp.o is dirty
ninja explain: sklearn/metrics/_pairwise_distances_reduction/_base.cpython-312-x86_64-linux-gnu.so is dirty
ninja explain: restat of output sklearn/metrics/_pairwise_distances_reduction/_middle_term_computer.cpython-312-x86_64-linux-gnu.so.p/sklearn/metrics/_pairwise_distances_reduction/_middle_term_computer.pyx.cpp older than most recent input /home/lesteve/dev/scikit-learn/build/cp312/sklearn/utils/_typedefs.pxd (1713170315819024853 vs 1713170351662119783)
ninja explain: sklearn/metrics/_pairwise_distances_reduction/_middle_term_computer.cpython-312-x86_64-linux-gnu.so.p/sklearn/metrics/_pairwise_distances_reduction/_middle_term_computer.pyx.cpp is dirty
ninja explain: sklearn/metrics/_pairwise_distances_reduction/_middle_term_computer.cpython-312-x86_64-linux-gnu.so.p/sklearn/metrics/_pairwise_distances_reduction/_middle_term_computer.pyx.cpp is dirty
ninja explain: sklearn/metrics/_pairwise_distances_reduction/_middle_term_computer.cpython-312-x86_64-linux-gnu.so.p/meson-generated_sklearn_metrics__pairwise_distances_reduction__middle_term_computer.pyx.cpp.o is dirty
ninja explain: sklearn/metrics/_pairwise_distances_reduction/_middle_term_computer.cpython-312-x86_64-linux-gnu.so is dirty
ninja explain: restat of output sklearn/metrics/_pairwise_distances_reduction/_argkmin.cpython-312-x86_64-linux-gnu.so.p/sklearn/metrics/_pairwise_distances_reduction/_argkmin.pyx.cpp older than most recent input /home/lesteve/dev/scikit-learn/build/cp312/sklearn/utils/_typedefs.pxd (1713170314829031437 vs 1713170351662119783)
ninja explain: sklearn/metrics/_pairwise_distances_reduction/_argkmin.cpython-312-x86_64-linux-gnu.so.p/sklearn/metrics/_pairwise_distances_reduction/_argkmin.pyx.cpp is dirty
ninja explain: sklearn/metrics/_pairwise_distances_reduction/_argkmin.cpython-312-x86_64-linux-gnu.so.p/sklearn/metrics/_pairwise_distances_reduction/_argkmin.pyx.cpp is dirty
ninja explain: sklearn/metrics/_pairwise_distances_reduction/_argkmin.cpython-312-x86_64-linux-gnu.so.p/meson-generated_sklearn_metrics__pairwise_distances_reduction__argkmin.pyx.cpp.o is dirty
ninja explain: sklearn/metrics/_pairwise_distances_reduction/_argkmin.cpython-312-x86_64-linux-gnu.so is dirty
ninja explain: restat of output sklearn/metrics/_pairwise_distances_reduction/_radius_neighbors.cpython-312-x86_64-linux-gnu.so.p/sklearn/metrics/_pairwise_distances_reduction/_radius_neighbors.pyx.cpp older than most recent input /home/lesteve/dev/scikit-learn/build/cp312/sklearn/utils/_typedefs.pxd (1713170315805691609 vs 1713170351662119783)
ninja explain: sklearn/metrics/_pairwise_distances_reduction/_radius_neighbors.cpython-312-x86_64-linux-gnu.so.p/sklearn/metrics/_pairwise_distances_reduction/_radius_neighbors.pyx.cpp is dirty
ninja explain: sklearn/metrics/_pairwise_distances_reduction/_radius_neighbors.cpython-312-x86_64-linux-gnu.so.p/sklearn/metrics/_pairwise_distances_reduction/_radius_neighbors.pyx.cpp is dirty
ninja explain: sklearn/metrics/_pairwise_distances_reduction/_radius_neighbors.cpython-312-x86_64-linux-gnu.so.p/meson-generated_sklearn_metrics__pairwise_distances_reduction__radius_neighbors.pyx.cpp.o is dirty
ninja explain: sklearn/metrics/_pairwise_distances_reduction/_radius_neighbors.cpython-312-x86_64-linux-gnu.so is dirty
ninja explain: restat of output sklearn/metrics/_pairwise_distances_reduction/_argkmin_classmode.cpython-312-x86_64-linux-gnu.so.p/sklearn/metrics/_pairwise_distances_reduction/_argkmin_classmode.pyx.cpp older than most recent input /home/lesteve/dev/scikit-learn/build/cp312/sklearn/utils/_typedefs.pxd (1713170315125696130 vs 1713170351662119783)
ninja explain: sklearn/metrics/_pairwise_distances_reduction/_argkmin_classmode.cpython-312-x86_64-linux-gnu.so.p/sklearn/metrics/_pairwise_distances_reduction/_argkmin_classmode.pyx.cpp is dirty
ninja explain: sklearn/metrics/_pairwise_distances_reduction/_argkmin_classmode.cpython-312-x86_64-linux-gnu.so.p/sklearn/metrics/_pairwise_distances_reduction/_argkmin_classmode.pyx.cpp is dirty
ninja explain: sklearn/metrics/_pairwise_distances_reduction/_argkmin_classmode.cpython-312-x86_64-linux-gnu.so.p/meson-generated_sklearn_metrics__pairwise_distances_reduction__argkmin_classmode.pyx.cpp.o is dirty
ninja explain: sklearn/metrics/_pairwise_distances_reduction/_argkmin_classmode.cpython-312-x86_64-linux-gnu.so is dirty
ninja explain: restat of output sklearn/metrics/_pairwise_distances_reduction/_radius_neighbors_classmode.cpython-312-x86_64-linux-gnu.so.p/sklearn/metrics/_pairwise_distances_reduction/_radius_neighbors_classmode.pyx.cpp older than most recent input /home/lesteve/dev/scikit-learn/build/cp312/sklearn/utils/_typedefs.pxd (1713170315229028777 vs 1713170351662119783)
ninja explain: sklearn/metrics/_pairwise_distances_reduction/_radius_neighbors_classmode.cpython-312-x86_64-linux-gnu.so.p/sklearn/metrics/_pairwise_distances_reduction/_radius_neighbors_classmode.pyx.cpp is dirty
ninja explain: sklearn/metrics/_pairwise_distances_reduction/_radius_neighbors_classmode.cpython-312-x86_64-linux-gnu.so.p/sklearn/metrics/_pairwise_distances_reduction/_radius_neighbors_classmode.pyx.cpp is dirty
ninja explain: sklearn/metrics/_pairwise_distances_reduction/_radius_neighbors_classmode.cpython-312-x86_64-linux-gnu.so.p/meson-generated_sklearn_metrics__pairwise_distances_reduction__radius_neighbors_classmode.pyx.cpp.o is dirty
ninja explain: sklearn/metrics/_pairwise_distances_reduction/_radius_neighbors_classmode.cpython-312-x86_64-linux-gnu.so is dirty
ninja explain: restat of output sklearn/linear_model/_sgd_fast.cpython-312-x86_64-linux-gnu.so.p/sklearn/linear_model/_sgd_fast.pyx.c older than most recent input /home/lesteve/dev/scikit-learn/build/cp312/sklearn/utils/_typedefs.pxd (1713170315105696264 vs 1713170351662119783)
ninja explain: sklearn/linear_model/_sgd_fast.cpython-312-x86_64-linux-gnu.so.p/sklearn/linear_model/_sgd_fast.pyx.c is dirty
ninja explain: sklearn/linear_model/_sgd_fast.cpython-312-x86_64-linux-gnu.so.p/sklearn/linear_model/_sgd_fast.pyx.c is dirty
ninja explain: sklearn/linear_model/_sgd_fast.cpython-312-x86_64-linux-gnu.so.p/meson-generated_sklearn_linear_model__sgd_fast.pyx.c.o is dirty
ninja explain: sklearn/linear_model/_sgd_fast.cpython-312-x86_64-linux-gnu.so is dirty
ninja explain: restat of output sklearn/linear_model/_sag_fast.cpython-312-x86_64-linux-gnu.so.p/sklearn/linear_model/_sag_fast.pyx.c older than most recent input /home/lesteve/dev/scikit-learn/build/cp312/sklearn/utils/_typedefs.pxd (1713170314809031570 vs 1713170351662119783)
ninja explain: sklearn/linear_model/_sag_fast.cpython-312-x86_64-linux-gnu.so.p/sklearn/linear_model/_sag_fast.pyx.c is dirty
ninja explain: sklearn/linear_model/_sag_fast.cpython-312-x86_64-linux-gnu.so.p/sklearn/linear_model/_sag_fast.pyx.c is dirty
ninja explain: sklearn/linear_model/_sag_fast.cpython-312-x86_64-linux-gnu.so.p/meson-generated_sklearn_linear_model__sag_fast.pyx.c.o is dirty
ninja explain: sklearn/linear_model/_sag_fast.cpython-312-x86_64-linux-gnu.so is dirty
ninja explain: restat of output sklearn/neighbors/_ball_tree.cpython-312-x86_64-linux-gnu.so.p/sklearn/neighbors/_ball_tree.pyx.c older than most recent input /home/lesteve/dev/scikit-learn/build/cp312/sklearn/utils/_typedefs.pxd (1713170317099016342 vs 1713170351662119783)
ninja explain: sklearn/neighbors/_ball_tree.cpython-312-x86_64-linux-gnu.so.p/sklearn/neighbors/_ball_tree.pyx.c is dirty
ninja explain: sklearn/neighbors/_ball_tree.cpython-312-x86_64-linux-gnu.so.p/sklearn/neighbors/_ball_tree.pyx.c is dirty
ninja explain: sklearn/neighbors/_ball_tree.cpython-312-x86_64-linux-gnu.so.p/meson-generated_sklearn_neighbors__ball_tree.pyx.c.o is dirty
ninja explain: sklearn/neighbors/_ball_tree.cpython-312-x86_64-linux-gnu.so is dirty
ninja explain: restat of output sklearn/neighbors/_kd_tree.cpython-312-x86_64-linux-gnu.so.p/sklearn/neighbors/_kd_tree.pyx.c older than most recent input /home/lesteve/dev/scikit-learn/build/cp312/sklearn/utils/_typedefs.pxd (1713170317259015279 vs 1713170351662119783)
ninja explain: sklearn/neighbors/_kd_tree.cpython-312-x86_64-linux-gnu.so.p/sklearn/neighbors/_kd_tree.pyx.c is dirty
ninja explain: sklearn/neighbors/_kd_tree.cpython-312-x86_64-linux-gnu.so.p/sklearn/neighbors/_kd_tree.pyx.c is dirty
ninja explain: sklearn/neighbors/_kd_tree.cpython-312-x86_64-linux-gnu.so.p/meson-generated_sklearn_neighbors__kd_tree.pyx.c.o is dirty
ninja explain: sklearn/neighbors/_kd_tree.cpython-312-x86_64-linux-gnu.so is dirty

The issue had originally been noticed by @ogrisel and @glemaitre inside a notebook (but is not notebook-related as the reproducer above show):

• the first import sklearn builds plenty of stuff, because switching between branches updates file timestamps (and ninja decides what to rebuild based on timestamp)
• the notebook then uses an estimator with n_jobs>1 and you can see again things rebuilding

[lesteve]

I believe I have a simpler reproducer for this in https://github.com/lesteve/meson-partial-build.

It seems to happen only when the pyx is a custom_target generated from a pyx.tp file somehow. I am going to wild-guess that the Meson handling of dependencies for Cython files is not correct in this case ...

[lorentzenchr]

I experienced the same strange behavior.

[glemaitre]

In pandas they had a similar issue with generated .pxi files: mesonbuild/meson-python#589. It requires to import twice. Wondering if the fix could be helping here.