Meson (ninja) fails building the FluidSynth wrap under MSYS2

[kcgen]

Yesterday's build ran fine

2022-06-16T12:54:38.2327740Z ninja: Entering directory `build'
2022-06-16T12:54:38.2333423Z [1/469] Compiling C object subprojects/fluidsynth-2.2.6/src/make_tables.exe.p/gentables_gen_conv.c.obj
2022-06-16T12:54:38.2375056Z [2/469] Compiling C object subprojects/fluidsynth-2.2.6/src/make_tables.exe.p/gentables_gen_rvoice_dsp.c.obj
2022-06-16T12:54:38.2379525Z [3/469] Compiling C object subprojects/fluidsynth-2.2.6/src/make_tables.exe.p/gentables_make_tables.c.obj
2022-06-16T12:54:38.2387116Z [4/469] Compiling C object subprojects/fluidsynth-2.2.6/test/test_sfont_loading.exe.p/test_sfont_loading.c.obj
2022-06-16T12:54:38.6339396Z [5/469] Linking target subprojects/fluidsynth-2.2.6/src/make_tables.exe
2022-06-16T12:54:38.6853370Z [6/469] Compiling C object subprojects/fluidsynth-2.2.6/test/test_sample_rate_change.exe.p/test_sample_rate_change.c.obj
2022-06-16T12:54:38.6930701Z ../subprojects/fluidsynth-2.2.6/test/test_sample_rate_change.c: In function 'main':
2022-06-16T12:54:38.6956329Z ../subprojects/fluidsynth-2.2.6/test/test_sample_rate_change.c:55:5: warning: 'fluid_synth_set_sample_rate' is deprecated [-Wdeprecated-declarations]
2022-06-16T12:54:38.7065535Z    55 |     fluid_synth_set_sample_rate(synth, sample_rate);
2022-06-16T12:54:38.7262932Z       |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~
2022-06-16T12:54:38.8295345Z In file included from subprojects/fluidsynth-2.2.6/include/fluidsynth.h:100,
2022-06-16T12:54:38.8893570Z                  from ../subprojects/fluidsynth-2.2.6/test/test_sample_rate_change.c:3:
2022-06-16T12:54:38.8984741Z ../subprojects/fluidsynth-2.2.6/include/fluidsynth/synth.h:253:38: note: declared here
2022-06-16T12:54:38.9752600Z   253 | FLUID_DEPRECATED FLUIDSYNTH_API void fluid_synth_set_sample_rate(fluid_synth_t *synth, float sample_rate);
2022-06-16T12:54:39.0282681Z       |                                      ^~~~~~~~~~~~~~~~~~~~~~~~~~~
2022-06-16T12:54:41.1648845Z [7/469] Generating subprojects/fluidsynth-2.2.6/src/gen_tables with a custom command (wrapped by meson to set PATH)
2022-06-16T12:54:41.1674673Z [8/469] Compiling C object subprojects/fluidsynth-2.2.6/test/test_preset_sample_loading.exe.p/test_preset_sample_loading.c.obj
2022-06-16T12:54:41.2235107Z [9/469] Compiling C object subprojects/fluidsynth-2.2.6/test/test_sample_cache.exe.p/test_sample_cache.c.obj
2022-06-16T12:54:41.2259943Z [10/469] Compiling C object subprojects/fluidsynth-2.2.6/libfluidsynth.dll.p/src_bindings_fluid_ladspa.c.obj
2022-06-16T12:54:41.2423788Z [11/469] Compiling C object subprojects/fluidsynth-2.2.6/libfluidsynth.dll.p/src_drivers_fluid_adriver.c.obj
2022-06-16T12:54:45.6302983Z [12/469] Compiling C object subprojects/fluidsynth-2.2.6/libfluidsynth.dll.p/src_drivers_fluid_mdriver.c.obj
2022-06-16T12:54:45.6322838Z [13/469] Compiling C object subprojects/fluidsynth-2.2.6/libfluidsynth.dll.p/src_bindings_fluid_cmd.c.obj
2022-06-16T12:54:45.6859614Z [14/469] Compiling C object subprojects/fluidsynth-2.2.6/libfluidsynth.dll.p/src_midi_fluid_midi_router.c.obj
2022-06-16T12:54:45.6882463Z [15/469] Compiling C object subprojects/fluidsynth-2.2.6/libfluidsynth.dll.p/src_bindings_fluid_filerenderer.c.obj
2022-06-16T12:54:45.6944617Z [16/469] Compiling C object subprojects/fluidsynth-2.2.6/libfluidsynth.dll.p/src_midi_fluid_midi.c.obj

... etc ...

Ref: https://pipelines.actions.githubusercontent.com/serviceHosts/2382ad32-0206-4e79-b5d8-85555a77d3b8/_apis/pipelines/1/runs/65738/signedlogcontent/3?urlExpires=2022-06-18T00%3A17%3A07.8193620Z&urlSigningMethod=HMACV1&urlSignature=9eFywpFaHHICysNA4HiRlezQxrRM%2B79uwHn1lj9bN3k%3D

good_run.txt

Today's build fails: meson --internal exe --unpickle ...

2022-06-17T23:41:12.8036928Z + ninja -C build
2022-06-17T23:41:12.8163313Z + tee build.log
2022-06-17T23:41:15.9098452Z ninja: Entering directory `build'
2022-06-17T23:41:15.9118586Z [1/469] Compiling C object subprojects/fluidsynth-2.2.6/src/make_tables.exe.p/gentables_gen_conv.c.obj
2022-06-17T23:41:15.9206153Z [2/469] Compiling C object subprojects/fluidsynth-2.2.6/src/make_tables.exe.p/gentables_make_tables.c.obj
2022-06-17T23:41:15.9215675Z [3/469] Compiling C object subprojects/fluidsynth-2.2.6/src/make_tables.exe.p/gentables_gen_rvoice_dsp.c.obj
2022-06-17T23:41:15.9232393Z [4/469] Linking target subprojects/fluidsynth-2.2.6/src/make_tables.exe
2022-06-17T23:41:16.3484466Z [5/469] Compiling C object subprojects/fluidsynth-2.2.6/test/test_sfont_loading.exe.p/test_sfont_loading.c.obj
2022-06-17T23:41:16.4136630Z [6/469] Compiling C object subprojects/fluidsynth-2.2.6/test/test_sample_cache.exe.p/test_sample_cache.c.obj
2022-06-17T23:41:16.5184343Z [7/469] Generating subprojects/fluidsynth-2.2.6/src/gen_tables with a custom command (wrapped by meson to set PATH)
2022-06-17T23:41:16.6074698Z FAILED: subprojects/fluidsynth-2.2.6/src/fluid_conv_tables.inc.h subprojects/fluidsynth-2.2.6/src/fluid_rvoice_dsp_tables.inc.h 
2022-06-17T23:41:19.0644986Z "D:/a/_temp/msys64/mingw64/bin/meson" "--internal" "exe" "--unpickle" "D:/a/dosbox-staging/dosbox-staging/build/meson-private/meson_exe_make_tables_76da7216b7ad891a922e4f7241616b5b130aafe7.dat"

Ref: https://pipelines.actions.githubusercontent.com/serviceHosts/2382ad32-0206-4e79-b5d8-85555a77d3b8/_apis/pipelines/1/runs/65897/signedlogcontent/3?urlExpires=2022-06-18T00%3A14%3A12.0587752Z&urlSigningMethod=HMACV1&urlSignature=ElrZPk617OICLlQaNrh9b4CAQnoRDN%2FBxo9UDdThYEk%3D

failed_run.txt

[kcgen]

@eli-schwartz , we haven't touched anything to do with Meson, FluidSynth, or the wraps between these builds.

Curious if you have any insight into what could be happening?

Both workflows are running Meson 0.62.2.

@shermp has confirmed the build works (from scratch) under his local MSYS2 environment, which is also running Meson 0.62.2.

Much thanks for any advice!

[eli-schwartz]

I know that @lazka saw similar mysterious failures with msys2. It's just build commands that execute meson's entry point to do work (i.e. --internal commands), apparently.

See e.g. msys2/MINGW-packages#11864

[eli-schwartz]

The last time I saw issues like this it was the fault of the "meson.exe" Windows wrapper launcher (that handles the windows specifics of executing python with the *.py script that is the real program entry point).

It was a really annoying thing to debug because somehow it only occurred when launching meson.exe from ninja and inside a CI environment.

[shermp]

Hmm, I wonder, does invoking ninja -C <build_dir> vs meson compile -C <build_dir> make any difference? I do the latter on my local builds, but the CI is currently doing the former.

[kcgen]

It was a really annoying thing to debug because somehow it only occurred when launching meson.exe from ninja and inside a CI environment.

Indeed. Troubleshooting CI-only failures is quite a nightmare.

Hmm, I wonder, does invoking ninja -C <build_dir> vs meson compile -C <build_dir> make any difference? I do the latter on my local builds, but the CI is currently doing the former.

Good eye @shermp , let's try: https://github.com/dosbox-staging/dosbox-staging/actions/runs/2518694201

[eli-schwartz]

Meson compile will just scan the build directory, detect that it uses ninja, and run ninja.

It exists solely for muscle memory, so that people don't have to think about whether the command is instead msbuild or xcodebuild.

If ninja has problems, running it as a subprocess of meson compile won't make those problems go away.

[kcgen]

@lazka mentioned:

This happens randomly, it sometimes works, sometimes doesn't in CI.

Maybe we need a while (failed) try again wrapper? GitHub gives us 6 hours per workflow, so that should be enough time to squeeze out a passing run 😮‍💨

[kcgen]

If ninja has problems, running it as a subprocess of meson compile won't make those problems go away.

... and confirmed: https://github.com/dosbox-staging/dosbox-staging/runs/6944628701?check_suite_focus=true failed just the same w/ meson compile -C ...

[kcgen]

Maybe we need a while (failed) try again wrapper?

Let's put this to the test. Brute-force, for the win 🤦‍♂️ bb3dd1d

https://github.com/dosbox-staging/dosbox-staging/actions/runs/2518727218

[shermp]

Maybe we need a while (failed) try again wrapper?

Let's put this to the test. Brute-force, for the win 🤦‍♂️ bb3dd1d

https://github.com/dosbox-staging/dosbox-staging/actions/runs/2518727218

Not pretty, but it did work that time.

[kcgen]

What's strange is that none of them needed the repeat attempts.

.. just the threat of do-overs was sufficient 😆

[lazka]

Sorry for the breakage :/

[eli-schwartz]

.. just the threat of do-overs was sufficient 😆

This is somehow amazing and hilarious. I wonder what that is even doing.