test suite unreliable since v1.2.0

[decathorpe]

Fedora package builds since v1.2.0 was released have been very unreliable. The test suite seems to fail in roughly 20% of test runs with various wild error messages, particularly from doctests:

   Doc-tests autocfg
     Running `/usr/bin/rustdoc --edition=2015 --crate-type lib --crate-name autocfg --test src/lib.rs --test-run-directory /builddir/build/BUILD/autocfg-1.3.0 -L dependency=/builddir/build/BUILD/autocfg-1.3.0/target/rpm/deps -L dependency=/builddir/build/BUILD/autocfg-1.3.0/target/rpm/deps --extern autocfg=/builddir/build/BUILD/autocfg-1.3.0/target/rpm/deps/libautocfg-0a1cc91a91fce621.rlib -C embed-bitcode=no --error-format human`
running 9 tests
test src/lib.rs - AutoCfg::probe_constant (line 450) ... ignored
test src/lib.rs - AutoCfg::probe_expression (line 432) ... ignored
test src/lib.rs - AutoCfg::probe_path (line 348) ... ignored
test src/lib.rs - AutoCfg::probe_sysroot_crate (line 329) ... ignored
test src/lib.rs - AutoCfg::probe_trait (line 376) ... ignored
test src/lib.rs - AutoCfg::probe_type (line 404) ... ignored
test src/lib.rs - (line 18) ... ok
test src/lib.rs - AutoCfg::probe_raw (line 291) ... FAILED
test src/lib.rs - AutoCfg::probe_raw (line 306) ... ok
failures:
---- src/lib.rs - AutoCfg::probe_raw (line 291) stdout ----
Test executable failed (exit status: 101).
stderr:
error: could not copy "target/probe1.probe1.a23b9cb2d7aa1a5f-cgu.0.rcgu.ll" to "target/probe1.ll": No such file or directory (os error 2)
error: aborting due to 1 previous error
thread 'main' panicked at src/lib.rs:7:1:
assertion failed: ac.probe_raw("#![no_builtins]").is_ok()
stack backtrace:
   0:     0x555a7d0debf2 - std::backtrace_rs::backtrace::libunwind::trace::h5f115adc35edca5e
                               at /builddir/build/BUILD/rustc-1.78.0-src/library/std/src/../../backtrace/src/backtrace/libunwind.rs:105:5
   1:     0x555a7d0debf2 - std::backtrace_rs::backtrace::trace_unsynchronized::h66ac6b5395987b96
                               at /builddir/build/BUILD/rustc-1.78.0-src/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x555a7d0debf2 - std::sys_common::backtrace::_print_fmt::h6baab85b78ec79f3
                               at /builddir/build/BUILD/rustc-1.78.0-src/library/std/src/sys_common/backtrace.rs:68:5
   3:     0x555a7d0debf2 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h8dd8d16e5e834e56
                               at /builddir/build/BUILD/rustc-1.78.0-src/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x555a7d10089c - core::fmt::rt::Argument::fmt::h5821514b45bde58f
                               at /builddir/build/BUILD/rustc-1.78.0-src/library/core/src/fmt/rt.rs:142:9
   5:     0x555a7d10089c - core::fmt::write::hd321594fe736ddf1
                               at /builddir/build/BUILD/rustc-1.78.0-src/library/core/src/fmt/mod.rs:1153:17
   6:     0x555a7d0dc92f - std::io::Write::write_fmt::h752ef8cc7d69adde
                               at /builddir/build/BUILD/rustc-1.78.0-src/library/std/src/io/mod.rs:1843:15
   7:     0x555a7d0de9c4 - std::sys_common::backtrace::_print::h0007831260368354
                               at /builddir/build/BUILD/rustc-1.78.0-src/library/std/src/sys_common/backtrace.rs:47:5
   8:     0x555a7d0de9c4 - std::sys_common::backtrace::print::h19e358810ed43c0a
                               at /builddir/build/BUILD/rustc-1.78.0-src/library/std/src/sys_common/backtrace.rs:34:9
   9:     0x555a7d0dfe7b - std::panicking::default_hook::{{closure}}::h149c09f4c91ac56c
  10:     0x555a7d0dfbd3 - std::panicking::default_hook::ha45b6f737d435c1e
                               at /builddir/build/BUILD/rustc-1.78.0-src/library/std/src/panicking.rs:292:9
  11:     0x555a7d0e031d - std::panicking::rust_panic_with_hook::haa066639b227c0f7
                               at /builddir/build/BUILD/rustc-1.78.0-src/library/std/src/panicking.rs:779:13
  12:     0x555a7d0e01b9 - std::panicking::begin_panic_handler::{{closure}}::h0bd6bf33af00b175
                               at /builddir/build/BUILD/rustc-1.78.0-src/library/std/src/panicking.rs:649:13
  13:     0x555a7d0df0c6 - std::sys_common::backtrace::__rust_end_short_backtrace::h1a5f92ec76439a12
                               at /builddir/build/BUILD/rustc-1.78.0-src/library/std/src/sys_common/backtrace.rs:171:18
  14:     0x555a7d0dff24 - rust_begin_unwind
                               at /builddir/build/BUILD/rustc-1.78.0-src/library/std/src/panicking.rs:645:5
  15:     0x555a7d0ba0a5 - core::panicking::panic_fmt::h9d6211e5dc5960e4
                               at /builddir/build/BUILD/rustc-1.78.0-src/library/core/src/panicking.rs:72:14
  16:     0x555a7d0ba163 - core::panicking::panic::h90be27f96fc60da6
                               at /builddir/build/BUILD/rustc-1.78.0-src/library/core/src/panicking.rs:145:5
  17:     0x555a7d0bb74c - rust_out::main::_doctest_main_src_lib_rs_291_0::h2218db1b7c4606ce
  18:     0x555a7d0bb646 - rust_out::main::h3215e911437994eb
  19:     0x555a7d0bace3 - core::ops::function::FnOnce::call_once::hab37cc49c02b0940
  20:     0x555a7d0ba756 - std::sys_common::backtrace::__rust_begin_short_backtrace::h9c3ad854545ff677
  21:     0x555a7d0babc9 - std::rt::lang_start::{{closure}}::h7b78b3ddce1acd98
  22:     0x555a7d0d9ba3 - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::h09c601785505d569
                               at /builddir/build/BUILD/rustc-1.78.0-src/library/core/src/ops/function.rs:284:13
  23:     0x555a7d0d9ba3 - std::panicking::try::do_call::hbfb70692164e52b9
                               at /builddir/build/BUILD/rustc-1.78.0-src/library/std/src/panicking.rs:552:40
  24:     0x555a7d0d9ba3 - std::panicking::try::hfb29355d55869458
                               at /builddir/build/BUILD/rustc-1.78.0-src/library/std/src/panicking.rs:516:19
  25:     0x555a7d0d9ba3 - std::panic::catch_unwind::h7ca6e679fd2bbbef
                               at /builddir/build/BUILD/rustc-1.78.0-src/library/std/src/panic.rs:146:14
  26:     0x555a7d0d9ba3 - std::rt::lang_start_internal::{{closure}}::hdf4dc303ca789fc0
                               at /builddir/build/BUILD/rustc-1.78.0-src/library/std/src/rt.rs:148:48
  27:     0x555a7d0d9ba3 - std::panicking::try::do_call::heeb94d5f180f6e63
                               at /builddir/build/BUILD/rustc-1.78.0-src/library/std/src/panicking.rs:552:40
  28:     0x555a7d0d9ba3 - std::panicking::try::hf9ecce418a9a6dbf
                               at /builddir/build/BUILD/rustc-1.78.0-src/library/std/src/panicking.rs:516:19
  29:     0x555a7d0d9ba3 - std::panic::catch_unwind::hb943d40d7fe0efe6
                               at /builddir/build/BUILD/rustc-1.78.0-src/library/std/src/panic.rs:146:14
  30:     0x555a7d0d9ba3 - std::rt::lang_start_internal::h2c0733911c428122
                               at /builddir/build/BUILD/rustc-1.78.0-src/library/std/src/rt.rs:148:20
  31:     0x555a7d0baba7 - std::rt::lang_start::h4af1243a7e175255
  32:     0x555a7d0bb795 - main
  33:     0x7f7c4bd9f1c8 - __libc_start_call_main
  34:     0x7f7c4bd9f28b - __libc_start_main_alias_1
  35:     0x555a7d0ba685 - _start
  36:                0x0 - <unknown>
failures:
    src/lib.rs - AutoCfg::probe_raw (line 291)
test result: FAILED. 2 passed; 1 failed; 6 ignored; 0 measured; 0 filtered out; finished in 0.38s
error: doctest failed, to rerun pass `--doc`
error: 1 target failed:
    `--doc`

I have seen failures like this now on x86_64, ppc64le, and s390x, so it does not appear to be architecture-specific, but more likely a race condition.

I can try running tests single-threaded if that helps?

[cuviper]

I can try running tests single-threaded if that helps?

I'll bet it will -- I suspect the reason is that the doc tests are compiled as separate binaries, so they're not sharing the static ID counter, but if they run in parallel they'll probably clobber each other.

The doc tests are not very interesting anyway, so I think you'd also be fine with cargo test --lib --tests.

[decathorpe]

I'll try that, thanks!

[decathorpe]

Looks like running tests with a single thread solves the issue 👍🏼