who tests the tester?

[nikomatsakis]

So, by and large, compiletest is untested, as far as I know. That is -- there are no 'self tests' to make sure that it's acting as it should, invoking revisions the way we expect, and so forth. This could be a bit of a tricky thing to do, but it's definitely worth the effort.

For example, an early version of #47605 had a subtle bug where it appeared to be working, but in fact was not passing the revision info through to the final test. I don't think this would have bothered travis one bit, as that would have happily run the same revision over and over.

cc @spastorino @oli-obk ... we need like a compiletest team, don't why? :)

[oli-obk]

I have some beef with compiletest in general. Instead of having this unstable (as in nightly) library that contains a somewhat random list of test kinds that all behave subtly different, we should probably write a stable extensible library that is distributed via crates.io, developed like any other library (so it has tests) and uses cargo instead of manually calling rustc and hoping for the best.

So, the list of issues off the top of my head:

• nightly only
• rustc submodules need to use the crates.io version, which breaks with rustc changes
• using dependencies in tests frequently causes "multiple matching crates" because of old compilation artifacts in the target direcory
• not extensible with new test kinds (cc @killercup for suggestion tests)

[spastorino]

I'd be happy to work on a library like this if we can have a conversation about what is exactly needed :).

[killercup]

@spastorino I started working on a very simple version of this to test clippy's diagnostics output as well as that its suggestions work as intended (when applied by rustfix). I'll have something up in later today, and will ping you in the clippy PR :)

In the meantime, feel free to say hello in the #clippy channel on irc.mozilla.org :)

[nikomatsakis]

I have some beef with compiletest in general.

While I don't disagree, I feel a bit wary here. I'm fine with making a new testing setup, but it feels like the kind of thing that will derail progress for a long time. And I'm not convinced that the compiler's needs (e.g., things like mir-opt tests) will be satisfied by a more generic setup.

Unless you have a clear candidate of what test runner we should adopt, I'd rather see us move towards simplifying and centralizing things gradually and then at some later point perhaps switch. (Centralizing run-pass, compile-fail, and ui seems like an example of something we can do to simplify things now.)

[steveklabnik]

Triage; Looks like Pietro has added a bit of testing, but it's "just a start"

[jyn514]

One way to test compiletest is to add more UI tests that test compiletests' behavior, instead of rustc's. See for example https://github.com/rust-lang/rust/pull/103298/files#diff-6542cbbb7ad12df6cb2daadaf279589c5c1712906cd14222c03132ad5a49d590. I think using that instead of a new framework has several advantages:

• No need to set up a new testing framework
• Existing contributors are already familiar with UI tests
• These double as "integration tests" since they test exactly the property we're interested in, how compiletest runs a UI test from beginning to end

I think it should be fairly simple to keep adding tests to src/test/ui/compiletest-selftest as necessary; I don't have a set of initial tests in mind though.

[oli-obk]

I wrote a new ui test framework that miri uses and I'm working on getting ready for clippy to use. It uses itself to test its own output.

[jyn514]

Sounds like there is duplicate work between that and #103266

[Alexendoo]

It could well be, mainly I went forward with #103266 to avoid the other duplicate work of adding things to https://github.com/Manishearth/compiletest-rs that the in tree one already has. For example the JSON output after every test failure is pretty rough