Split SQL Server functional tests into two projects

[ajcvickers]

Improves total test execution time by:

• 7:32 to 6:49 on my old laptop (10%)
• 4:06 to 2:46 on my dev box (48%)

This is for all tests, excluding Cosmos. That is, 128,062 passed and 1,482 skipped.

[ajcvickers]

Will fix this if we decide to take this change.

[AndriySvyryd]

This isn't the way I'd like us to go, as it creates an "arbitrary" split in our tests. But, on the other hand, it's a very straightforward way of making them faster. So, unless we stop using xunit, this looks like the most practical approach.

[AndriySvyryd]

Also, on my devbox it's less than 10% improvement.

As an alternative, can you try adding this to Directory.Build.props and see whether it results in an improvement for you? (you'd probably need to tweak the value)
<TestRunnerAdditionalArguments>-maxthreads 128</TestRunnerAdditionalArguments>

[roji]

This isn't the way I'd like us to go, as it creates an "arbitrary" split in our tests.

I generally agree... It's a cool thing, but given that our tests already run really fast I'm not sure it's worth it...

So, unless we stop using xunit, this looks like the most practical approach.

Another idea if we think reducing test runtime is valuable enough - we could try to fully understand what exactly is going on inside that makes the tests run faster when split across assemblies (I think at this point it's mainly an empirical thing we don't actually understand). This may lead us to tweaks which could improve the perf without splitting the assembly. I also saw some tooling at some point which is designed specifically to show the concurrency inside your test sessions, e.g. as a way to locate stragglers, problematic tests, etc.

[ajcvickers]

it creates an "arbitrary" split in our tests.

What if I could come up with a reason to do a non-arbitrary split? Like an advantage of having all the query tests in one place, and all the non-query tests in another place? :-)

Another thought. What if I am able to do make it continue to be a single test except with a certain solution/filter where it runs as two?

I think at this point it's mainly an empirical thing we don't actually understand

Yes, totally.

[ajcvickers]

I think at this point it's mainly an empirical thing we don't actually understand.

I could try to create a simple repro without tests that either uses one process or multiple processes to slam the database.

[roji]

One additional thought on this - isn't there a risk of the two SQL Server test projects running in parallel and stepping over each other's toes, e.g. both trying to create the Northwind database at the same time etc.? Could be another reason to not split...