Thinking about remote testing scenarios for v3?

[riverar]

I see there's v3 progress being made, but I don't see any discussion/planning around testing on remote targets. Is this something you're considering for v3? For example, I'm authoring a cross-platform library and would like to run tests against Linux, macOS, and Windows simultaneously from Visual Studio on any of these platforms (though prefer Windows). Ideally, this would be easily configurable at project scope and Just Work(tm) with existing Visual Studio test surfaces (including Live Unit Testing).

[bradwilson]

How do you intend to launch these from Visual Studio, if you have to run them remotely? There is no gesture that I'm aware of inside Test Explorer that would allow for such a thing.

[riverar]

I think a simpler approach would be to allow for some sort of override that skips the "open local port" logic you documented above and simply connects to a given host and port (e.g. $env:XUNIT_TARGET=1.2.3.4:11223). Then some enterprising external developer can handle all the rest (deployment, firewalls, plumbing, etc.) and xunit will be none the wiser. Am I missing something?

(I'm looking at this from a very restrictive .net5/netcore C# library unit testing lens.)

[bradwilson]

This is really not the hard problem to be solved (passing a hostname in addition to port when launching a test executable). What's going to be listening on the other side? Who is going to launch this? Since this now becomes a TCP port open ostensibly to the world, what security mechanisms are required to be in place to ensure that connections are only issued from legitimate clients and can't cause the box to become hacked or DoS'd?

I know you think this is seems trivial on the surface, but it's really not.

[riverar]

I don't believe those particular problems are problems in your domain. This is dev tooling--msvsmon, windbg, and friends operate similarly and operate with an assumption that devs know a bit better.

But I digress.

Based on this discussion, it doesn't sound like my scenario fits your plans or v3 roadmap. That's okay!

I appreciate the consideration and thanks for xunit.

[bradwilson]

I still firmly believe that the solution to this problem lies entirely in the hands of VSTest, not partially. I say this because that's how it's been done historically with VSTest with devices, which is effectively a "remote" execution. No code changes were required on our side to enable that, other than plugins supporting the device platform in question; here, where the request is to support .NET Core on Linux and macOS, even that is not necessary, since our .NET Core VSTest plugin support is already operating system agnostic.

[bradwilson]

Going to mark this as answered. Please let me know if you have any further questions.