Unable to specify custom test adapter path

[07souravkunda]

Description

I am trying to create a custom test adapter which is intended to run before nunit3.testadapter. I have a requirement of executing tests with custom test adapter and for some conditions tests should run using nunit3.testadapter executor something like this.

I tried out the following solutions:

1. dotnet test --test-adapter-path C:\Adapters
2. vstest.console.exe dll_path /TestAdapterPath:"C:\Adapters"
3. Created a .runsettings file with RunConfiguration.TestAdaptersPaths node as given below and specified this runsettings path to dotnet test

<RunSettings>
  <RunConfiguration>
    <TestAdaptersPaths>C:\Adapters;</TestAdaptersPaths>
  </RunConfiguration>
</RunSettings>

but every time *.TestAdapter.dll from source dll directory is getting picked.

The custom test adapter should be picked in an ideal scenario, but this isn't working. Can someone please help me to achieve this?

Steps to reproduce

Expected behavior

Custom test adapter should be picked.

Actual behavior

*.TestAdapter.dll at source dll directory is picked.

Diagnostic logs

Environment

[MarcoRossignoli]

@Haplois do you think that the extension loading strategy you've implemented here #3380 could help for this use case?

[Haplois]

@MarcoRossignoli, the TestAdapterLoadingStrategy can be used to facilitate this. If you set it to Explicit, DefaultRuntimeProviders, it should only load test adapters under the given path and required runtime providers to run the test. If you also provide the runtime provider, then you can just use Explicit.

[microsoft-github-policy-service]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 10 days.

[holbizmetrics]

Ever tried something like that?

   [ExtensionUri("executor://YourTestDiscoverer")]
   public class YourTestExecutor : ITestExecutor
   {
   }
 
    [ExtensionUri("executor://YourTestDiscoverer")]
    [DefaultExecutorUri("executor://YourTestDiscoverer")]
    [FileExtension(".dll")]
    [FileExtension(".exe")]
    [Category("managed")]
    public class YourTestDiscoverer : ITestDiscoverer
    {
    }

Because I just started to create from scratch and with some trial and error, this worked.
Now I got to work on the test discoverer implementation which is empty.

Thus, I get the message that no test is found which is correct so far.

HTH. :-)

[nohwnd]

@holbizmetrics here is a minimal implementation of a discoverer and executor, and a test framework if you need some reference. Biggest hurdle is getting the attributes right (you already got that down), and having the dll named *.TestAdapter.dll. https://github.com/nohwnd/intent

[holbizmetrics]

@nohwnd Yes. Meanwhile even the test appears in the Test Explorer in Visual Studio and one debug commands as well as the traits are working as well. The TestAdapter I already named correctly, but that was a coincidence.
Since I've called my Framework NextUnit, I called the TestAdapter was called NextUnit.TestAdapter. ;-)

What worked for me was this:

[ExtensionUri(DiscovererURI)] //[ExtensionUri("executor://NextUnitTestDiscoverer")]
DefaultExecutorUri(DiscovererURI)]//[DefaultExecutorUri("executor://NextUnitTestDiscoverer")]
[FileExtension(".dll")]
[FileExtension(".exe")]
[Category("managed")]
public class NextUnitTestDiscoverer : ITestDiscoverer
{
...

In the TestDiscoverer.

And in the TestExecutor:

/// <summary>
/// 
/// </summary>
[ExtensionUri(Definitions.DiscovererURI)] //[ExtensionUri("executor://NextUnitTestDiscoverer")]
public class NextUnitTestExecutor : NextUnitBaseExecutor, ITestExecutor
{
...

For anyone who may still encounter the same problem...

P.S.: I don't know why the formatting now looks like this considering the code. Tried to change that multiple times.
But I hope it's clearly readable anyway. Latest when you may post that in your IDE, hopefully.

[nohwnd]

Fixed your comment, you need to use ``` not ` to delimit your code when doing multiline code.

[holbizmetrics]

Firstly, to the original poster:

As far as I understand it is the normally behavior THAT *.TestAdapter.dll will be called.

*.TestAdapter.dll

But other hints have been added already.
And you guys may already even know more about this than me.

An additional info for anyone who's trying to debug a TestAdapter:

Until now, I could do the following:

/// <summary>
/// 
/// </summary>
/// <param name="tests"></param>
/// <param name="runContext"></param>
/// <param name="frameworkHandle"></param>
public void RunTests(IEnumerable<TestCase>? tests, IRunContext? runContext, IFrameworkHandle? frameworkHandle)
{
#if ADAPTER_TEST
	Debugger.Launch();
#endif
	var settings = runContext?.RunSettings?.SettingsXml;
	frameworkHandle.SendMessage(Microsoft.VisualStudio.TestPlatform.ObjectModel.Logging.TestMessageLevel.Informational, "RunTests");
	foreach (TestCase test in tests)
	{
		// Example: Mark the start of the test
		frameworkHandle.RecordStart(test);

Which is:

Adding a Debugger.Launch() statement.
Thus, when the TestRunner was running for the executor, I could debug into the executor.

The same worked for the TestDiscoverer.
I don't know if that's the right way, but that's how I did it.

And it should also be possible to obtain an XUnit debuggable package, I guess.

A hint for now:

This worked flawlessly for quite some time, even with

Visual Studio 2022 version 17.9 Release

Now I got number 10, Visual Studio 2022 version 17.10 Release

Can't seem to make Debugger.Launch() appear/have an effect anymore like before.

Anyone encountering the same problem?

This seems to be broken now, in 17.10 preview. I'll verify at home on another computer. If it will be broken after updating as well it's pretty sure.

[nohwnd]

There is a built in collection of env variables that will wait for you to attach debugger to host or runner:

VSTEST_HOST_DEBUG=1
VSTEST_RUNNER_DEBUG=1

Or if you use our Playground, or build the AttachVS project that is part of our solution, and put the exe on PATH. Then it can attach automatically back to your VS. Either the one that started the execution, or one of your choosing:

VSTEST_RUNNER_DEBUG_ATTACHVS=1
VSTEST_HOST_DEBUG_ATTACHVS=1

If you set 1 in this env variable it will take the current process and search for parent VS. If you set any other positive number, it will try to find that process ID and will try to find parent ID.

https://github.com/microsoft/vstest/blob/main/src/Microsoft.TestPlatform.Execution.Shared/WellKnownDebugEnvironmentVariables.cs

[microsoft-github-policy-service]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 10 days.