nunit-agent.exe not terminating form time to time

[michael77777777777]

Error Description
We have a gitlab-runner Windows service installed on our "Windows11 Pro 22H2" build machine. This service allows up to 4 parallel gitlab-pipelines. In each gitlab-pipeline we call nunit3-console.exe 3.16.3 to execute our test assembly with nunit 3.13.3.

From time to time we observe that the nunit-agent.exe process ist not terminated.
Using Microsofts "ProcessExplorer" tool to analyze the stuck nunit-agent.exe process shows that its parent process is not existing anymore. Moreover, when switching to the "Threads" tab in ProcessExplorer, a Windows internal process lifecycle-check must be triggered as the stuck nunit-agent.exe terminates immediately.

In MsBuild file, we call nunit3-console.exe with the following parameters_
nunit3-console.exe $(testAssemblyList) --dispose-runners --skipnontestassemblies --framework="net-4.5" --where="$(nunit_where)" --encoding=utf-8 --labels=all --result=TestResult.xml --result=junit-results.xml;transform=nunit3-junit.xslt

Remarks:

• the bug occurs rarely and we did not find any possibility to reproduce it. We observed a higher occurrence of the bug when gitlab runners were executed parallel
• there is only one test assembly, so executing with flag "inprocess" has no effects
• our tests do not use threads

Logging:
Gitlab console logfile shows that all tests have been executed successfully before process get stuck

LAST UNIT TEST IN THE LOGFILE

Run Settings
RequestedRuntimeFramework: net-4.5
DisposeRunners: True
SkipNonTestAssemblies: True
WorkDirectory: C:\Tools\gitlab-runner\builds\moKKEX1Y\0******
ImageRuntimeVersion: 4.0.30319
ImageRequiresX86: False
ImageRequiresDefaultAppDomainAssemblyResolver: False
TargetRuntimeFramework: net-4.5
NumberOfTestWorkers: 20

Test Run Summary
Overall result: Passed
Test Count: 293, Passed: 293, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0
Start time: 2023-07-05 07:08:52Z
End time: 2023-07-05 07:23:51Z
Duration: 898.792 seconds

Results (nunit3) saved as TestResult.xml
Results (user) saved as junit-results.xml

NUnit.Engine.NUnitEngineUnloadException : Agent Process was terminated successfully after error.
----> NUnit.Engine.NUnitEngineUnloadException : Unable to unload application domain: unload thread timed out after 30 seconds.
Application domain name: domain-
Application domain BaseDirectory: C:\Tools\gitlab-runner\builds\moKKEX1Y\0*******

Assumption:
Due to the behavior of ProcessExplorer this seems to be a threading issue. Maybe due to open TCP-connections?

Machine configuration:

• windows 11 pro 22h2 build server
• gitlab-runner 15.9.1 windows service with up to 4 parallel gitlab-pipelines
• nunit3-console.exe 3.16.3 from NuGet
• nunit 3.13.3 from NuGet

Any help is highly appreciated. Thank you in advance!

[michael77777777777]

Is there really nobody who can help with this issue? Still facing this very anoying problem.

[OsirisTerje]

There are multiple of these issues reported, but we do lack a usable repro to be able to figure out what is wrong.

[michael77777777777]

Thanks for the reply @OsirisTerje . I understand your situation. I try to figure out more background information the next time it occurrs. Maybe something new catches my eyes...