Windows 2019 Agent Is Affected By PowerShell Debugger Regression

[aolszowka]

Description

Background

We are lifting a self hosted build agent and replacing it with a Microsoft Agent. Our pipeline is testing and packaging a PowerShell 5.1 Module, a very simplified workflow looks like this:

• Check out Code
• Run PowerShell Script Analyzer
• Run Pester Tests (With Code Coverage)
• Publish to NuGet Feed

When attempting to lift this agent it has been discovered that our Pester Tests run an order of magnitude slower (45 Minutes vs 3 minutes).

Here is a screenshot of the pipeline when using windows-2019:

Virtual environments affected

• Ubuntu 18.04
• Ubuntu 20.04
• macOS 10.15
• macOS 11
• Windows Server 2016
• Windows Server 2019
• Windows Server 2022

Image version and build link

Virtual Environment
Environment: windows-2019
Version: 20211018.0
Included Software: https://github.com/actions/virtual-environments/blob/win19/20211018.0/images/win/Windows2019-Readme.md
Image Release: https://github.com/actions/virtual-environments/releases/tag/win19%2F20211018.0

Is it regression?

No response

Expected behavior

For Pester Tests to run in a reasonable amount of time

Actual behavior

No response

Repro steps

Research

Performing research on this has shown that we are most likely affected by this Pester Issue (pester/Pester#1318). HOWEVER it is not Pester's bug, rather the root cause is this PowerShell Issue (PowerShell/PowerShell#10269 (comment)) which caused a large regression in the performance of debug breakpoints (Pester leverages these to provide code coverage).

As per the PowerShell issue this was backported to PowerShell 5.1 and fixed, HOWEVER it was only fixed on Windows 20H1, unfortunately the build agents appear to have 10.0.17763 (aka 1809) which does not have this fix.

This is a huge ask I realize, but this hurts this workflow incredibly.

Work Arounds

I have confirmed that when using windows-2022 this issue is remediated:

However as 2022 is not in General Availability yet, it is not recommended for production workloads (which basically means we can't use it).

[dibir-magomedsaygitov]

Hello @aolszowka. Thank you for your report. Could you please provide us link to the builds and yml file with repro steps?

[miketimofeev]

@aolszowka the windows-2022 will be GA very soon after the stable Visual Studio 2022 release on November, 8th. I'm not sure we can do anything with the windows-2019 as we always use the latest version available.

[aolszowka]

@miketimofeev If we're going to hit GA that soon I'll move efforts to using that agent and advocate internally for it.

@dibir-magomedsaygitov Unfortunately I am unable to share the code base. In addition this issue is more pronounced with large code bases, as the number of breakpoints increase so does the time, meaning that it is non-trivial to create a simple reproducing test case. As per the linked Pester Issue it seems like the Pester Project itself was able to reproduce it with their code base which is open source. However I cannot commit to performing comparisons between windows-2019 and windows-2022.

I think the take away for us is get to windows-2022 and close this issue as wontfix and hope the next guy looking for the issue comes across this?

[miketimofeev]

@aolszowka thanks for all the data. I would prefer to close this issue as an "external" one as Pester folks are aware of the problems and will probably fix this in the future.
Please feel free to contact us if you have any concerns.
Thank you!