Added PowerShell real-world benchmarks #3000

TIHan · 2023-05-02T23:20:41Z

This is a snapshot of the PowerShell benchmarks found here.

We will update these as the PowerShell team updates their benchmarks.

kunalspathak · 2023-05-02T23:22:36Z

We will update these as the PowerShell team updates their benchmarks.

We want to take the startup benchmark for sure (sometime next week), but after that probably we should update it once in a release so our test history maintains stable.

kunalspathak · 2023-05-02T23:22:58Z

Can you add a small README.md?

src/benchmarks/real-world/PowerShell.Benchmarks/README.md

kunalspathak · 2023-05-03T03:39:06Z

@LoopedBard3 @cincuranet @sblom

cincuranet · 2023-05-03T08:29:32Z

@kunalspathak Do you want to setup runs, auto-filing, etc.?

kunalspathak · 2023-05-03T15:04:32Z

@kunalspathak Do you want to setup runs, auto-filing, etc.?

yes. We might want to add "startup" benchmark to the real world dashboard (when we add it).

LoopedBard3

Generally looks good, just a few questions to answer before merging. Also, there are no runs specifically setup currently. We can add the runs in a separate PR or in this PR, an example for adding the runs are https://github.com/dotnet/performance/blob/main/azure-pipelines.yml#L225-L238 for public runs, and https://github.com/dotnet/performance/blob/main/azure-pipelines.yml#L452-L467 for internal runs. Let me know and I can also help out for getting the runs setup 👍.

LoopedBard3 · 2023-05-04T20:18:08Z

src/benchmarks/real-world/PowerShell.Benchmarks/Engine.Compiler.cs

+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+#if NET6_0


Do we want to benchmark these benchmarks on newer dotnet versions than Net6.0?

Good catch, yes we probably want to run these on versions newer than net6.0.

@LoopedBard3 given these benchmarks are going to be copied verbatim from https://github.com/powershell/powershell - I think it's fine we can leave these benchmarks to be NET6.0 for now. The PowerShell team can update these benchmarks to allow newer versions.

Just to make sure that I understand. In our lab, we always run against current which is .net8. Leaving it as .NET 6 will not execute this benchmark?

That is correct as a whole. Benchmarks in the main branch are only run against whatever latest is. Each new release we make a branch to correspond to the previous release and hold the tests to be run against that release while it is in support, but we don't generally backport new tests to these branches.

It shouldn't. If the project is compiled to a target other than NET6, then these benchmarks will not be included.

src/benchmarks/real-world/PowerShell.Benchmarks/Categories.cs

LoopedBard3 · 2023-05-10T17:37:44Z

@cincuranet @caaavik-msft any reason not to add these as a new runkind (powershell)?

cincuranet · 2023-05-10T17:41:39Z

This has my full support.

Add the runs to the yaml files.

LoopedBard3 · 2023-05-17T19:12:22Z

It looks like the ubuntu run is hitting the following error:

Assembly PowerShell.Benchmarks which defines benchmarks references non-optimized System.Management.Automation
If you own this dependency, please, build it in RELEASE.
If you don't, you can disable this policy by using 'config.WithOptions(ConfigOptions.DisableOptimizationsValidator)'.

Are you aware of whether the dependency is being built in RELEASE? Otherwise, we can disable the policy as long as we keep the understanding of the potential affect on the benchmarks.

kunalspathak · 2023-05-17T21:11:31Z

@adityapatwardhan might know, but did you find out why it just fails on Ubuntu and not windows?

LoopedBard3 · 2023-05-17T21:36:08Z

@kunalspathak I have not found a reason why it only fails on Ubuntu and not Windows, and it is not something that I have any ideas about what could be causing it.

adityapatwardhan · 2023-05-17T21:48:21Z

looking

adityapatwardhan · 2023-05-18T16:40:36Z

The SDK package Microsoft.PowerShell.SDK has all assemblies as MSIL, is this the issue?

adityapatwardhan · 2023-05-18T20:10:58Z

@kunalspathak @LoopedBard3

Actually this could be the reason:

 <!-- Define non-windows, release configuration properties -->
  <PropertyGroup Condition=" '$(Configuration)' == 'Release' And '$(IsWindows)' != 'true' ">
    <!-- Set-Date fails with optimize enabled in NonWindowsSetDate
         Debugging the issues resolves the problem
     -->
    <Optimize>false</Optimize>
  </PropertyGroup>

https://github.com/PowerShell/PowerShell/blob/c444645b0941d73dc769f0bba6ab70d317bd51a9/PowerShell.Common.props#LL188C2-L194C19

kunalspathak · 2023-05-18T20:13:27Z

Actually this could be the reason:

Thanks @adityapatwardhan . Not sure if I understand "Debugging the issues resolves the problem". Do you know what errors you get if we Optimize=true? Also, does that mean that powershell is not optimized at all for non-windows?

adityapatwardhan · 2023-05-18T20:26:01Z

This might be something very old. I am going to try and repro it.

kunalspathak · 2023-05-19T01:50:30Z

This might be something very old. I am going to try and repro it.

Opened PowerShell/PowerShell#19677 to track it.

TIHan · 2023-05-22T19:15:38Z

I've recently come back from vacation today, is there anything I can do to help with the failures on Ubuntu?

kunalspathak · 2023-05-25T05:25:20Z

FYI- PowerShell/PowerShell#19701

LoopedBard3 · 2023-06-07T21:28:27Z

/azp run

azure-pipelines · 2023-06-07T21:28:41Z

Azure Pipelines successfully started running 1 pipeline(s).

kunalspathak · 2023-06-08T18:50:37Z

FYI- PowerShell/PowerShell#19701

This is merged. What are the next steps remaining here?

LoopedBard3 · 2023-06-08T19:47:26Z

FYI- PowerShell/PowerShell#19701

This is merged. What are the next steps remaining here?

It looks like the ubuntu run is still failing due to the Release vs Debug issue based on my recent run. The Microsoft.PowerShell.SDK version probably needs to be updated, although it does not look like there has been a recent update pushed out: https://www.nuget.org/packages/Microsoft.PowerShell.SDK/. @adityapatwardhan, do you have an idea of when the next package version with the update will be released or if there is a way to get a version with the changes?

Aside from that, it seems windows defender is stopping it on the public windows, but there was a working windows run before so it should not act as a blocker to merging. Although we may need to remove the public windows run until the issue is fixed.

LoopedBard3 · 2023-08-29T22:13:51Z

/azp run

azure-pipelines · 2023-08-29T22:14:03Z

Azure Pipelines successfully started running 1 pipeline(s).

src/benchmarks/real-world/PowerShell.Benchmarks/PowerShell.Benchmarks.csproj

LoopedBard3 · 2023-10-09T18:12:56Z

@kunalspathak @TIHan It appears that the packages have been updated to the point that the debug errors are fixed and we should be able to add these benchmarks. I added a suggestion with the necessary package updates based on some of my testing: https://dev.azure.com/dnceng/internal/_build/results?buildId=2284858&view=results. With the package suggestion in and the pipeline run, we should be good to add these benchmarks.

kunalspathak · 2023-10-09T18:51:25Z

Thanks @LoopedBard3 . If the CI is happy, please commit the suggested suggestions and merge the PR.

…chmarks.csproj

kunalspathak · 2023-10-09T19:59:44Z

i am assuming the CI failures are unrelated. @TIHan we should add this benchmark in the superpmi collection.
CC: @BruceForstall

TIHan · 2023-10-09T20:06:32Z

Yea that makes sense to me.

LoopedBard3 · 2023-10-09T20:17:23Z

i am assuming the CI failures are unrelated. @TIHan we should add this benchmark in the superpmi collection. CC: @BruceForstall

Is that in addition to here or instead of here?

kunalspathak · 2023-10-09T21:37:24Z

Is that in addition to here or instead of here?

It is unrelated to this.

LoopedBard3 · 2023-10-09T21:45:06Z

Is that in addition to here or instead of here?

It is unrelated to this.

Cool, I will go ahead and merge this in then.

Added PowerShell real-world benchmarks

6ee20d6

Added README.md

61027e8

kunalspathak reviewed May 3, 2023

View reviewed changes

src/benchmarks/real-world/PowerShell.Benchmarks/README.md Show resolved Hide resolved

Update README.md

d077961

kunalspathak previously approved these changes May 3, 2023

View reviewed changes

LoopedBard3 reviewed May 4, 2023

View reviewed changes

Add the runs to the yaml files.

9e264cb

Merge pull request #1 from LoopedBard3/powershell-bench-add-runs

e8362f9

Add the runs to the yaml files.

TIHan dismissed kunalspathak’s stale review via e8362f9 May 12, 2023 19:02

Fix typo

27e36c3

danmoseley added the adds new benchmarks label May 24, 2023

build-analysis bot mentioned this pull request Jun 7, 2023

NuGet failing with Response status code does not indicate success: 503 (Service Unavailable) dotnet/arcade#11723

Open

5 tasks

LoopedBard3 reviewed Oct 9, 2023

View reviewed changes

src/benchmarks/real-world/PowerShell.Benchmarks/PowerShell.Benchmarks.csproj Outdated Show resolved Hide resolved

kunalspathak previously approved these changes Oct 9, 2023

View reviewed changes

LoopedBard3 dismissed kunalspathak’s stale review via f823609 October 9, 2023 19:06

Update src/benchmarks/real-world/PowerShell.Benchmarks/PowerShell.Ben…

f823609

…chmarks.csproj

LoopedBard3 requested review from cincuranet and kunalspathak October 9, 2023 19:14

kunalspathak approved these changes Oct 9, 2023

View reviewed changes

LoopedBard3 merged commit a73d7fd into dotnet:main Oct 10, 2023
26 of 33 checks passed

TIHan mentioned this pull request Oct 13, 2023

Added PowerShell.Benchmarks to SuperPMI collections dotnet/runtime#93493

Merged

Added PowerShell real-world benchmarks #3000

Added PowerShell real-world benchmarks #3000

Conversation

TIHan commented May 2, 2023

kunalspathak commented May 2, 2023

kunalspathak commented May 2, 2023

kunalspathak commented May 3, 2023

cincuranet commented May 3, 2023

kunalspathak commented May 3, 2023

LoopedBard3 left a comment

Choose a reason for hiding this comment

LoopedBard3 May 4, 2023

Choose a reason for hiding this comment

TIHan May 5, 2023

Choose a reason for hiding this comment

TIHan May 8, 2023

Choose a reason for hiding this comment

kunalspathak May 8, 2023

Choose a reason for hiding this comment

LoopedBard3 May 8, 2023 • edited

Choose a reason for hiding this comment

TIHan May 8, 2023

Choose a reason for hiding this comment

LoopedBard3 commented May 10, 2023

cincuranet commented May 10, 2023 • edited

LoopedBard3 commented May 17, 2023

kunalspathak commented May 17, 2023

LoopedBard3 commented May 17, 2023

adityapatwardhan commented May 17, 2023

adityapatwardhan commented May 18, 2023

adityapatwardhan commented May 18, 2023 • edited

kunalspathak commented May 18, 2023

adityapatwardhan commented May 18, 2023

kunalspathak commented May 19, 2023

TIHan commented May 22, 2023

kunalspathak commented May 25, 2023

LoopedBard3 commented Jun 7, 2023

azure-pipelines bot commented Jun 7, 2023

kunalspathak commented Jun 8, 2023

LoopedBard3 commented Jun 8, 2023

LoopedBard3 commented Aug 29, 2023

azure-pipelines bot commented Aug 29, 2023

LoopedBard3 commented Oct 9, 2023

kunalspathak commented Oct 9, 2023

kunalspathak commented Oct 9, 2023

TIHan commented Oct 9, 2023

LoopedBard3 commented Oct 9, 2023

kunalspathak commented Oct 9, 2023

LoopedBard3 commented Oct 9, 2023

LoopedBard3 May 8, 2023 •

edited

cincuranet commented May 10, 2023 •

edited

adityapatwardhan commented May 18, 2023 •

edited