CPU usage increases after upgrading to .NET 6

[t-bzhan]

Description

Our ASP.NET Core service's CPU usage increases a lot after upgrading from .NET Core 3.1 to .NET 6.0.14, we collected a trace using dotnet-trace and found that our problem is pretty similar with #62967. We tried to disable the usage of portable thread pool with setting COMPlus_ThreadPool_UsePortableThreadPool=0, the trick works in our test environment, we observed CPU usage decreased as expected under stressing test, however in production, we noticed that the problem persisted.

I compared the trace under ".NET 6" with ".NET Core 3.1" and found the increased CPU usage might be caused by Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.Heartbeat.TimerLoop() and System.Diagnostics.Tracing.CounterGroup.PollForValues()

Could anyone suggest what steps I need to take to resolve this?

I could not share the netrace file here as it contains production data but I could share it to MS internal members, feel free to drop me a message if a netrace file could help.

Configuration

# dotnet --info
.NET SDK (reflecting any global.json):
Version:   6.0.406
Commit:    2988897946 Runtime Environment:
OS Name:     ubuntu
OS Version:  22.04
OS Platform: Linux
RID:         ubuntu.22.04-x64
Base Path:   /usr/share/dotnet/sdk/6.0.406/ 

global.json file:
  Not found
Host:
  Version:      6.0.14
  Architecture: x64
  Commit:       2a90daa2cc 

.NET SDKs installed:
  6.0.406 [/usr/share/dotnet/sdk] 

.NET runtimes installed:
  Microsoft.AspNetCore.App 6.0.14 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 6.0.14 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

Regression?

That looks like a regression, I did not observe similar CPU usage in .NET Core 3.1

Data

CPU usage increased after we rollout new version which upgrade to .NET 6.

Analysis

akkadotnet/akka.net#5385 (comment) mentioned that similar CPU regression could be fixed by upgrading the runtime version to 6.0.6, however our runtime version is 6.0.14 already.

For Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.Heartbeat.TimerLoop()'s CPU usage dotnet/aspnetcore#47014 explained that the data is actually somewhat misleading, since it includes time that the thread was waiting as well.

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[ghost]

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

Our ASP.NET Core service's CPU usage increases a lot after upgrading from .NET Core 3.1 to .NET 6.0.14, we collected a trace using dotnet-trace and found that our problem is pretty similar with #62967. We tried to disable the usage of portable thread pool with setting COMPlus_ThreadPool_UsePortableThreadPool=0, the trick works in our test environment, we observed CPU usage decreased as expected under stressing test, however in production, we noticed that the problem persisted.

I compared the trace under ".NET 6" with ".NET Core 3.1" and found the increased CPU usage might be caused by Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.Heartbeat.TimerLoop() and System.Diagnostics.Tracing.CounterGroup.PollForValues()

Could anyone suggest what steps I need to take to resolve this?

I could not share the netrace file here as it contains production data but I could share it to MS internal members, feel free to drop me a message if a netrace file could help.

Configuration

# dotnet --info
.NET SDK (reflecting any global.json):
Version:   6.0.406
Commit:    2988897946 Runtime Environment:
OS Name:     ubuntu
OS Version:  22.04
OS Platform: Linux
RID:         ubuntu.22.04-x64
Base Path:   /usr/share/dotnet/sdk/6.0.406/ 

global.json file:
  Not found
Host:
  Version:      6.0.14
  Architecture: x64
  Commit:       2a90daa2cc 

.NET SDKs installed:
  6.0.406 [/usr/share/dotnet/sdk] 

.NET runtimes installed:
  Microsoft.AspNetCore.App 6.0.14 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 6.0.14 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

Regression?

That looks like a regression, I did not observe similar CPU usage in .NET Core 3.1

Data

CPU usage increased after we rollout new version which upgrade to .NET 6.

Analysis

akkadotnet/akka.net#5385 (comment) mentioned that similar CPU regression could be fixed by upgrading the runtime version to 6.0.6, however our runtime version is 6.0.14 already.

For Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.Heartbeat.TimerLoop()'s CPU usage dotnet/aspnetcore#47014 explained that the data is actually somewhat misleading, since it includes time that the thread was waiting as well.

Author:	t-bzhan
Assignees:	-
Labels:	area-System.Threading, tenet-performance, untriaged
Milestone:	-

[mangod9]

@kouvel to take a look. @t-bzhan, assume you see the same behavior on 7?

[En3Tho]

One more question: have you observed other metrics change? Like RPS for example or reduced latency? Maybe something else?

[t-bzhan]

@kouvel to take a look. @t-bzhan, assume you see the same behavior on 7?

I did not test our service behavior on .NET 7 as our pipeline does not support it yet.

[t-bzhan]

One more question: have you observers other metrics change? Like RPS for example or reduced latency? Maybe something else?

I noticed that the average response time increase around 10ms when the number of request counts remains the same level.

I've also collected the dontnet perf counters in .NET 6, not sure whether that is helpful:

I kind of feeling like the issue might be caused by threadpool, I checked the Microsoft-Windows-DotNETRuntime/ThreadPoolWorkerThreadAdjustment/Adjustment event with perfview, however I am not able to figure out whether that is abnormal or not:

[kouvel]

Chatted offline, the times shown in the first two stacks are likely mostly blocked time and not CPU time, suggested using perfcollect instead.

[t-bzhan]

That is related to #84760, close it.