Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tracking issue for "WORKLOAD TIMED OUT" #90309

Open
steveharter opened this issue Aug 10, 2023 · 5 comments
Open

Tracking issue for "WORKLOAD TIMED OUT" #90309

steveharter opened this issue Aug 10, 2023 · 5 comments
Labels
area-Infrastructure blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' Known Build Error Use this to report build issues in the .NET Helix tab os-ios Apple iOS os-tvos Apple tvOS untriaged New issue has not been triaged by the area owner

Comments

@steveharter
Copy link
Member

steveharter commented Aug 10, 2023 •
edited by build-analysis bot

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=368974
Build error leg or test failing:
Pull request: #90119

Error Message

{
  "ErrorMessage": "ERROR: WORKLOAD TIMED OUT - Killing user command",
  "BuildRetry": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=368974
Error message validated: ERROR: WORKLOAD TIMED OUT - Killing user command
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 8/10/2023 1:38:38 PM UTC

Report

Build Definition Test Pull Request
686902 dotnet/runtime JIT_opt.WorkItemExecution
682021 dotnet/runtime Loader_classloader.WorkItemExecution
681751 dotnet/runtime System.Security.Cryptography.Tests.WorkItemExecution
681987 dotnet/runtime Loader_classloader.WorkItemExecution
680972 dotnet/runtime System.Runtime.Tests.WorkItemExecution #102439
679966 dotnet/runtime JIT_opt.WorkItemExecution
678653 dotnet/runtime Loader_classloader.WorkItemExecution
664858 dotnet/runtime System.Runtime.Tests.WorkItemExecution #101853
664789 dotnet/runtime System.Runtime.Tests.WorkItemExecution #101850
664560 dotnet/runtime System.Runtime.Tests.WorkItemExecution #101844
664282 dotnet/runtime iOS.Device.ExportManagedSymbols.Test.WorkItemExecution #101836
664190 dotnet/runtime iOS.Device.Aot.Test.WorkItemExecution #101717
664107 dotnet/runtime iOS.Device.LibraryMode.Test.WorkItemExecution
663982 dotnet/runtime iOS.Device.ExportManagedSymbols.Test.WorkItemExecution
663779 dotnet/runtime System.Runtime.Tests.WorkItemExecution #101823
663696 dotnet/runtime iOS.Device.LibraryMode.Test.WorkItemExecution #101773
663579 dotnet/runtime System.Runtime.Tests.WorkItemExecution #101814
663454 dotnet/runtime iOS.Device.ExportManagedSymbols.Test.WorkItemExecution
663336 dotnet/runtime System.Runtime.Tests.WorkItemExecution #101762
662930 dotnet/runtime iOS.Device.Aot.Test.WorkItemExecution #101761
662333 dotnet/runtime iOS.Device.ExportManagedSymbols.Test.WorkItemExecution #101744
662225 dotnet/runtime iOS.Device.LibraryMode.Test.WorkItemExecution #101151
662112 dotnet/runtime iOS.Device.Aot.Test.WorkItemExecution #101451
661965 dotnet/runtime System.Runtime.Tests.WorkItemExecution #101766
661843 dotnet/runtime iOS.Device.LibraryMode.Test.WorkItemExecution #101760
661633 dotnet/runtime iOS.Device.LibraryMode.Test.WorkItemExecution #101151
660693 dotnet/runtime iOS.Device.LibraryMode.Test.WorkItemExecution #101660
660568 dotnet/runtime iOS.Device.Aot.Test.WorkItemExecution #99818
660440 dotnet/runtime iOS.Device.LibraryMode.Test.WorkItemExecution #100266
660198 dotnet/runtime System.Runtime.Numerics.Tests.WorkItemExecution
660077 dotnet/runtime iOS.Device.Aot.Test.WorkItemExecution #101212
659861 dotnet/runtime iOS.Device.ExportManagedSymbols.Test.WorkItemExecution #101693
659688 dotnet/runtime iOS.Device.Aot.Test.WorkItemExecution #101379
659595 dotnet/runtime iOS.Device.Aot.Test.WorkItemExecution #101686
659286 dotnet/runtime iOS.Device.ExportManagedSymbols.Test.WorkItemExecution #101679
659190 dotnet/runtime iOS.Device.ExportManagedSymbols.Test.WorkItemExecution #101624
659022 dotnet/runtime iOS.Device.LibraryMode.Test.WorkItemExecution #101605
658984 dotnet/runtime iOS.Device.Aot.Test.WorkItemExecution #101001
658682 dotnet/runtime iOS.Device.LibraryMode.Test.WorkItemExecution #101330
657924 dotnet/runtime iOS.Device.Aot.Test.WorkItemExecution #101624
657876 dotnet/runtime iOS.Device.Aot.Test.WorkItemExecution #101336
657814 dotnet/runtime iOS.Device.Aot.Test.WorkItemExecution #101566

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
0 5 42
@steveharter steveharter added blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' area-Infrastructure Known Build Error Use this to report build issues in the .NET Helix tab labels Aug 10, 2023
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Aug 10, 2023
@ghost
Copy link

ghost commented Aug 10, 2023

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=368974
Build error leg or test failing:
Pull request: #90119

Error Message

{
  "ErrorMessage": "ERROR: WORKLOAD TIMED OUT - Killing user command..",
  "BuildRetry": false
}
Author: steveharter
Assignees: -
Labels:

blocking-clean-ci, area-Infrastructure, Known Build Error
Milestone: -

@stephentoub stephentoub mentioned this issue Aug 15, 2023
Merged
@ericstj ericstj added os-ios Apple iOS os-tvos Apple tvOS labels Sep 14, 2023
@ghost
Copy link

ghost commented Sep 14, 2023

Tagging subscribers to 'os-ios': @steveisok, @akoeplinger, @kotlarmilos
See info in area-owners.md if you want to be subscribed.

Issue Details

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=368974
Build error leg or test failing:
Pull request: #90119

Error Message

{
  "ErrorMessage": "ERROR: WORKLOAD TIMED OUT - Killing user command",
  "BuildRetry": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=368974
Error message validated: ERROR: WORKLOAD TIMED OUT - Killing user command
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 8/10/2023 1:38:38 PM UTC

Report

Build Definition Test Pull Request
406029 dotnet/runtime Loader_classloader.WorkItemExecution #91261
406036 dotnet/runtime System.Security.Cryptography.Tests.WorkItemExecution
405860 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
405505 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
405371 dotnet/runtime JIT_opt.WorkItemExecution
405156 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
405201 dotnet/runtime iOS.Device.ExportManagedSymbols.Test.WorkItemExecution #91753
405175 dotnet/runtime iOS.Device.Aot.Test.WorkItemExecution #91913
404815 dotnet/runtime iOS.Device.Aot.Test.WorkItemExecution #91498
404502 dotnet/runtime JIT_opt.WorkItemExecution
403846 dotnet/runtime System.Security.Cryptography.Tests.WorkItemExecution
404006 dotnet/runtime iOS.Device.ExportManagedSymbols.Test.WorkItemExecution #91964
403649 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
403432 dotnet/runtime iOS.Device.ExportManagedSymbols.Test.WorkItemExecution #91945
403331 dotnet/runtime System.Security.Cryptography.Tests.WorkItemExecution #91910
403278 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
402917 dotnet/runtime JIT_opt.WorkItemExecution #91925
386099 dotnet/runtime iOS.Device.ExportManagedSymbols.Test.WorkItemExecution #90764
402796 dotnet/runtime JIT_opt.WorkItemExecution
402586 dotnet/runtime JIT_opt.WorkItemExecution
402622 dotnet/runtime iOS.Device.Aot.Test.WorkItemExecution #91915
402416 dotnet/runtime JIT_opt.WorkItemExecution
402225 dotnet/runtime JIT_opt.WorkItemExecution
402136 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
402090 dotnet/runtime iOS.Device.Aot.Test.WorkItemExecution #90436
401838 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
401993 dotnet/runtime iOS.Device.Aot.Test.WorkItemExecution #91892
398263 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #91690
401505 dotnet/runtime JIT_opt.WorkItemExecution #91042
401473 dotnet/runtime JIT_opt.WorkItemExecution
401430 dotnet/runtime JIT_opt.WorkItemExecution #91788
401363 dotnet/runtime iOS.Device.Aot.Test.WorkItemExecution #91671
401270 dotnet/runtime System.Security.Cryptography.Tests.WorkItemExecution
401274 dotnet/runtime JIT_opt.WorkItemExecution #91808
401015 dotnet/runtime JIT_opt.WorkItemExecution
400995 dotnet/runtime iOS.Device.Aot.Test.WorkItemExecution #91482
400925 dotnet/runtime JIT_opt.WorkItemExecution #91831
400893 dotnet/runtime JIT_opt.WorkItemExecution
400630 dotnet/runtime JIT_opt.WorkItemExecution
400465 dotnet/runtime iOS.Device.ExportManagedSymbols.Test.WorkItemExecution #91828
400257 dotnet/runtime System.Security.Cryptography.Tests.WorkItemExecution
400363 dotnet/runtime JIT_opt.WorkItemExecution #91468
400157 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #91753
399738 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #91790
399748 dotnet/runtime iOS.Device.ExportManagedSymbols.Test.WorkItemExecution #91792
399536 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #91785
399499 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
399419 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #91774
399275 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
399203 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #91713
398919 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
398113 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #91659
398621 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #91520
398510 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
398372 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #91019
398117 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
398093 dotnet/runtime JIT_opt.WorkItemExecution #91714
398051 dotnet/runtime JIT_opt.WorkItemExecution #91520
398047 dotnet/runtime iOS.Device.Aot.Test.WorkItemExecution #91520
397851 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
397767 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #91703
397525 dotnet/runtime JIT_opt.WorkItemExecution
397329 dotnet/runtime JIT_opt.WorkItemExecution
397233 dotnet/runtime JIT_opt.WorkItemExecution #91684
397052 dotnet/runtime JIT_opt.WorkItemExecution #91458
397028 dotnet/runtime JIT_opt.WorkItemExecution #91290
396987 dotnet/runtime iOS.Device.Aot.Test.WorkItemExecution #91290
396849 dotnet/runtime JIT_opt.WorkItemExecution #91019
396755 dotnet/runtime JIT_opt.WorkItemExecution #91660
396634 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
396691 dotnet/runtime JIT_opt.WorkItemExecution
396584 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #91542
396520 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
396111 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
396177 dotnet/runtime iOS.Device.ExportManagedSymbols.Test.WorkItemExecution #91290
395791 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #91458
395609 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #91542
395438 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #91458
395416 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
395450 dotnet/runtime iOS.Device.ExportManagedSymbols.Test.WorkItemExecution #91536
395268 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
395186 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
394754 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #91116
394959 dotnet/runtime iOS.Device.ExportManagedSymbols.Test.WorkItemExecution #91536
394722 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #91542
394654 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
394471 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
394475 dotnet/runtime iOS.Device.ExportManagedSymbols.Test.WorkItemExecution #91524
394436 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #91520
394347 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
394281 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
394161 dotnet/runtime iOS.Device.Aot.Test.WorkItemExecution #91502
394046 dotnet/runtime iOS.Device.ExportManagedSymbols.Test.WorkItemExecution #91497
393987 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
393980 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
393667 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
393578 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
393552 dotnet/runtime iOS.Device.Aot.Test.WorkItemExecution #91469
393434 dotnet/runtime System.Text.Json.Tests.WorkItemExecution
393189 dotnet/runtime System.Text.Json.Tests.WorkItemExecution #91458
Displaying 100 of 246 results

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
8 55 246
Author: steveharter
Assignees: -
Labels:

blocking-clean-ci, area-Infrastructure, untriaged, os-ios, os-tvos, Known Build Error
Milestone: -

@steveisok
Copy link
Member

This is a variation of #82637

@steveisok
Copy link
Member

steveisok commented Dec 10, 2023 •
edited

@akoeplinger something is wrong w/ our classification when there is a crash in a test suite.

Check out https://dev.azure.com/dnceng-public/public/_build/results?buildId=494600&view=ms.vss-test-web.build-test-results-tab&runId=11481726&resultId=179767 for example. The net.dot log is huge (350MB) and seems to be logging a crash over and over again.

@akoeplinger
Copy link
Member

akoeplinger commented Dec 11, 2023 •
edited

The classification itself is correct, we're timing out because of the crash log loop. I see a potential reason for this, discussing it on Discord.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area-Infrastructure blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' Known Build Error Use this to report build issues in the .NET Helix tab os-ios Apple iOS os-tvos Apple tvOS untriaged New issue has not been triaged by the area owner
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@steveisok @akoeplinger @steveharter @ericstj