Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

With many large files: stops progressing, OOMs or InvalidOperation in Roslyn code #877

Open
jnnnnn opened this issue Apr 14, 2022 · 10 comments
Open

With many large files: stops progressing, OOMs or InvalidOperation in Roslyn code #877

jnnnnn opened this issue Apr 14, 2022 · 10 comments

Comments

@jnnnnn
Copy link

jnnnnn commented Apr 14, 2022

Steps to reproduce

  1. Convert a proprietary codebase (which I can't share, my apologies).
  2. Wait a couple of hours
  3. In phase 2, simplification, it crashes.

Error message shown

The crash is

https://github.com/dotnet/roslyn threw an exception: System.InvalidOperationException: Exception of type 'System.InvalidOperationException' was thrown.
   at System.Runtime.CompilerServices.ConditionalWeakTable`2.VerifyIntegrity()
   at System.Runtime.CompilerServices.ConditionalWeakTable`2.Add(TKey key, TValue value)
   at Microsoft.CodeAnalysis.GreenNode..ctor(ObjectReader reader)

Full log at https://gist.github.com/jnnnnn/2b7a739170467e125d0a05b0a263774a (with filenames hidden, sorry)

Details

VS extension version 8.5.0.0 (installed 2022-04-14)

  • Did you see it working in a previous version, which? no
  • I can follow reasonable requests to debug, but can't share the code it is crashing on
@GrahamTheCoder
Copy link
Member

GrahamTheCoder commented Apr 14, 2022 •
edited

Thanks for the report (and sharing as much info as you could manage)

The roslyn formatter's/simplifier's memory usage scales very badly with file size. So for designer forms which end up as very large files (especially after conversion), assuming your RAM isn't being fully used you might want to try either VS2022, or the command line which are 64-bit and hence can use more RAM.

Someone else had success in a similar situation here: #605 (comment)
You may find enough linked info to workaround it by splitting larger files or by using the command line.

Unfortunately I haven't found time to workaround the issue within the tool yet. The plan is to split files into partial classes and/or format/simplify smaller chunks at a time, and possibly use memoryfailpoint to detect when things are about to go wrong and take evasive action

@GrahamTheCoder GrahamTheCoder changed the title Simplification InvalidOperationException (in Roslyn?) Large files cause OOM or InvalidOperation in Roslyn code Apr 14, 2022
@GrahamTheCoder GrahamTheCoder mentioned this issue Apr 23, 2022
Closed
@a-wallen
Copy link

@GrahamTheCoder, thank you for supporting this tool! I am trying to convert a project with ~2k-3k files. Phase 1 completes smoothly, but in Phase 2 there is one file that refuses to be simplified:

Public Class Common
#Region "Constructor"
    Private Sub New()
    End Sub
#End Region     'Constructor

#Region "Constants"

#If CLR2 Then
    'HKCR path
    private const DataRegistryKey as String = "Infragistics\NetAdvantage\Net\Full\WinForms\CLR2x\Version" & Infragistics.Shared.AssemblyVersion.MajorMinor & "\WinDataDir"
#Else
    'HKCR path
    Private Const DataRegistryKey As String = "Infragistics\NetAdvantage\Net\Full\WinForms\CLR1x\Version" & Infragistics.Shared.AssemblyVersion.MajorMinor & "\WinDataDir"
#End If

#End Region  'Constants

#Region "DataPath"
    ' <summary>
    ' Path to the data installed by the install.
    ' </summary>
    Public Shared ReadOnly Property DataPath() As String
        Get
            Dim dataRegKey As Microsoft.Win32.RegistryKey = Microsoft.Win32.Registry.ClassesRoot.OpenSubKey(DataRegistryKey)
            Dim path As String = Nothing

            If Not dataRegKey Is Nothing Then
                path = CType(dataRegKey.GetValue(Nothing), String)
                dataRegKey.Close()
            End If

            Return path
        End Get
    End Property
#End Region   'DataPath
End Class

There's been no progress for hours.

I am using VS2022 and my machine has 32GB RAM.

@GrahamTheCoder
Copy link
Member

GrahamTheCoder commented Jul 1, 2022 •
edited

Thanks for getting in touch.
If it's just that single file that's really interesting! Does the same happen if you right click the file in solution explorer and convert just that one?
Note that the process is parallel, so there may be some earlier (much larger) file still being simplified too at the same time.

While it's making no progress, what's the CPU/RAM usage like? Maybe leave task manager open on the performance tab for 30 seconds and send a screenshot of the little graphs down the side just to give a general overview.

If it doesn't repro with just the one class, I don't suppose there's any chance that you're able to share the whole codebase to test with? It'd be great to have a real repro for this.

Note that in the past, the command line (see readme for details) has worked for people in similar situations...though I don't know why, I suspect VS has a special implementation/configuration of some cache or other.

@a-wallen
Copy link

a-wallen commented Jul 3, 2022 •
edited

Hi @GrahamTheCoder, sorry for the late response it takes a long time to convert the project that I'm working on.

It looks like it's just that one file that I sent that won't convert. I tried to convert the whole project with both the command line and VS 2022 - the project conversion stalled on just that one file (Common.vb). If I right-click on that file and convert it the same thing happens (stalls at the simplification stage).

Generally, this is what my task manager shows with no apps open besides either VS2022 or the windows command prompt.
image

Do you have any other recommendations for how to proceed?

@GrahamTheCoder
Copy link
Member

GrahamTheCoder commented Jul 6, 2022 •
edited

Sorry I missed your reply. Thanks for the information. It looks unlikely to be memory related in this case then! The CPU load looks like it is probablby attempting a large single-threaded operation.

I tried creating a project with just the file you posted (and then created one other file defining Infragistics.Shared.AssemblyVersion.MajorMinor to make it compile). When I converted that it went through basically instantly. So it's either system-dependant, or related to the processing of other files too (even the single file conversion uses the context of other files around it)

The next thing I'd suggest is to run from source. At the point when it's been well and truly stuck for a few minutes, if you pause the debugger and grab a screenshot of the parallel stacks window, that should be a big help in identifying the area causing the issue. You may also be able to look in the locals window at which bit of code is being simplified specifically. If you are able to share the whole project with me either publicly or privately I can try to repro/debug this myself (though I understand this may not be possible).

In terms of working around the issue to try to get you some useful output, I've pushed a branch called "skip-simplification". If you check that out and run it. In the instance where it's running, set the option called "Tools->Options->Code Converter->Comment and formatting timeout (minutes)" to something like 1 and it will also apply to the simplification phase (which it doesn't on master). Hopefully that will let it get to the end and write out the files (some of which presumably won't be simplified so will have extra qualifiers / casts / parentheses etc.)

@GrahamTheCoder GrahamTheCoder mentioned this issue Aug 8, 2022
Closed
@GrahamTheCoder GrahamTheCoder changed the title Large files cause OOM or InvalidOperation in Roslyn code Large files cause stops progressing, OOMs or InvalidOperation in Roslyn code Aug 8, 2022
@GrahamTheCoder GrahamTheCoder changed the title Large files cause stops progressing, OOMs or InvalidOperation in Roslyn code With many large files: stops progressing, OOMs or InvalidOperation in Roslyn code Aug 8, 2022
@GrahamTheCoder GrahamTheCoder mentioned this issue Aug 10, 2022
Closed
@AptiviCEO
Copy link

I have confirmed this. When I started the conversion of an over 80000+ lines of code project, it hangs at this file: https://github.com/Aptivi/Kernel-Simulator/blob/4a7c852d7f25886e134605ce3f6462736de08e91/Kernel%20Simulator/Shell/Shells/UESH/UESHShellCommon.vb

All files were converted except this, so I had to use an alternative converter to finish it. Then, I had to manually edit everything so it points to .csproj instead of .vbproj.

During the hang, my processor, Intel Core i7-8700, had 100% usage on one of the 12 cores.

@GrahamTheCoder
Copy link
Member

@EoflaOE thanks, that example is likely going to be a massive help! I'll try to repro with it as soon as I can find time and hopefully will finally be able to make progress on this issue 😊

@GrahamTheCoder GrahamTheCoder pinned this issue Aug 21, 2022
@GrahamTheCoder GrahamTheCoder mentioned this issue Aug 25, 2022
Merged
3 tasks
@GrahamTheCoder
Copy link
Member

Thanks @EoflaOE this definitely seems to repro the issue even in total isolation. I've created a draft PR with an initial workaround which skips files that take longer than the configurable timeout. Hopefully I'll find a way to improve the performance by calling roslyn in a different way (or possibly making a fix to roslyn)

Here's a snippet of C# which takes more than a minute to simplify (though admittedly contains about 10K characters in a single expression.

using System;using System.Collections;using System.Collections.Generic;using System.Data;using System.Diagnostics;

// Kernel Simulator  Copyright (C) 2018-2022  Aptivi
// 
// This file is part of Kernel Simulator
// 
// Kernel Simulator is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
// 
// Kernel Simulator is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
// GNU General Public License for more details.
// 
// You should have received a copy of the GNU General Public License
// along with this program.  If not, see <https://www.gnu.org/licenses/>.
using global::System.IO;using System.Linq;using System.Net;using System.Xml.Linq;using ColorSeq;using Extensification.DictionaryExts;using Extensification.StringExts;using FluentFTP;using FluentFTP.Helpers;using KS.ConsoleBase;using KS.ConsoleBase.Colors;using KS.ConsoleBase.Inputs;using KS.ConsoleBase.Themes;using KS.ConsoleBase.Themes.Studio;using KS.Files;using KS.Kernel;using KS.Languages;using KS.Login;using KS.Misc.Platform;using KS.Misc.Probers;using KS.Misc.Threading;using KS.Misc.Writers.ConsoleWriters;using KS.Misc.Writers.DebugWriters;using KS.Misc.Writers.FancyWriters;using KS.Shell;using KS.Shell.ShellBase;using KS.Shell.ShellBase.Commands;using KS.Shell.ShellBase.Shells;using global::KS.Shell.Shells.UESH.Commands;using Microsoft.VisualBasic.Constants;using Newtonsoft.Json;using Renci.SshNet;

namespace KS.Shell.Shells.UESH
{
    public static class UESHShellCommon
    {

        internal readonly static global::System.Collections.Generic.Dictionary<global::System.String, global::KS.Shell.ShellBase.Commands.CommandInfo> ModCommands = new global::System.Collections.Generic.Dictionary<global::System.String, global::KS.Shell.ShellBase.Commands.CommandInfo>();

        /// <summary>
        /// List of commands
        /// </summary>
                                                                                                                                                                                                                                                                public readonly static global::System.Collections.Generic.Dictionary<global::System.String, global::KS.Shell.ShellBase.Commands.CommandInfo> Commands = new global::System.Collections.Generic.Dictionary<global::System.String, global::KS.Shell.ShellBase.Commands.CommandInfo>() { { "adduser", new global::KS.Shell.ShellBase.Commands.CommandInfo("adduser", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Adds users", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "<userName> [password] [confirm]" }, true, 1), new global::KS.Shell.Shells.UESH.Commands.AddUserCommand(), global::KS.Shell.ShellBase.Commands.CommandFlags.Strict) }, { "alias", new global::KS.Shell.ShellBase.Commands.CommandInfo("alias", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Adds aliases to commands", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { $"<rem/add> <{global::System.String.Join("/", global::System.Enum.GetNames(typeof(global::KS.Shell.ShellBase.Shells.ShellType)))}> <alias> <cmd>" }, true, 3), new global::KS.Shell.Shells.UESH.Commands.AliasCommand(), global::KS.Shell.ShellBase.Commands.CommandFlags.Strict) }, { "arginj", new global::KS.Shell.ShellBase.Commands.CommandInfo("arginj", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Injects arguments to the kernel (reboot required)", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "[Arguments separated by spaces]" }, true, 1), new global::KS.Shell.Shells.UESH.Commands.ArgInjCommand(), global::KS.Shell.ShellBase.Commands.CommandFlags.Strict) }, { "beep", new global::KS.Shell.ShellBase.Commands.CommandInfo("beep", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Beep in 'n' Hz and time in 'n' milliseconds", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "<37-32767 Hz> <milliseconds>" }, true, 2), new global::KS.Shell.Shells.UESH.Commands.BeepCommand()) }, { "blockdbgdev", new global::KS.Shell.ShellBase.Commands.CommandInfo("blockdbgdev", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Block a debug device by IP address", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "<ipaddress>" }, true, 1), new global::KS.Shell.Shells.UESH.Commands.BlockDbgDevCommand(), global::KS.Shell.ShellBase.Commands.CommandFlags.Strict) }, { "calc", new global::KS.Shell.ShellBase.Commands.CommandInfo("calc", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Calculator to calculate expressions.", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "<expression>" }, true, 1), new global::KS.Shell.Shells.UESH.Commands.CalcCommand()) }, { "calendar", new global::KS.Shell.ShellBase.Commands.CommandInfo("calendar", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Calendar, event, and reminder manager", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "<show> [year] [month]", "<event> <add> <date> <title>", "<event> <remove> <eventid>", "<event> <list>", "<event> <saveall>", "<reminder> <add> <dateandtime> <title>", "<reminder> <remove> <reminderid>", "<reminder> <list>", "<reminder> <saveall>" }, true, 1), new global::KS.Shell.Shells.UESH.Commands.CalendarCommand()) }, { "cat", new global::KS.Shell.ShellBase.Commands.CommandInfo("cat", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Prints content of file to console", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "[-lines|-nolines] <file>" }, true, 1), new global::KS.Shell.Shells.UESH.Commands.CatCommand(), global::KS.Shell.ShellBase.Commands.CommandFlags.Wrappable) }, { "cdbglog", new global::KS.Shell.ShellBase.Commands.CommandInfo("cdbglog", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Deletes everything in debug log", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(), new global::KS.Shell.Shells.UESH.Commands.CdbgLogCommand(), global::KS.Shell.ShellBase.Commands.CommandFlags.Strict) }, { "chattr", new global::KS.Shell.ShellBase.Commands.CommandInfo("chattr", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Changes attribute of a file", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "<file> +/-<attributes>" }, true, 2), new global::KS.Shell.Shells.UESH.Commands.ChAttrCommand()) }, { "chdir", new global::KS.Shell.ShellBase.Commands.CommandInfo("chdir", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Changes directory", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "<directory/..>" }, true, 1), new global::KS.Shell.Shells.UESH.Commands.ChDirCommand()) }, { "chhostname", new global::KS.Shell.ShellBase.Commands.CommandInfo("chhostname", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Changes host name", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "<HostName>" }, true, 1), new global::KS.Shell.Shells.UESH.Commands.ChHostNameCommand(), global::KS.Shell.ShellBase.Commands.CommandFlags.Strict) }, { "chlang", new global::KS.Shell.ShellBase.Commands.CommandInfo("chlang", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Changes language", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "[-alwaystransliterated|-alwaystranslated|-force] <language>" }, true, 1), new global::KS.Shell.Shells.UESH.Commands.ChLangCommand(), global::KS.Shell.ShellBase.Commands.CommandFlags.Strict) }, { "chmal", new global::KS.Shell.ShellBase.Commands.CommandInfo("chmal", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Changes MAL, the MOTD After Login", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "[Message]" }, false, 0), new global::KS.Shell.Shells.UESH.Commands.ChMalCommand(), global::KS.Shell.ShellBase.Commands.CommandFlags.Strict) }, { "chmotd", new global::KS.Shell.ShellBase.Commands.CommandInfo("chmotd", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Changes MOTD, the Message Of The Day", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "[Message]" }, false, 0), new global::KS.Shell.Shells.UESH.Commands.ChMotdCommand(), global::KS.Shell.ShellBase.Commands.CommandFlags.Strict) }, { "choice", new global::KS.Shell.ShellBase.Commands.CommandInfo("choice", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Makes user choices", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "[-o|-t|-m|-a] [-multiple|-single] <$variable> <answers> <input> [answertitle1] [answertitle2] ..." }, true, 3), new global::KS.Shell.Shells.UESH.Commands.ChoiceCommand(), global::KS.Shell.ShellBase.Commands.CommandFlags.SettingVariable) }, { "chpwd", new global::KS.Shell.ShellBase.Commands.CommandInfo("chpwd", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Changes password for current user", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "<Username> <UserPass> <newPass> <confirm>" }, true, 4), new global::KS.Shell.Shells.UESH.Commands.ChPwdCommand(), global::KS.Shell.ShellBase.Commands.CommandFlags.Strict) }, { "chusrname", new global::KS.Shell.ShellBase.Commands.CommandInfo("chusrname", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Changes user name", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(new[] { "<oldUserName> <newUserName>" }, true, 2), new global::KS.Shell.Shells.UESH.Commands.ChUsrNameCommand(), global::KS.Shell.ShellBase.Commands.CommandFlags.Strict) }, { "clearfiredevents", new global::KS.Shell.ShellBase.Commands.CommandInfo("clearfiredevents", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Clears all fired events", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(), new global::KS.Shell.Shells.UESH.Commands.ClearFiredEventsCommand()) }, { "cls", new global::KS.Shell.ShellBase.Commands.CommandInfo("cls", global::KS.Shell.ShellBase.Shells.ShellType.Shell, "Clears the screen", new global::KS.Shell.ShellBase.Commands.CommandArgumentInfo(), new global::KS.Shell.Shells.UESH.Commands.ClsCommand()) } };

    }
}

@a-wallen
Copy link

a-wallen commented Aug 25, 2022 •
edited

cc @Nortus222 if #942 lands, then you should talk to @stfedwards about converting our projects from VB to C# since we previously failed due to this bug.

@stfedwards
Copy link

stfedwards commented Aug 26, 2022 via email

@GrahamTheCoder GrahamTheCoder mentioned this issue Sep 20, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@jnnnnn @GrahamTheCoder @AptiviCEO @stfedwards @a-wallen