Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generator may fail with "Could not load file or assembly 'YamlDotNet..." #221

Closed
AraHaan opened this issue Mar 20, 2021 · 38 comments · Fixed by #272
Closed

Generator may fail with "Could not load file or assembly 'YamlDotNet..." #221

AraHaan opened this issue Mar 20, 2021 · 38 comments · Fixed by #272
Assignees
Labels
bug Something isn't working

Comments

@AraHaan
Copy link
Contributor

AraHaan commented Mar 20, 2021

Currently for some reason the source generator starts to bail out when it sees GetCurrentProcess when called inside of dotnet build.

Oddly it seems to generate just fine from within Visual Studio even if I use the .NET 6 Preview 1/2 SDK's.

CSC : warning CS8785: Generator 'SourceGenerator' failed to generate source. It will not contribute to the output and compilation errors may occur as a result. Exception was of type 'GenerationFailedException' with message 'Failed while generating extern method: GetCurrentProcess' [C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump.csproj]
C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump\MiniDump.cs(15,21): error CS0234: The type or namespace name 'Windows' does not exist in the namespace 'Microsoft' (are you missing an assembly reference?) [C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump.csproj]
C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump\MinidumpTypes.cs(9,21): error CS0234: The type or namespace name 'Windows' does not exist in the namespace 'Microsoft' (are you missing an assembly reference?) [C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump.csproj]
C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump\MiniDump.cs(88,132): error CS0246: The type or namespace name 'MINIDUMP_EXCEPTION_INFORMATION' could not be found (are you missing a using directive or an assembly reference?) [C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump.csproj]
C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump\MiniDump.cs(88,180): error CS0246: The type or namespace name 'MINIDUMP_USER_STREAM_INFORMATION' could not be found (are you missing a using directive or an assembly reference?) [C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump.csproj]
C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump\MiniDump.cs(88,231): error CS0246: The type or namespace name 'MINIDUMP_CALLBACK_INFORMATION' could not be found (are you missing a using directive or an assembly reference?) [C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump.csproj]
C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump\MinidumpTypes.cs(19,18): error CS0103: The name 'MINIDUMP_TYPE' does not exist in the current context [C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump.csproj]
C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump\MinidumpTypes.cs(25,24): error CS0103: The name 'MINIDUMP_TYPE' does not exist in the current context [C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump.csproj]
C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump\MinidumpTypes.cs(31,26): error CS0103: The name 'MINIDUMP_TYPE' does not exist in the current context [C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump.csproj]
C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump\MinidumpTypes.cs(37,26): error CS0103: The name 'MINIDUMP_TYPE' does not exist in the current context [C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump.csproj]
C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump\MinidumpTypes.cs(43,24): error CS0103: The name 'MINIDUMP_TYPE' does not exist in the current context [C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump.csproj]
C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump\MinidumpTypes.cs(49,22): error CS0103: The name 'MINIDUMP_TYPE' does not exist in the current context [C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump.csproj]
C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump\MinidumpTypes.cs(56,31): error CS0103: The name 'MINIDUMP_TYPE' does not exist in the current context [C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump.csproj]
C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump\MinidumpTypes.cs(63,42): error CS0103: The name 'MINIDUMP_TYPE' does not exist in the current context [C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump.csproj]
C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump\MinidumpTypes.cs(70,29): error CS0103: The name 'MINIDUMP_TYPE' does not exist in the current context [C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump.csproj]
C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump\MinidumpTypes.cs(77,33): error CS0103: The name 'MINIDUMP_TYPE' does not exist in the current context [C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump.csproj]
C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump\MinidumpTypes.cs(84,38): error CS0103: The name 'MINIDUMP_TYPE' does not exist in the current context [C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump.csproj]
C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump\MinidumpTypes.cs(91,31): error CS0103: The name 'MINIDUMP_TYPE' does not exist in the current context [C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump.csproj]
C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump\MinidumpTypes.cs(98,30): error CS0103: The name 'MINIDUMP_TYPE' does not exist in the current context [C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump.csproj]
C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump\MinidumpTypes.cs(105,26): error CS0103: The name 'MINIDUMP_TYPE' does not exist in the current context [C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump.csproj]
C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump\MinidumpTypes.cs(112,24): error CS0103: The name 'MINIDUMP_TYPE' does not exist in the current context [C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump.csproj]
C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump\MinidumpTypes.cs(115,33): error CS0103: The name 'MINIDUMP_TYPE' does not exist in the current context [C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump.csproj]
C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump\MinidumpTypes.cs(121,34): error CS0103: The name 'MINIDUMP_TYPE' does not exist in the current context [C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump.csproj]
C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump\MinidumpTypes.cs(128,38): error CS0103: The name 'MINIDUMP_TYPE' does not exist in the current context [C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump.csproj]
C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump\MinidumpTypes.cs(135,36): error CS0103: The name 'MINIDUMP_TYPE' does not exist in the current context [C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump.csproj]
C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump\MinidumpTypes.cs(142,32): error CS0103: The name 'MINIDUMP_TYPE' does not exist in the current context [C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump.csproj]
C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump\MinidumpTypes.cs(149,29): error CS0103: The name 'MINIDUMP_TYPE' does not exist in the current context [C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump.csproj]
C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump\MinidumpTypes.cs(156,24): error CS0103: The name 'MINIDUMP_TYPE' does not exist in the current context [C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump.csproj]
C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump\MinidumpTypes.cs(159,32): error CS0103: The name 'MINIDUMP_TYPE' does not exist in the current context [C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump.csproj]
C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump\MinidumpTypes.cs(162,24): error CS0103: The name 'MINIDUMP_TYPE' does not exist in the current context [C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump.csproj]
C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump\MinidumpTypes.cs(165,40): error CS0103: The name 'MINIDUMP_TYPE' does not exist in the current context [C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump.csproj]
C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump\MinidumpTypes.cs(170,26): error CS0103: The name 'MINIDUMP_TYPE' does not exist in the current context [C:\Users\User\Desktop\github\MyPackages\MiniDump\MiniDump.csproj]

This happens with my project with the following NativeMethods.txt:

GetCurrentProcess
GetCurrentProcessId
GetCurrentThreadId
MiniDumpWriteDump

The first 2 is used when calling MiniDumpWriteDump and the 3rd one is for when constructing the EXCEPTION_INFORMATION structure for the call.

and the following NativeMethods.json file:

{
  "$schema": "https://raw.githubusercontent.com/microsoft/CsWin32/main/src/Microsoft.Windows.CsWin32/settings.schema.json",
  "allowMarshaling": false
}
@AraHaan
Copy link
Contributor Author

AraHaan commented Mar 20, 2021

I could use System.Diagnostics.Process however the issue is that it does not:

  • return the process ids as uint's.
  • Seems a bit useless if I get the current process from System.Diagnostics.Process only to use the handle on the call.

@AraHaan AraHaan changed the title warning CS8785: with GetCurrentProcess warning CS8785 with GetCurrentProcess Mar 20, 2021
@AraHaan AraHaan changed the title warning CS8785 with GetCurrentProcess warning CS8785 and then compile errors with GetCurrentProcess Mar 20, 2021
@jnm2
Copy link
Contributor

jnm2 commented Mar 20, 2021

  • return the process ids as uint's.

@AraHaan If you are using .NET 5+, you should be using https://docs.microsoft.com/en-us/dotnet/api/system.environment.processid.

@AraHaan
Copy link
Contributor Author

AraHaan commented Mar 20, 2021

I use the .NET 6 SDK but on that project I target netstandard2.0, due to also needing to have the same build work with net461+, .net core 2.0+, and anything else that implement .NET Standard 2.0+.

@AraHaan
Copy link
Contributor Author

AraHaan commented Mar 24, 2021

It also seems to do it with MessageBoxW as well when on another project.

@AraHaan
Copy link
Contributor Author

AraHaan commented Mar 24, 2021

Hmm it does it on every Windows API function when targeting .NET Standard 2.0.

Edit: even the enums fail too...

@AraHaan AraHaan changed the title warning CS8785 and then compile errors with GetCurrentProcess warning CS8785 and then compile errors when targeting .NET STandard 2.0 Mar 24, 2021
@AraHaan
Copy link
Contributor Author

AraHaan commented Mar 24, 2021

@AArnott Any idea why it wont at least let the generations of the enums when using the SourceGenerator on netstandard2.0 targets? Enums and structs at least (if not the actual p/invokes) should be safe for .NET Standard 2.0.

I at least need the source generator to maintain the values of a specific enum despite targeting netstandard2.0.

@AraHaan
Copy link
Contributor Author

AraHaan commented Mar 24, 2021

Also I feel like the warning it prints should have included the stack trace of the exception that caused the GenerationFailedException exception so that way I know where something would need a patch on the source generator.

@AArnott
Copy link
Member

AArnott commented Mar 30, 2021

What version of CsWin32 are you using, @AraHaan? With the latest CI version, this doesn't repro for me.

@AArnott
Copy link
Member

AArnott commented Mar 30, 2021

It doesn't repro for me on 0.1.422-beta (latest on nuget.org) either.

@jnm2
Copy link
Contributor

jnm2 commented Mar 30, 2021

Could this be a result of combining a particular version of win32metadata?

@AArnott
Copy link
Member

AArnott commented Mar 30, 2021

@jnm2 potentially yes. We only support using CsWin32 with the metadata that it brings in automatically, because the metadata churns so much right now. Updating metadata on your own is taking on risk that things break.
@AraHaan please let us know if you're using a non-default metadata version as well.

@AraHaan
Copy link
Contributor Author

AraHaan commented Apr 20, 2021

Ok, I have simplified the reproducibility of this issue down to clone git clone https://github.com/Elskom/Sdk.git --recursive --branch use-sourcegenerators followed by a build dotnet build --configuration Release 😂.

Also, I use the default one now, even with non default it still would get this error, and I tried every build of CsWin32 that was released to nuget.org with the same issue.

@AraHaan AraHaan changed the title warning CS8785 and then compile errors when targeting .NET STandard 2.0 warning CS8785 and then compile errors when targeting .NET Standard 2.0 Apr 20, 2021
@AraHaan
Copy link
Contributor Author

AraHaan commented Apr 22, 2021

I decided I wanted to test it on github actions using ubuntu-latest and I found it also reproduces there too 😂 https://github.com/Elskom/Sdk/pull/190/checks.

@AraHaan
Copy link
Contributor Author

AraHaan commented Apr 27, 2021

I tested locally a change and still could not fix it on my end.

@AArnott
Copy link
Member

AArnott commented May 20, 2021

I can repro the exception to generate MiniDumpWriteDump at the tip of main. I'll investigate.

AArnott added a commit that referenced this issue May 20, 2021
This prevents the whole generator from failing to generate anything when something goes wrong.

Closes #221
@AArnott
Copy link
Member

AArnott commented May 20, 2021

Well, the repro I'm seeing is unlikely to be related to what you were seeing, since it's a new failure from #270. But I have a PR to improve the experience either way so we can better diagnose such failures in the future.

@AraHaan
Copy link
Contributor Author

AraHaan commented May 21, 2021

warning CS8785: Generator 'SourceGenerator' failed to generate source. It will not contribute to the output and compilation errors may occur as a result. Exception was of type 'GenerationFailedException' with message 'Failed to generate MINIDUMP_TYPE'

I tried copiling again with 0.1.474-beta and still get this for that enum. I do not see any places where the data in that enum is not consistent across all platforms of Windows. I think that enum should be emitted for AnyCPU builds if they ask for only the enum to be generated.

@AArnott
Copy link
Member

AArnott commented May 21, 2021

The MINIDUMP_TYPE enum is indeed available for AnyCPU generation. I still have no idea why it's failing. Tonight's CI will have a better error message to explain it, but in the meantime if you share a repro I can take a look, since it works on the projects I'm testing with.

@AraHaan
Copy link
Contributor Author

AraHaan commented May 21, 2021

Currently it is reproducible with git clone https://github.com/Elskom/Sdk.git --recursive --branch use-sourcegenerators and then dotnet build --configuration Release.

@AArnott
Copy link
Member

AArnott commented May 24, 2021

Great. With those repro steps I can see this error with the latest CsWin32 version:

error PInvoke000: An internal error occurred: Failed to generate MINIDUMP_TYPE. The type initializer for 'Microsoft.Windows.CsWin32.Docs' threw an exception. Could not load file or assembly 'YamlDotNet, Version=11.0.0.0, Culture=neutral, PublicKeyToken=ec19458f3c15af5e'. Could not find or load a specific file. (0x80131621). [C:\temp\Sdk\MiniDump\MiniDump.csproj]

So this may be a duplicate of #261, which is probably not completely fixed. YamlDotNet 11.0.0.0 is certainly included in the package, so it is probably some dependency whose assembly version is not precisely satisfied.

@AArnott AArnott reopened this May 24, 2021
@AArnott AArnott changed the title warning CS8785 and then compile errors when targeting .NET Standard 2.0 Generator may fail with "Could not load file or assembly 'YamlDotNet..." May 24, 2021
@AArnott AArnott self-assigned this May 24, 2021
@AArnott AArnott added the bug Something isn't working label May 24, 2021
@AArnott
Copy link
Member

AArnott commented May 24, 2021

Ok, by using a debugger I found that YamlDotNet.dll has already been loaded from %userprofile%.nuget\packages\securitycodescan.vs2019\5.1.0\analyzers\dotnet\YamlDotNet.dll

I don't know what securitycodescan.vs2019 is, but this package includes YamlDotNet v8.1.2.0 which loads first and therefore causes .NET Core to be unwilling to load the 11.0.0.0 used by CsWin32 later in the build.
I therefore do not expect this to be a problem to hit most people.

I am concerned however at the lack of isolation between analyzers that the C# compiler IMO ought to be providing so that one analyzer's dependencies does not cause problems for another's dependencies. I'll follow up with the Roslyn team on this.

@AArnott
Copy link
Member

AArnott commented May 24, 2021

@AraHaan, when I remove

    <PackageReference Include="SecurityCodeScan.VS2019" IsImplicitlyDefined="true" Version="*-*">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>

from your Sdk.props file, the CsWin32 error goes away (and is replaced by others that appear unrelated).

I don't know if dropping that dependency for now is an option for you though. I've already reached out to the Roslyn team to ask for recommendations.

@AraHaan
Copy link
Contributor Author

AraHaan commented May 24, 2021

Thanks so the issue is a combination of things. Wonder what other Analyzers that I use use a different version of YamlDotNet causing this conflict.

@JarLob
Copy link

JarLob commented May 25, 2021

I am concerned however at the lack of isolation between analyzers that the C# compiler IMO ought to be providing so that one analyzer's dependencies does not cause problems for another's dependencies. I'll follow up with the Roslyn team on this.

Hey @AArnott, I'm interested in hearing what is the take of the Roslyn team on it. Is there a link? Is it public issue?

@AArnott
Copy link
Member

AArnott commented May 25, 2021

Here is what I heard back from the Roslyn team:

Essentially the build works when you use msbuild but not “dotnet build”. The latter is happening on .NET Core and we don’t fully isolate anaylzers in independent ALC yet. It’s on the backlog for Dev17 / C# 10.

@jnm2
Copy link
Contributor

jnm2 commented May 25, 2021

Is that this issue? dotnet/roslyn#52177

@AraHaan
Copy link
Contributor Author

AraHaan commented May 25, 2021

It looks like it.

@JarLob
Copy link

JarLob commented May 26, 2021

Does /p:BuildInParallel=false as described avoids the issue in your case?

@AraHaan
Copy link
Contributor Author

AraHaan commented May 26, 2021

Does /p:BuildInParallel=false as described avoids the issue in your case?

I tried that and this was the result:

NativeMethods.txt(1,1): error PInvoke000: An internal error occurred: Failed to generate MINIDUMP_TYPE. The type initializer for 'Microsoft.Windows.CsWin32.Docs' threw an exception. Could not load file or assembly 'YamlDotNet, Version=11.0.0.0, Culture=neutral, PublicKeyToken=ec19458f3c15af5e'. Could not find or load a specific file. (0x80131621). Could not load file or assembly 'YamlDotNet, Version=11.0.0.0, Culture=neutral, PublicKeyToken=ec19458f3c15af5e'.

@AraHaan
Copy link
Contributor Author

AraHaan commented May 26, 2021

Hmm I wonder if all analyzers should start doing binding redirects of all of their dependencies from 0.0.0.0-65535.65535.65535.65535 (aka up to whatever the max valid possible version number is) for the oldVersion element, and as for newVersion not sure if that can be wildcarded to * and then have whatever the first version of said dependency to be loaded to be the one that wins the war.

@jnm2
Copy link
Contributor

jnm2 commented May 26, 2021

@JarLob /p:BuildInParallel=false only worked for me as an accident. It's the explanation for why msbuild always worked and dotnet build often failed. But the fact that things were loading in the right order was specific to my repro; I think in this case, they load in the wrong order.

@AArnott
Copy link
Member

AArnott commented Jun 9, 2021

It isn't parallelism that breaks it. It may simply be a build ordering issue. And it's super complex for the roslyn team to fix. It's not likely something we can totally workaround within the source generator, although we may be able to improve some scenarios.

@AArnott
Copy link
Member

AArnott commented Jun 9, 2021

The roslyn issue tracking it is dotnet/roslyn#53672

@AArnott
Copy link
Member

AArnott commented Jun 12, 2021

I'm blocked by this myself in dotnet/Nerdbank.GitVersioning#616. msbuild.exe and VS work but dotnet build fails.

@AraHaan
Copy link
Contributor Author

AraHaan commented Jun 18, 2021

@AArnott what if the part of the generator that uses YamlDotNet was split into a console program that the generator ships with and somehow is able to execute in a subprocess of the compiler to work around this?

Is it possible to do that and permanently fix the YamlDotNet issue for now?

@AArnott
Copy link
Member

AArnott commented Jun 21, 2021

We use YamlDotNet to deserialize a file into a memory model we need. Moving it out of proc would defeat the point.
However as part of fixing #38 we may remove use of YamlDotNet anyway.

@AArnott
Copy link
Member

AArnott commented May 13, 2022

We did indeed drop the YamlDotNet dependency.

@AArnott AArnott closed this as completed May 13, 2022
@AraHaan
Copy link
Contributor Author

AraHaan commented May 13, 2022

Nice

AArnott pushed a commit that referenced this issue Jan 4, 2024
Bumps [powershell](https://github.com/PowerShell/PowerShell) from 7.3.7 to 7.3.8.
- [Release notes](https://github.com/PowerShell/PowerShell/releases)
- [Commits](PowerShell/PowerShell@v7.3.7...v7.3.8)

---
updated-dependencies:
- dependency-name: powershell
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants