Support for Deterministic Builds of .NET Runtime library

[GlebChili]

The original discussion: https://groups.google.com/g/antlr-discussion/c/DAuWTC1tmrE/m/atKoQx2bAQAJ

Hi!

I have noticed recently, that Antlr4.Runtime.Standard runtime library for .NET is not using deterministic builds feature (one can read briefly about it here: https://github.com/clairernovotny/DeterministicBuilds). Microsoft is pushing for adoption of deterministic builds and NuGet.org will use deterministic builds metadata for validating packages. Deterministic builds are also compulsory for all .NET Foundation projects. I think it is good idea to enable it for Antlr4.Runtime.Standard. If you are interested, I am ready to help with it.

Best regards,
Gleb Krasilich

[kaby76]

I have scores of my own tools in nuget.org, so this seems like something I should learn. What exactly does "NuGet.org will use deterministic builds metadata for validating packages" mean?

My view of nuget.org is a secure file server. When I upload a .nupkg and .snupkg to nuget.org, I've already made sure I did the build in a standardized virtual OS environment, already did my exhaustive tests to know that it works, and checked via a round trip fetch gives the same file. Doesn't nuget.org keep checksums on the files I uploaded, doesn't it disallow for these files to be mutated by someone, and the files in an msbuild or dotnet install are validated with a checksum with those critical tools? Nuget does not have my sources, nor does it know my build env or how I do the build.

[GlebChili]

As you mentioned, "build in a standardized virtual OS environment" part is up to the developer. NuGet.org does not prevent anyone from building and publishing a package from a source other than the one provided in the metadata. Microsoft wants to change it for at least Open Source projects with a combination of SourceLink and Deterministic Builds. In the future, NuGet.org will be able to read package's metadata and verify that it was built from the specified source.

[parrt]

I use nuget to create artifacts when I release since I can't figure it out locally. haha

[GlebChili]

The enabling of Deterministic Builds should be quite easy:

1. Antlr4.Runtime.Standard should reference Microsoft.SourceLink.GitHub NuGet package.
2. Antlr4.Runtime.Standard release package should be built with a recent version of .NET SDK (like 3.1.400+) and ContinuousIntegrationBuild property set to true, like dotnet pack /p:ContinuousIntegrationBuild=true.

The package does not necessarily need to be built in CI, ContinuousIntegrationBuild property can be used for local builds. It will prevent compiler from relaying on sources outside of the project repository.

[kaby76]

I'm trying to see how this prevents some typical blunders I might make. So, I wanted to see if this helps solve something I learned 40 years ago: the need to build in a standard environment directly from version control in order to actually have a "reproducible build".

I created a library from scratch (dotnet new classlib), added the nuget package declarations and reference mentioned above, and made sure not to check these changes into github repo. I also did a fresh clone of one of my FOSS repos from github. In the .csproj file, I add <ItemGroup><PackageReference Include="Microsoft.SourceLink.GitHub" Version="1.1.1" /></ItemGroup>.

• For the library that I just created, dotnet build warns that the source doesn't exist in github. That's good! I don't want to upload a .nupkg for sources that don't exist in github.com.
• For the FOSS library, a dotnet build had no warnings. Presumably, this is a "deterministic" build, yet it had an uncommitted change to the .csproj file for the package reference.
• I further modify a .cs source file in the FOSS library and do a dotnet build. There are no warnings. This clearly is not reproducible because I can nuke the local files after uploading to nuget.org (though I haven't tried).

Can a deterministic NET build prevent unchanged files from being used in building a .nupkg that I updated to nuget.org? What am I missing here?

[GlebChili]

I believe .NET SDK does not warn you about some blunders you've described (yet). What .NET SDK deterministic builds does already is the following:

1. Enables deterministic compilation for Roslyn compiler as described here.
2. Embeds compiler flags into assembly metadata. In other words, resulting assembly will contain the full information about how it was built (the full list of parameters passed to Roslyn's csc.exe).
3. Checks that SourceLink information is well-formed.

The NuGet.org server will be able to read SourceLink information and compiler flags, reproduce builds, and check that published assemblies are byte-wise identical. The verified packages will get some special trusted status on NuGet.org.

So, for now you can think about deterministic builds in .NET SDK not as a tool to help you avoid some blunders while developing locally, but as an attempt to create a subset of trusted verifiable packages in the NuGet ecosystem.

A side note about SourceLink. It also helps consumers to debug library code, since SourceLink adds links to the original source files in debug symbols. Visual Studio (and I believe Rider too) can automatically download source files from repository when stepping into SourceLink-enabled library's code. Here is an article on this feature: https://devblogs.microsoft.com/dotnet/improving-debug-time-productivity-with-source-link/

[jaredpar]

The enabling of Deterministic Builds should be quite easy

I think there is a bit of a misconception here. Deterministic builds is simply the property that given the same input a build tool will produce the same output (byte for byte). This is the default for .NET SDK style projects as it's an inherent property of the C# compiler.

This thread seems to be concerned more with reproducible builds. That is effectively the ability for two different developers or environments to get the same output (byte for byte) for a given build. That relies on the underlying build tool being deterministic. But it is primarily concerned with making sure the inputs to the build are the same in both cases.

For compilers that typically comes down to path normalization, ensuring you have the same compiler, etc ...

Enables deterministic compilation for Roslyn compiler as described

Nothing is needed to enable deterministic roslyn builds. That is the default.

[ericvergnaud]

Smells like OSGI ;-)

…

Le 19 janv. 2022 à 04:21, Gleb Krasilich ***@***.***> a écrit : I believe .NET SDK does not warn you about some blunders you've described (yet). What .NET SDK deterministic builds does already is the following: Enables deterministic compilation for Roslyn compiler as described here <https://github.com/dotnet/roslyn/blob/main/docs/compilers/Deterministic%20Inputs.md>. Embeds compiler flags into assembly metadata. In other words, resulting assembly will contain the full information about how it was built (the full list of parameters passed to Roslyn's csc.exe). Checks that SourceLink information is well-formed. The NuGet.org server will be able to read SourceLink information and compiler flags, reproduce builds, and check that published assemblies are byte-wise identical. The verified packages will get some special trusted status on NuGet.org. So, for now you can think about deterministic builds in .NET SDK not as a tool to help you avoid some blunders while developing locally, but as an attempt to create a subset of trusted verifiable packages in the NuGet ecosystem. A side note about SourceLink. It also helps consumers to debug library code, since SourceLink adds links to the original source files in debug symbols. Visual Studio (and I believe Rider too) can automatically download source files from repository when stepping into SourceLink-enabled library's code. Here is an article on this feature: https://devblogs.microsoft.com/dotnet/improving-debug-time-productivity-with-source-link/ <https://devblogs.microsoft.com/dotnet/improving-debug-time-productivity-with-source-link/> — Reply to this email directly, view it on GitHub <#3498 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZNQJBDXN7YNXW6KRXL4LTUWYU4RANCNFSM5MHF5Z3A>. You are receiving this because you are subscribed to this thread.

[kaby76]

Adding to my confusion, I read the information on the C# compiler deterministic builds. https://github.com/dotnet/roslyn/blob/main/docs/compilers/Deterministic%20Inputs.md
It says that the default for a csc is "/deterministic". Sure enough, it is the default because "mkdir test; cd test; dotnet new classlib; dotnet build -v diag" outputs this:

C:\Program Files\dotnet\dotnet.exe exec "C:\Program Files\dotnet\sdk\6.0.101\Roslyn\bincore\csc.dll" /noconfig /unsafe- /checked- /nowarn:1701,1702,1701,1702 /fullpaths /nostdlib+ /errorreport:prompt /warn:6 /define:TRACE;DEBUG;NET;NET6_0;NETCOREAPP;NET5_0_OR_GREATER;NET6_0_OR_GREATER;NETCOREAPP1_0_OR_GREATER;NETCOREAPP1_1_OR_GREATER;NETCOREAPP2_0_OR_GREATER;NETCOREAPP2_1_OR_GREATER;NETCOREAPP2_2_OR_GREATER;NETCOREAPP3_0_OR_GREATER;NETCOREAPP3_1_OR_GREATER /highentropyva+ /nullable:enable /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\Microsoft.CSharp.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\Microsoft.VisualBasic.Core.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\Microsoft.VisualBasic.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\Microsoft.Win32.Primitives.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\Microsoft.Win32.Registry.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\mscorlib.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\netstandard.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.AppContext.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Buffers.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Collections.Concurrent.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Collections.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Collections.Immutable.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Collections.NonGeneric.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Collections.Specialized.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.ComponentModel.Annotations.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.ComponentModel.DataAnnotations.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.ComponentModel.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.ComponentModel.EventBasedAsync.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.ComponentModel.Primitives.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.ComponentModel.TypeConverter.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Configuration.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Console.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Core.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Data.Common.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Data.DataSetExtensions.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Data.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Diagnostics.Contracts.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Diagnostics.Debug.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Diagnostics.DiagnosticSource.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Diagnostics.FileVersionInfo.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Diagnostics.Process.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Diagnostics.StackTrace.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Diagnostics.TextWriterTraceListener.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Diagnostics.Tools.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Diagnostics.TraceSource.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Diagnostics.Tracing.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Drawing.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Drawing.Primitives.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Dynamic.Runtime.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Formats.Asn1.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Globalization.Calendars.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Globalization.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Globalization.Extensions.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.IO.Compression.Brotli.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.IO.Compression.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.IO.Compression.FileSystem.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.IO.Compression.ZipFile.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.IO.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.IO.FileSystem.AccessControl.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.IO.FileSystem.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.IO.FileSystem.DriveInfo.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.IO.FileSystem.Primitives.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.IO.FileSystem.Watcher.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.IO.IsolatedStorage.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.IO.MemoryMappedFiles.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.IO.Pipes.AccessControl.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.IO.Pipes.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.IO.UnmanagedMemoryStream.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Linq.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Linq.Expressions.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Linq.Parallel.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Linq.Queryable.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Memory.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Net.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Net.Http.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Net.Http.Json.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Net.HttpListener.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Net.Mail.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Net.NameResolution.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Net.NetworkInformation.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Net.Ping.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Net.Primitives.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Net.Requests.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Net.Security.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Net.ServicePoint.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Net.Sockets.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Net.WebClient.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Net.WebHeaderCollection.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Net.WebProxy.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Net.WebSockets.Client.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Net.WebSockets.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Numerics.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Numerics.Vectors.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.ObjectModel.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Reflection.DispatchProxy.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Reflection.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Reflection.Emit.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Reflection.Emit.ILGeneration.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Reflection.Emit.Lightweight.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Reflection.Extensions.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Reflection.Metadata.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Reflection.Primitives.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Reflection.TypeExtensions.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Resources.Reader.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Resources.ResourceManager.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Resources.Writer.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Runtime.CompilerServices.Unsafe.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Runtime.CompilerServices.VisualC.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Runtime.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Runtime.Extensions.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Runtime.Handles.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Runtime.InteropServices.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Runtime.InteropServices.RuntimeInformation.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Runtime.Intrinsics.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Runtime.Loader.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Runtime.Numerics.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Runtime.Serialization.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Runtime.Serialization.Formatters.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Runtime.Serialization.Json.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Runtime.Serialization.Primitives.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Runtime.Serialization.Xml.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Security.AccessControl.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Security.Claims.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Security.Cryptography.Algorithms.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Security.Cryptography.Cng.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Security.Cryptography.Csp.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Security.Cryptography.Encoding.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Security.Cryptography.OpenSsl.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Security.Cryptography.Primitives.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Security.Cryptography.X509Certificates.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Security.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Security.Principal.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Security.Principal.Windows.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Security.SecureString.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.ServiceModel.Web.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.ServiceProcess.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Text.Encoding.CodePages.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Text.Encoding.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Text.Encoding.Extensions.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Text.Encodings.Web.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Text.Json.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Text.RegularExpressions.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Threading.Channels.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Threading.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Threading.Overlapped.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Threading.Tasks.Dataflow.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Threading.Tasks.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Threading.Tasks.Extensions.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Threading.Tasks.Parallel.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Threading.Thread.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Threading.ThreadPool.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Threading.Timer.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Transactions.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Transactions.Local.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.ValueTuple.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Web.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Web.HttpUtility.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Windows.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Xml.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Xml.Linq.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Xml.ReaderWriter.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Xml.Serialization.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Xml.XDocument.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Xml.XmlDocument.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Xml.XmlSerializer.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Xml.XPath.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\System.Xml.XPath.XDocument.dll" /reference:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\ref\net6.0\WindowsBase.dll" /debug+ /debug:portable /filealign:512 /optimize- /pathmap:c:/msys64/home/kenne/test=c:/msys64/home/kenne/test2 /out:obj\Debug\net6.0\test.dll /refout:obj\Debug\net6.0\ref\test.dll /target:library /warnaserror- /utf8output /deterministic+ /langversion:10.0 /analyzerconfig:obj\Debug\net6.0\test.GeneratedMSBuildEditorConfig.editorconfig /analyzerconfig:"C:\Program Files\dotnet\sdk\6.0.101\Sdks\Microsoft.NET.Sdk\analyzers\build\config\analysislevel_6_default.editorconfig" /analyzer:"C:\Program Files\dotnet\sdk\6.0.101\Sdks\Microsoft.NET.Sdk\targets\..\analyzers\Microsoft.CodeAnalysis.CSharp.NetAnalyzers.dll" /analyzer:"C:\Program Files\dotnet\sdk\6.0.101\Sdks\Microsoft.NET.Sdk\targets\..\analyzers\Microsoft.CodeAnalysis.NetAnalyzers.dll" /analyzer:"C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Ref\6.0.1\analyzers/dotnet/cs/System.Text.Json.SourceGeneration.dll" Class1.cs obj\Debug\net6.0\test.GlobalUsings.g.cs "obj\Debug\net6.0\.NETCoreApp,Version=v6.0.AssemblyAttributes.cs" obj\Debug\net6.0\test.AssemblyInfo.cs /warnaserror+:NU1605 (TaskId:80)

But, then I try to verify the claim This means that the "same inputs" will cause the compilers [sic] to produce the "same outputs" byte for byte, and supposedly solved as documented in this issue dotnet/roslyn#56904. I create a classlib and copy the entire source to a different path. Now I do a build of both (identical sources) and compare the .dll generated for both.

Even after adding several permuations of the element <PropertyGroup><PathMap>source-path-for-test=dest-path-for-test</PathMap></PropertyGroup> to the .csproj (see this https://blog.paranoidcoding.com/2016/04/05/deterministic-builds-in-roslyn.html#what-if-i-build-from-different-enlistment-paths ), the .dll's for identical source on two different paths on my file system are different.

Sure, I do get an identical .dll if I do a dotnet build at two different times of the same source in the same path and compare the .dlls, so timestamps are identical in the assemblies, I guess.

strings test.dll doesn't contain any of the compiler flags that I thought would be there, e.g. "deterministic" itself. Likewise, I didn't see anything like that using dotpeek or my own assembly tool (I ported DotNetAnywhere and greatly enhanced it).

Is there an example that demonstrates identical dll's built for the same source on two different paths?

[jaredpar]

I create a classlib and copy the entire source to a different path. Now I do a build of both (identical sources) and compare the .dll generated for both.

The full path of source files in the build is part of the deterministic input to a compilation. Hence the expectation is building the same code at different root locations will produce different output. If it produced the same output that would be a bug.

Even after adding several permuations of the element source-path-for-test=dest-path-for-test to the .csproj (see this https://blog.paranoidcoding.com/2016/04/05/deterministic-builds-in-roslyn.html#what-if-i-build-from-different-enlistment-paths ), the .dll's for identical source on two different paths on my file system are different.

This likely means you've setup the <PathMap> incorrectly. If you have a concrete example I can help you work through the setup issues.

This goes through extensive testing in our repositories. The entire Roslyn code base is built using the latest compiler on different drives using path map and verifies that every assembly is byte for byte identical. This happens on every PR into the compiler.

Sure, I do get an identical .dll if I do a dotnet build at two different times of the same source in the same path and compare the .dlls, so timestamps are identical in the assemblies, I guess.

When doing a deterministic build time stamps are not used. Yes the PE file format has an explicit location for timestamps but in determinitsic mode we write a determistic value to that field which is not a real timestamp.

Is there an example that demonstrates identical dll's built for the same source on two different paths?

https://gist.github.com/jaredpar/c07ab29a4f323aef9386a80559bf2d78

That is a log of my powershell console doing the following:

1. Creating a class lib, building and recording the hash of the DLL
2. Using subst to move the code to a different drive
3. Building and verifying the hash is different because different paths
4. Rebuilding using PathMap to normalize and verifying hash matches 1 above

[kaby76]

@jaredpar Thanks for the info Jared. This helps a lot.

[kaby76]

Yeah, I can't duplicate this. I even created the same directory structure--paying close attention to every character in the path, executed exactly the same commands (esp. dotnet build -p:PathMap="q:\=C:\Users\jaredpar\temp\" -t:build)--paying close attention to the case of every character. The .dll's are different. dotnet --version => 6.0.101. There's no warning if the -p:PathMap is used, contains an invalid path (e.g., dotnet build -p:PathMap="q:\=C:\Users\jaredpar\teasdfasdfasdsfassdfasdfasdfmp\" -t:build), is used, etc., which it should. The first dotnet build call you had didn't have arg -t:build, the second did. So I tried being consistent in using -t:build or without. It didn't make any difference. The example.dll's are not the same. It does not work for me whatsoever.