Simplify RID Model #260
Simplify RID Model #260
Conversation
Co-authored-by: Jan Kotas <jkotas@microsoft.com>
Co-authored-by: Eric Erhardt <eric.erhardt@microsoft.com>
Co-authored-by: Omair Majid <omajid@redhat.com>
|
|
||
| For Red Hat, it makes sense to accept and support `linux-x64` NuGet assets, but not to produce or offer them. Instead, Red Hat would want to produce `rhel-x64` runtime pack assets. It's easiest to describe this design-point in terms of concrete scenarios. | ||
|
|
||
| **NuGet packages** -- NuGet authors will typically value breadth and efficiency. For example, a package like `SkiaSharp` might target `linux-x64`, but not `rhel-x64`. There is no technical reason for that package to include distro-specific assets. Also, if the author produced a `rhel-x64` asset, they would need to consider producing an `ubuntu-x64` and other similar assets for other distros, and that's not scalable. The expectation is that the `rhel-x64` .NET supports `linux-x64` NuGet assets, enabling NuGet authors to target a minimal set of RIDs. |
There was a problem hiding this comment.
I feel like this document generally, and this section specifically, is making some assumptions that I don't think are justified. There are absolutely scenarios where distro-specific assets need to be shipped in a NuGet package.
As an example of this, take a look at https://www.nuget.org/packages/LibGit2Sharp.NativeBinaries/2.0.306
LibGit2Sharp is a managed wrapper around the native libgit2 library and has to accommodate the native dependencies of libgit2. One of the dependencies is OpenSSL, which varies wildly among all the distros that .NET supports.
In order to have a reasonable chance of having a native binary that just works on all platforms, I have had to spend a lot of time understanding the RID graph and shipping enough distro-specific binaries to cover all the supported Linux distros
I will point out that libgit2 has recently changed how they are binding to OpenSSL which has let me simplify things down to more of an "ideal" situation (see the newer package), but that was largely out of my control since I'm not a libgit2 maintainer.
If the ability to ship distro-specific assets in a NuGet package and have the proper one selected had not been a feature that I could count on, then I would not have been able to support LibGit2Sharp on Linux at all for the past several years. Maybe I'm misunderstanding something, but it seems like this proposal is removing this feature.
There was a problem hiding this comment.
Mind you, libgit2/libgit2sharp#1714 changed the model used by libgit2sharp. AFAIK, the change basically makes libgit2sharp try and load all native libgit2 libraries one by one until one of them can be loaded successfully (ie, links to a version of OpenSSL available on the system). The actual name (and RID) is not too relevant in this scenario.
I feel like your use-case would still work if nuget did not "know" about distro-specific RIDs.
There was a problem hiding this comment.
This is great feedback. We discussed it.
There are three models:
- Don't support specific distros as a general concept (the current proposal).
- Fully support specific distros as a general concept (some variation on status-quo).
- Adopt a model that is a variation on the Python manylinux wheel plan, including support for OpenSSL.
I like what @omajid is proposing. It's the same approach we use for OpenSSL. We don't use RIDs. Also, the current proposal works for the vast majority of cases and is a massive simplification. I'm hoping we can keep it.
There was a problem hiding this comment.
libgit2sharp was suffering from the rid scalability issue: it was unable to deal with unknown rids.
Using NativeLibrary.SetDllImportResolver it now tries the different so files. The code is here: https://github.com/libgit2/libgit2sharp/blob/97bee65fd296f1c7dd2d1d64581c170f45b584e1/LibGit2Sharp/Core/NativeMethods.cs#L99-L121
This way, the library can have its own logic for picking so files. It's not limited to/by rids.
There was a problem hiding this comment.
Yes, this logic was added, but I consider it a fallback solution, and not something desirable to rely on as the primary logic, for a couple of reasons:
- It only works if the entire runtimes folder is shipped as part of the application, which is only going to be true if you use one of the cross-platform publishing options
- Having to attempt to load every binary until you find one that works seems like it would have a negative impact on startup perf.
If libraries no longer can assume that the best native asset is selected for them by the framework/NuGet, then that means they have to manually ensure that all binaries are included in the publish output to be able to select from them at runtime. That can pretty massively increase the size of the output. Using LibGit2Sharp.NativeBinaries 2.0.306 as an example, that means going from having a single ~1MB .so file in your output to having 9 different copies, and that's assuming I'd have some way to know to only copy the Linux binaries and not the entire runtimes folder.
There was a problem hiding this comment.
For example add a property to csproj that enables the user to choose the supported OpenSSL versions, and include so files based on that.
The problem does not end with OpenSSL. The same problem exists for many other libraries.
I believe that the only truly scalable option for Linux ecosystem is to have building from source as an option (NuGet/Home#9631). Building from source is capable of producing a binary that works for the specific configuration of your system without the package maintainer pre-building it for you.
There was a problem hiding this comment.
The problem does not end with OpenSSL. The same problem exists for many other libraries.
Yes, and this was the main point of the feedback I was trying to raise here. The proposal currently assumes that there is never a reason to care about the specific distro you're running on, but that is demonstrably not true.
This all comes down to there being a disconnect between the .NET ecosystem expectations of sharing pre-compiled binaries, and the Linux ecosystem that expects you to share source and compile it on your own system. Shipping a Linux binary in a package is always going to be an uphill battle, so I would like to see the .NET side of things evolve in a way that makes it easier, not harder.
There was a problem hiding this comment.
Notice that it does not have support for probing for OpenSSL 3 yet so it is going to break once distros switch to OpenSSL 3 only. dotnet/runtime has probing for OpenSSL3 that adds a whole new level of shimming and complexity.
Let's assume OpenSSL 3 won't be supported by a libgit2 shimming (immediately) and you want to solve it issue using distro rids.
Fedora 36 should use the OpenSSL 3 build.
Fedora 37 should be using the OpenSSL 3 build also. That doesn't work automagically today, because the rid grah doesn't describe a supports relationship between Fedora 37 and Fedora 36.
For your package to remain maintainable, this needs to be addressed (and preferably not by bloating the graph but by making the rid logic smarter).
I've only mentioned Fedora here, but OpenSSL will be adopted by Ubuntu, Debian, so you need to manage this problem for them too.
Your package now looks like this:
linux-x64 -> supports OpenSSL 1, 2, 3 and picks using `NativeLibrary`
fedora.36 -> OpenSSL 3
ubuntu.22.04 -> OpenSSL 3
debian.12 -> OpenSSL 3
...
Note that you are including the distro specific assets solely for the purpose of rid-based trimming. They bloat your package and require you to track in what version a distro adopts a new OpenSSL version.
The proposal currently assumes that there is never a reason to care about the specific distro you're running on
NativeLibrary can be used, and should be used to make linux-x64 work across a range of distros.
What is lost, is the ability to trim against a rid.
There was a problem hiding this comment.
@bording I hope this example helped illustrate how unscalable the rid mechanism is?
And, if it should stay, inheritance between successive distro versions is a must-have, which is now missing.
There was a problem hiding this comment.
And, if it should stay, inheritance between successive distro versions is a must-have, which is now missing.
To elaborate. The rids as we have them are not scalable. The alternative to the simple model that is proposed here, is to have semantic rids. That means rids follow a <distro>.<version>-<arch> naming scheme. Information is derived from that instead of hard coding it in the json file (e.g. Fedora 36 -> Fedora 35). The json file should then only contain non-derivable information (like what distros are glibc/musl based).
This is a much more complex solution.
|
|
||
| ## Minimum CRT/libc version | ||
|
|
||
| This scheme doesn't explicitly define a CRT/libc version. Instead, we will define a minimum CRT/libc version, per major .NET version. That minimum CRT/libc version will form a contract and should be documented, per .NET version. Practically, it defines the oldest OS version that you can run an app on, for a given .NET version. |
There was a problem hiding this comment.
Using C runtime version to define the OS version makes sense on Linux. It is the best approximation of the OS version to use for determining binary compatibility (of binaries written in C/C++).
Using C runtime version for this purpose makes less sense on Windows and macOS. Windows and macOS have well-defined OS version. Both Windows (Windows SDK) and macOS (XCode) native toolchains are oriented on using the OS version for this purpose (WINVER on Windows and -mmacos-version-min on macOS). We should use OS version for determining binary compatibility on Windows and macOS.
There was a problem hiding this comment.
Do you want me to add that text, or adapt its meaning into the current text?
There was a problem hiding this comment.
I think adapting the meaning into the current text would look better. I would lead with minimum OS version as a documented contract, per .NET version. libc version is Linux-specific approximation of OS version across distros for the purposes of this contract.
|
I think the main impact of the design is that package maintainers must change from using distro-rids for separating native libraries to group them under a portable rid (e.g. (In practice) we'll also loose the ability to trim for a distro-specific rid. This came for free because the native assets were organized by distro-rid. If the package maintainer needs to maintain a portable rid ( |
The problem I see with saying package maintainers have to use this approach is that it is overlooking how publish output changes if you specify a rid or not. A portable publish will copy all of the rid folders into the output, but a publishing with That makes writing code that can use |
|
My comment was to highlight the complexity that gets pushed towards the package maintainers. Note that if your library has different native libraries for glibc Linux distros, in order to make With this proposal, in the case the user doesn't specify a rid, the library now also needs to use |
👍 And that is what I've been somewhat trying to push back on through my comments here. Or at least acknowledgement that this is understood and documented appropriately on how to author packages in this more complicated way.
And my previous comment was trying to point out that writing this |
It depends. If you know the names of the libraries, you can pass them to If you want to iterate yourself the directories being searched to discover at runtime what native libraries are available (e.g. based on a naming pattern), then there is no way to get these directories. They could be exposed through a property on |
To make sure everyone is talking about the same thing, I want to be clear and describe the scenario. We are talking about a NuGet package that has a distro-specific native dependency. Currently, the way to do that is to ship a package with as many distro-specific RIDs as needed. For example, the runtimes folder might look something like: This is tricky to get right and does require ongoing work to ensure the binaries shipped are comprehensive. It also requires that the RID graph is maintained and accurate. The proposed alternative here seems to be something like the following instead: With this sort of package layout, it is now on the author to write code that can search inside the As things currently stand, I see some problems with this proposal, and it actually isn't possible to author a package like this with the way NuGet works right now. Problems:
When you build a project or publish it without a RID specified ( However, as soon as you publish with a RID specified, the folder structure is collapsed and the contents of the native folder are now top-level in the publish output. Using the proposed folder structure above, with something like What happens with If this was true, how are you supposed to author the However, NuGet does not actually seem to work this way currently.
When you publish with It is complaining about seeing more than one copy of the binary as if it's trying to put all of them into the publish folder instead of maintaining the defined folder structure. If you instead try to put the distro subfolders directly under You could potentially work around the NETSDK1152 error by naming every single binary differently, but that's not always under your control. It also doesn't solve the Ultimately, my goal with all of this is to ensure that whatever change ends up happening is done with an awareness that work needs to be done to continue to support packages that need to have distro-specific native binaries. The burden can't be shifted entirely to package maintainers, and in fact the way things currently work make that nearly impossible. |
The proposed alternative here is the following structure instead: where |
|
@jkotas I don't see how that is meaningfully different from what I had in my post. And it doesn't solve any of the problems I described. This isn't just about glibc vs musl. This is about any sort of native dependency that is different per distro. For example, OpenSSL. |
|
The assumption in this proposal is that it is just about glibc vs. musl for the vast majority of native libraries bundled into packages. The Python solution is based on the same assumption. For OpenSSL 2 vs. 3 problem, you can build one version of the .so that dynamically binds to OpenSSL2 or 3. It is what .NET runtime itself is doing. libgit2 mirrored that solution as well (https://github.com/libgit2/libgit2/blob/main/src/libgit2/streams/openssl_dynamic.c). Or you can build two version of the so like libgit2_openssl2.so and libgit2_openssl3.so and implement a loader that loads the right one at runtime. If you have a complex library with a many different dependencies, you can do a custom loader that picks up the right flavor for given distro. Yes, it is complex. The assumption is that only a small fraction of packages would need to do something like this. |
|
This code works both when publishing with or without a rid. foreach (var nativeLibrary in new[] { "mylib-opensslv3.so", "mylib-opensslv2.so", "mylib-opensslv1.so" })
{
if (NativeLibrary.TryLoad(nativeLibrary, typeof(SomeType).Assembly, DllImportSearchPath.ApplicationDirectory, out handle)
break;
}
If there are use-cases like this, which require looking at the included native libraries, the directories could be made available through an API. foreach (var nativeLibraryDir in NativeLibrary.NativeLibraryDirectories)
{
foreach (var nativeLibrary in Directory.GetFiles(nativeLibraryDir, "mylib-*.so"))
{
if (NativeLibrary.TryLoad(nativeLibrary, typeof(SomeType).Assembly, DllImportSearchPath.ApplicationDirectory, out handle)
break;
}
}The downsides to this design are:
The upside of the
Are there some numbers that tell us how many packages on nuget.org have native libraries for non-portable rids? |
Yes, that would be useful data to have, together with the usage numbers. I believe that people tend to stay away from packages with native dependencies on Linux today since they are broken too often. For example, we had to drop the dependency on LibGit2Sharp in source link (dotnet/sourcelink#288) and reimplement it in C#. One of the reasons was that the distro-specific libgit2 builds caused too many problems. |
|
This issue seems to be focused on relieving of the complexity of having to manage various Linux distros and their versions. What about other OSes that are completely different from Linux in ways that are not only related to native library ABI, but also managed system library code? dotnet/runtime#90695, which seems to be related to this issue, is currently breaking managed library builds for Haiku, since the official .NET SDK is not aware of Haiku and the build is now configured to ignore the Unlike different Linux distros like Ubuntu and Alpine, which can use the same IL code as any |
|
The intent of this proposal is to move away from OS flavor- and version-specific targeting. This is most notable for Linux because there are so many versions and flavors, but it's equally true for Windows and Mac, where you will no longer be able to target versions individually. However, different OSes are still classified as different portable RIDs. A portable RID is essentially |
dotnet/runtime#86391 (comment) It is being done here, but other members of dotnet are questioning whether a RID for Haiku should be added.
This means unlike what dotnet/runtime#90695 states, the list of RIDs should not be frozen for good, it should just be less volatile since version numbers and distro flavors are not being updated anymore? |
This plan is intended to:
runtimes.jsonto being used by NuGet only (by default).linux-x64means (same for Arm64 and x86).linux-x64vs source-build.More generally, we waste significant amount of time talking about RIDs every release. This proposal is intended to simplify the most problematic aspect of the RID topic. Once that is resolved, we can think about taking on other RID challenges.
Related to:
Rendered view