Make PSWSMan module the only source of libmi and libpsrpclient binaries for PowerShell remoting over WSMan on Linux and macOS?

[daxian-dbw]

Background

The PowerShell remoting over WSMan on Linux and macOS depends on the WSMan client libraries libmi.so/libmi.dylib and libpsrpclient.so/libpsrpclient.dylib, which were from the microsoft/omi repository. The OMI project is in maintenance mode and its team has decided to not support PowerShell anymore. Therefore, the PowerShell WSMan remoting is becoming more and more broken on Linux and macOS.

Proposal

How about making the PSWSMan module the only source of libmi and libpsrpclient binaries for PowerShell remoting over WSMan on Linux and macOS?

The PSWSMan module is created by @jborean93 to install and manage the WSMan client libraries libmi and libpsrpclient produced from his forked omi repository for Linux and macOS.

The libmi/libpsrpclient shipped by PowerShell are not going to be updated. They may still work on some old Linux distros, but very likely not on newer ones. @jborean93, what if PowerShell doesn't ship libmi/libpsrpclient at all, but instead tell users to leverage PSWSMan module to enable WSMan remoting support on Linux/macOS?

Today, to use PSWSMan, one has to replace the WSMan client libraries shipped by PowerShell with the ones contained in PSWSMan with Install-WSMan. The user needs to have root privilege and needs to restart PowerShell for that.

If PowerShell doesn't ship those libs, then all PSWSMan needs to do is to set a DllImportResolver callback for the Microsoft.Management.Infrastructure assembly upon module loading, which will then be used by the runtime to resolve the P/Invoke's from MMI to libmi/libpsrpclient.

[edit] I missed that System.Management.Automation also references libpsrpclient in PInvoke's. NativeLibrary.SetDllImportResolver only allows one callback per assembly, when it comes to S.M.A, the risk of collision would be too high to be ignored. So, instead of using DllImportResolver, we can use AssemblyLoadContext.ResolvingUnmanagedDll instead, which is a multi-cast event. See the algorithm for loading unmanaged lib.

I saw in PSWSMan.psm1 that you have a set of steps to resolve which libs to be used based on the runtime/environment. That logic can be put in the ResolvingUnmanagedDll handler. You register the handler to AssemblyLoadContext.Default when the PSWSMan module is being loaded, and then unregister it upon module unloading.

@jborean93, what do you think about this proposal? This will essentially make your forked omi repository the advertised "official" recommended repo for WSMan client support in PowerShell.

/cc @TravisEz13 @SteveL-MSFT @PaulHigin @JamesWTruher

[jborean93]

needs to do is to set a DllImportResolver callback for the Microsoft.Management.Infrastructure assembly upon module loading, which will then be used by the runtime to resolve the P/Invoke's from MMI to libmi/libpsrpclient

This is something that I have investigated when someone tried to get this working in Azure Cloud Shell where you don't have root access jborean93/omi#24 (comment)

Add-Type -TypeDefinition @'
using System;
using System.Collections.Generic;
using System.Reflection;
using System.Runtime.InteropServices;

namespace CloudShell
{
    public static class MI
    {
        public static Dictionary<string, string> DllMap = new Dictionary<string, string>();
        
        public static IntPtr ImportResolver(string libraryName, Assembly assembly, DllImportSearchPath? searchPath)
        {
            if (DllMap.ContainsKey(libraryName))
                libraryName = DllMap[libraryName];

            return NativeLibrary.Load(libraryName, assembly, searchPath);
        }
    }
}
'@

$libbase = Join-Path (Import-Module -Name PSWSMan -PassThru).ModuleBase lib glibc-1.1
[CloudShell.MI]::DllMap['libpsrpclient'] = Join-Path $libBase libpsrpclient.so
[Runtime.InteropServices.NativeLibrary]::SetDllImportResolver([PSObject].Assembly, [CloudShell.MI]::ImportResolver)

While this does work and allows you to use the fork without root access and the restart there is one catch with SetDllImportResolver that makes me uncomfortable. The docs for SetDllImportResolver states that

Only one resolver can be registered per assembly. Trying to register a second resolver fails with an InvalidOperationException.

Right now there is no custom import resolver for the S.M.A assembly but I have no idea if that assumption will be true in the future. It also means that any 3rd party code that does this themselves for some reason will not be usable with this module and vice versa. I don't know whether this is really something to be concerned about but I have mostly avoided going that route because I didn't want to paint myself into a corner.

I see in your comment you are talking about setting this for the Microsoft.Management.Infrastructure assembly. I've personally not used my fork for anything in MMI, my focus was on PSRemoting over WSMan, so I couldn't even tell you if it works at all with the changes I've made. I'm not even sure when it's used in PowerShell on Linux as the CIM cmdlets aren't present. If getting MMI working is an end goal then it would mean the DLL resolver needs to be set on both S.M.A for PSRemoting over WSMan and for MMI increasing the risk of a collision in the future.

I saw in PSWSMan.psm1 that you have a set of steps to resolve which libs to be used based on the runtime/environment. That logic can be put in the DllImportResolver callback. You set the callback when the PSWSMan module is being loaded, and then unset the callback (NativeLibrary.SetDllImportResolver(wmiAssembly, null)) upon module unloading.

This is definitely possible but there are 2 wrinkles that make this complicated. The linkage between libmi and the OpenSSL libraries mostly just works but on EL7 and macOS the linked lib path essentially requires either a rewrite of the ELF headers to the correct location or a symlink to point to the proper one. I don't think this is a massive issue as essentially a symlink would be made in that directory which shouldn't be that difficult.

@jborean93, what do you think about this proposal? This will essentially make your forked omi repository the advertised "official" repo for WSMan client support in PowerShell. In fact

I'm not against the idea, I do have some concerns about SetDllImportResolver and how only 1 can be set per assembly. I know it does at least work which is great I just don't want it to break at a future point because PowerShell now adds it's own resolver for x feature.

My only other qualm is how this works from a support perspective, While I'm happy to help out when I can I cannot give any guarantees that any bugs or feature requests will be addressed at all. This is fine if this is a fork and people opt into that knowing the "risks" involved but when something comes with the "official" status that may change the expectation of the users. While I don't wish to cause any offense to anybody who works on this part of PowerShell the builtin implementation is a bit lacking and the fork should provide a better experience it's still not a 1 for 1 comparison with what Windows provides.

I see 4 potential options that are available

• Do nothing and continue the status quo
  • I don't think you can really continue with this due to the reliance on OpenSSL 1.0.x which is fast dissapearing from most distributions
• At least upgrade the builtin libmi lib with a newer one from the OMI repo that is built against OpenSSL 1.1.x
  • I think from the pwsh team's perspective this is a very quick win with very little loss
  • The PowerShell support might not be there anymore but it hasn't stopped the OMI team from producing build https://github.com/microsoft/omi/releases
  • You "should" just be able to take one of the newer ulinux builds for OpenSSL 1.1.x and ship that for distributions that on that version
  • It's essentially the same code from the same team, just compiled against a more modern OpenSSL build which is probably the worst problem with the builtin version today
• "Promote" PSWSMan as the official-ish way of using WSMan on Linux
  • Solves the OpenSSL issue as well as a whole bunch of other things mentioned in my fork
  • From my perspective the most important change is TLS certificate validation being done on HTTPS connection
  • Isn't a nice works out of the box scenario, some people may be wary about external dependencies
  • "Support" is also questionable but that probably depends on how it is documented as the "official" way
  • The TLS cert validation works but it can be difficult to trust your own CA chains, this isn't as straightforward as it would be on Windows
• Scrap all those and actually implement WSMan as a client in C#
  • This is big undertaking and I would love for it to be the case but I don't see it really happening
  • Will solve a whole bunch of issues and would simplify the whole TLS logic behind more well tested and known code that is the .NET library
  • Would also require a lot more work in PowerShell, like the pluginable transports that has been mentioned, for it to actually be possible

I know 4 is a pipe dream and I personally don't have time to work on it and the same probably also applies to anybody on the PowerShell team.

TLDR: I'm happy to investigate this further but have some concerns around limtations of SetDllImportResolver as it applies to the assembly and the perception of support that comes from the "official" status.

[daxian-dbw]

Yeah, I missed that, and thus deleted my previous comment 😄
Setting the callback on S.M.A does increase the risk of confliction. It's possible someone already does that in another module.

In this case, let's use AssemblyLoadContext.ResolvingUnmanagedDll. It's essentially a multi-cast event, so you can simply add/remove your handler targeting the default AssemblyLoadContext upon module loading/unloading.

[jborean93]

Yeah, I missed that, and thus deleted my previous comment smile

Ah my apologies I was just too eager to reply :)

In this case, let's use AssemblyLoadContext.ResolvingUnmanagedDll

Based on my very limited understanding of this, when PowerShell goes to load libpsrpclient and it's no longer in the repo/load path this event will fire and our handler will tell it to try our location as a fallback instead? If so that sounds like it would work quite nicely.

[daxian-dbw]

Regarding the supporting story, my words 'advertised "official" repo' were wrong and misleading. We will never say the PSWSMan is official story for WSMan remoting on Linux/macOS, we will just recommend it to users who wants WSMan on Linux/mac like we may recommend powershell-yaml to someone who is seeking a powershell YAML parser. We will make it very clear that it's a fork and maintained by a community member. @SteveL-MSFT, can you please also comment on this aspect?

[daxian-dbw]

when PowerShell goes to load libpsrpclient and it's no longer in the repo/load path this event will fire and our handler will tell it to try our location as a fallback instead?

Yes, that's how this event works. PowerShell already registered one, called NativeDllHandler. But you will have to wait till powershell stops shipping those 2 libs, otherwise those 2 libs will be resolved and loaded and the event won't be triggered.

There may be some complication when the isolated module kicks in (we have a feature work to make module able to be loaded in separate AssemblyLoadContext, I'm still working on the design of it). But we can worry about that later.

[daxian-dbw]

I see 4 potential options that are available

To me, '"Promote" PSWSMan as the official-ish way of using WSMan on Linux' is the only realistic option. OMI doesn't support powershell anymore, so powershell cannot support the WSMan remoting on Linux/macOS anymore.

If not promoting PSWSMan, then the only option would be waiting for the remoting subsystem to be done, but then it's still up to the community to provide an implementation of the remoting subsystem over WSMan on Linux/macOS, and it would likely be PSWSMan again 😄 So, why not now?

[SteveL-MSFT]

Regarding MMI, it's still needed on Windows for CimCmdlets, but is out of scope for this discussion. The only other thing MMI was used for on Linux/macOS was DSC configuration compilation to a mof file. Since we are no longer using MOF for DSC going forward, it is ok to break this dependency.

I would think PSWSMan would be as official as Pester. We (or CloudShell team) will try to submit PRs where it makes sense, but the community should expect "best effort GitHub based support" for PSWSMan module.

[jborean93]

I guess this also brings into question the "support" statement of the EXO2 module on Linux. The docs https://docs.microsoft.com/en-us/powershell/exchange/exchange-online-powershell-v2?view=exchange-ps states

The latest version of the EXO V2 module is officially supported in PowerShell 7 on Windows, Linux, and Apple macOS.

In particular it singles out Ubuntu 18.04/20.04 and macOS 10.14 or newer. If WSMan is not going to be in box then what level of support for Linux will they have when it comes to this module as it heavily relies on WSMan to connect to the O365 instance. Basically for their module to work on non-Windows they would need to have a dependency on PSWSMan which will work but may have unknown bugs I haven't encountered.

[daxian-dbw]

This is a good question and totally valid concern. Do you know what the current working status of the EXO V2 module is on different PS 7+ versions for different versions of Ubuntu and macOS today? A matrix of where it works and where it doesn't would be very helpful. Also loop in @joeyaiello for comments.

[jborean93]

AFAIK if you can get WSMan working then the modules just work on Linux. I've only done very basic sanity tests for EXO2 but it sort off works. My concern is that an official MS supported module now has a dep on something not maintained by MS.

[daxian-dbw]

That's absolutely a reasonable concern. AFAIK, @joeyaiello had a conversation with Jeffrery Snover earlier this week about the compatibility issues with Officen365 modules, we are following up on that, but it may take a long time to fully resolve them.

[iSazonov]

but it may take a long time to fully resolve them.

Who is the person who could give this a boost? @jpsnover? Maybe it's time to create an experimental project and try something like gRPC?
(Total downloads 13.7M 😺 )

[daxian-dbw]

Even though we won't use SetDllImportResolver for the real implementation, but the prototype can be done using this API, so you don't need to wait for PowerShell package to stop shipping those libs (work will be needed in the Microsoft.PowerShell.Native NuGet package for that). Once it's proven to work properly, it should be easy to move the code to an event handler of AssemblyLoadContext.ResolvingUnmanagedDll.

PowerShell needs some additional changes besides removing those 2 libs -- the code path that creates a new PSSession on WSMan should detect if the required libs are available, and if not, throw the appropriate message pointing to a aka.ms link explaining how to get WSMan support through PSWSMan.

[iSazonov]

I'd like to know when it comes to Enterprises that are still paying for overdue Windows PowerShell. It looks ridiculous to put a patch on another patch. :-) We didn't get WSMan in OOB before and never will. Same with SSH. With this suggestion, we stay in the same place again, or even take a step back (well, to a side :-) ).
There are two suggestions:

• Rename PowerShell to PowerShell Cloud so that it is clear that this is the only place where it works.
• Create a state-of-the-art PowerShell remoting system so that Enterprises get it OOB.

[daxian-dbw]

I'd like to know when it comes to Enterprises that are still paying for overdue Windows PowerShell. It looks ridiculous to put a patch on another patch. :-)

Here we are not talking about Windows PowerShell. Nothing is changed to Windows PowerShell. What do you mean by "a patch on another patch"?

We didn't get WSMan in OOB before and never will. Same with SSH. With this suggestion, we stay in the same place again, or even take a step back (well, to a side :-) ).

It's not very clear what you mean here, could you please elaborate some more?

[iSazonov]

It is impossible to deploy PowerShell 7 in Enterprise and get it working in heterogenous environment.

[iSazonov]

Here we are not talking about Windows PowerShell. Nothing is changed to Windows PowerShell. What do you mean by "a patch on another patch"?

No matter how you package the problem it remains a problem and it is not possible to use the remoting in Enterprise.

[daxian-dbw]

If there are gaps that prevent the SSH remoting from working out of the box, then efforts should be made to close those gaps.
As for WSMan remoting on Linux/macOS, as far as I understand, we are moving away from it simply because it's not supported anymore. Maybe @PaulHigin or @SteveL-MSFT can chime in to talk a bit more about why, and how we are going to deal with Exchange Online scenario.

[Jaykul]

As far as I can tell, it's up to you (Microsoft) to support WSMan and WinRM. It's your tech and your library, and it always has been. Nowadays, we really only need it to connect to your services and your operating systems ...

It feels disingenuous to say "it's not supported anymore" as if you'd taken a dependency on some 3rd party library that is no longer being updated.

There needs to be a lot better communication about why you're retiring this, and a default-on story for connecting to Windows Servers....

[briantist]

If there are gaps that prevent the SSH remoting from working out of the box, then efforts should be made to close those gaps.

@daxian-dbw Even though SSH on Windows continue to improve (Kerberos auth, etc.), using it for PowerShell remoting is still a nonstarter for me until PSRP over SSH supports JEA. It's 2024 and still there's no JEA support.

I know it never really caught on as well as it could have but in my opinion, JEA is the killer feature of PSRP. I am only finding more uses for it in a cloud-heavy world, and I find myself still stuck with Windows PowerShell 5.1 and a frustrating lack of support from Microsoft on how to connect to these endpoints from non-Windows platforms.

[jborean93]

@daxian-dbw I've had a play with this today and feel like this should work going forward

Add-Type -TypeDefinition @'
using System;
using System.Management.Automation;
using System.Reflection;
using System.Runtime.InteropServices;
using System.Runtime.Loader;

namespace PSWSMan
{
    public class Resolver : IDisposable
    {
        private static readonly string PSRP_LIB_NAME = "libpsrpclient";

        private string libPath = null;
        private IntPtr libHandle = IntPtr.Zero;

        public Resolver(string libPath, bool setPwshResolver)
        {
            this.libPath = libPath;
            AssemblyLoadContext.Default.ResolvingUnmanagedDll += ResolvePSRPClient;

            if (setPwshResolver)
            {
                try
                {
                    NativeLibrary.SetDllImportResolver(typeof(PSObject).Assembly, ImportResolver);
                }
                catch (InvalidOperationException) {}  // Only 1 resolver allowed per assembly
            }
        }

        private IntPtr ImportResolver(string libraryName, Assembly assembly, DllImportSearchPath? searchPath)
        {
            if (libraryName == PSRP_LIB_NAME)
                return GetPSRPClientLibrary(assembly, searchPath);

            return NativeLibrary.Load(libraryName, assembly, searchPath);
        }

        private IntPtr ResolvePSRPClient(Assembly assembly, string libraryName)
        {
            if (libraryName != PSRP_LIB_NAME)
                return IntPtr.Zero;

            return GetPSRPClientLibrary(assembly, null);
        }

        private IntPtr GetPSRPClientLibrary(Assembly assembly, DllImportSearchPath? searchPath)
        {
            if (libHandle == IntPtr.Zero)
                libHandle = NativeLibrary.Load(libPath, assembly, searchPath);

            return libHandle;
        }

        public void Dispose()
        {
            if (libHandle != IntPtr.Zero)
            {
                NativeLibrary.Free(libHandle);
                libHandle = IntPtr.Zero;
            }

            AssemblyLoadContext.Default.ResolvingUnmanagedDll -= ResolvePSRPClient;
            GC.SuppressFinalize(this);
        }
        ~Resolver() { this.Dispose(); }
    }
}
'@

# Will be selected at runtime based on the host environment - just used for testing here
$libPath = "/home/jborean/dev/omi/PSWSMan/lib/glibc-1.1/libpsrpclient.so"

# Will check if pwsh still ships libpsrpclient, if so it's going to call SetDllImportResolver to ensure
# pwsh no longer uses the shipped copy. If not present then SetDllImportResolver is not called
# so that in case pwsh adds it's own resolver won't impact this module
$pwshDir = Split-Path -Path ([PSObject].Assembly.Location) -Parent
$libPresent = Test-Path -LiteralPath (Join-Path $pwshDir libpsrpclient.so)

# Will live for the scope of the module - disposes itself on unload
[PSWSMan.Resolver]::new($libPath, $libPresent)

Do you see any glaring problems with this approach? It should work today where the builtin libs are present and in the future where they are no longer present with PowerShell. Things are a bit more complicated on macOS as it needs to set up some symlinks for the OpenSSL libs at runtime but I think I have a separate solution for that which I don't hate too much.

The only downside I can see with this approach over the current just copy into the pwsh dir is that someone needs to import this module. If any WSMan commands are run then the existing libpsrpclient is loaded forever and the user would need to restart pwsh and import the module first. This should hopefully be less of a problem in the future when libpsrpclient is no longer shipped.

[daxian-dbw]

This is great! Glad that you get it to work!
nitpick:

  public Resolver(string libPath, bool setPwshResolver)
  {
      this.libPath = libPath;
      AssemblyLoadContext.Default.ResolvingUnmanagedDll += ResolvePSRPClient;  //<=== maybe skip this when 'setPwshResolver' is true?

      if (setPwshResolver)
          ....

  private IntPtr ImportResolver(string libraryName, Assembly assembly, DllImportSearchPath? searchPath)
  {
      if (libraryName == PSRP_LIB_NAME)
          return GetPSRPClientLibrary(assembly, searchPath);

      return NativeLibrary.Load(libraryName, assembly, searchPath);  //<=== shouldn't you return IntPtr.Zero here?
  }

[daxian-dbw]

... is that someone needs to import this module

Right, previously the user only needs to run Install-WSMan once. We will improve this experience with the PS subsystem work.

If any WSMan commands are run then the existing libpsrpclient is loaded forever and the user would need to restart pwsh and import the module first.

I didn't try the native dll loading behavior myself, so the second loading of libpsrpclient.so will fail?

[jborean93]

<=== maybe skip this when 'setPwshResolver' is true?

I'll probably just keep it as it is to make the dispose behaviour simpler (always remove the event handler). I can't think of a downside to this except the event is never going to do any work.

<=== shouldn't you return IntPtr.Zero here?

I'll have to play around with this a bit more but it looks like you are right and returning IntPtr.Zero will just fallback to the existing lookup logic. Thanks for the pickup.

We will improve this experience with the PS subsystem work.

Will be interested to hear more about this when more details are known. Making sure you run Import-Module isn't the end of the world it's just a note that it's a change in behaviour in how PSWSMan works today.

I didn't try the native dll loading behavior myself, so the second loading of libpsrpclient.so will fail?

Yea once a native library is loaded it will stay loaded for that assembly (maybe even process wide?). I'm not aware of a way to unload these libraries outside of changing how the code ultimately calls the functions in that library which is outside the control of PSWSMan. This is somewhat lessened as in most cases libpsrpclient will just fail to load on newer distros so even one of the WSMan methods are called before Import-Module is run there is no library loaded and thus importing will get it working again. It only affects setups where libpsrpclient shipped by PowerShell is still usable but someone wishes to use the one in PSWSMan without importing the module first.

[daxian-dbw]

@jborean93 Thanks for the clarification!

[iSazonov]

I wonder we are trying to resolve the fundamental problem with PowerShell remoting based on the principle of minimal change.
It would be more correct to create a new modern design to support different implementations. We already have a RFC for this, but it is also based on the principle of minimal change and would rather confuse and make things worse.
Why not make it a subsystem? And not create a new one based on gRPC, for example?
It would be a great fundamental investment for years to come. MSFT team has been talking about it for years. Isn't the next non-LTS milestone the most appropriate time for this work, when MSFT has completed another cycle of OS (Windows 11) development?

[jborean93]

It would be more correct to create a new modern design to support different implementations

You might be right but until then things need to continue working. Having some new gRPC mechanism, or subsystem for remoting won't magically allow you to connect to Windows hosts (without ssh) or Exchange endpoints. I might disagree with the dropping of work with WSMan while it's still prevalent in today's environments but unless MS change their mind the best I can do is maintain my own fork.

[iSazonov]

I am very grateful for your work! It will be in demand even if we get alternatives.

But we are in an absurd situation for many years now where WSMan remoting does not work well OOB on Unix and SSH remoting on Windows. At the same time, MSFT announced PowerShell Core as the only area for innovation against Windows PowerShell.
I'm still hoping that we can join forces to push innovation faster - there are too many compelling and pressing ideas gathering dust on the shelf.