Calling C# SDK from Python results in "Unable to load DLL 'Yubico.NativeShims'" error #47
Comments
|
Ping @coonsd. |
|
I'll take a look at the sample project. But this is outside of our usual supported scenarios. I do want to make it very clear though - .NET is a managed memory runtime, just like Python. While we do have some better facilities for managing sensitive memory in .NET, we too are not able to clear every instance in every case. There are cases where scrubbing the memory completely is essentially impossible - and we made the call at the time to clear memory in a best effort fashion. To be more specific: The .NET memory allocator and garbage collector can relocate memory at any time. That means, even if we clear the memory using the variable (handle) that we have, the data may still exist in an older and now inaccessible region of memory. Given some signals we saw from the .NET project as well as some internal security audits, we arrived at the conclusion that while non-zero, the risk was so small, and the attack vector so difficult, that we accepted this risk. We talk a little bit about that in this document: https://docs.yubico.com/yesdk/users-manual/sdk-programming-guide/sensitive-data.html Essentially, we consider that if someone has the ability to dump your process RAM and access it - there's really not much we can do. At some point - I did see some chatter within the .NET project to add a runtime flag or something that would zero out old memory locations when a relocation occurs. You may want to investigate whether or not that has been implemented on their side and how to activate it. If you need full memory protection, have you considered using the C library libykpiv instead? That would not be subject to the relocation issues, and I'm quite certain that we zero out sensitive data after use. |
|
Thanks for looking into this @GregDomzalski. I certainly understand there are limits to what you can support. I agree with everything you say. At the same time, I have the dilemma that the requirement for memory wipe is something I need to fulfill. What I’m doing now is investigating every technical possibility to fulfill it. Regarding libykpiv. I believe you haven’t published that as a separate project, right? At least I don’t find it as a separate repo. Only as part of the yubico-piv-tool repo. I did start to hack away at it but it involves calling some undocumented functions and basically reverse engineering yubico-piv-tool. It’s one path forward but it’s a time consuming one. Also, the C language is responsible for almost every security vulnerability ever so one must be careful. By trying to increase security I may end up in a situation where I make it worse. I believe using C# and accepting the best effort nature of the current limitations would be preferable. The error I describe in this issue is interesting in that it's the same that David reported on the vstest project. It leads me to believe it might have something to do with how the SDK is published. If that means there is a configuration issue or actually a bug in dotnet publish is difficult to tell. So I really appreciate you taking the time to look into it. |
|
Took a look at your repro setup. Thank you very much for providing such a stripped down repro! I believe this is a separate bug than what we reported on VSTest. Same symptom but different underlying causes. In your case, you can fix this by copying the correct version of the Yubico.NativeShims.dll file into the path where you're running python from. So for example, I ran Unfortunately .NET has very poor guidance on how library authors should create, package, and integrate native components. The setup we have tends to work well enough within the .NET ecosystem (except for the VSTest bug we opened). It gets tricky once you exit the I don't think there's any action that I can take here other than advising you of this workaround. Your python build process would need to be enlightened about this DLL, or Pythonnet would need to somehow understand and interpret the Sorry I don't really have a better answer that this. Hopefully this workaround is workable for your needs. |
|
@GregDomzalski For some reason that doesn't work on Windows. I get the same error even when I copy the file to the same folder as the python script. Here's my dotnet details: All of this is running inside a powershell shell with this info: Using this conda environment: |
|
I would expect that you need to put the DLL in the same folder as the executable that is being run. This might be Python. There's also the heavy hammer of putting the DLL in What you could do that may help inform a better decision is to use ProcMon to see what process is trying to load NativeShims.dll and where it is looking for it. https://learn.microsoft.com/en-us/sysinternals/downloads/procmon You can set up a filter on the path and look for things that contain "Yubico.NativeShims.dll". I would expect that when you run your program, we would get a bunch of "FILE_NOT_FOUND" entries and various paths that contain the DLL that don't actually exist. Typically the first path that it tries will be the one that we'd like to use. |
|
Funny thing. The only places it seems to read is:
So I copied it to C:\Windows\System32\WindowsPowerShell\v1.0\Yubico.NativeShims without .dll, otherwise it doesn't work. And now I am getting a issue with system.memory. But this is progress. The question is why is it only loading those two locations and why no dll extension. |
|
So, there are multiple versions of our assemblies being packaged. There's one for net472 and one for netstandard20. If you are targeting the PowerShell that is built into Windows (which it looks like from those paths), you should be using the net472 version. That should have the correction to use the .dll extension and may also resolve some of the other System.Memory and other dependency issues. |
|
Switching to building for net472, fix that the Yubico.NativeShims didnt have a dll extention. But I still have the different versions of System.Memory. Looks like I need to figure out how to do bindingRedirect. But this is really helpful. I would love not to need to put the DLL in the protected folders but that might be out of your hands. |
|
Sorry, I meant that you need the net472 version of Yubico.YubiKey, Yubico.Core, and Yubico.DotnetPolyfills. You can likely remain using the Yubico.NativeShims version that you have already been using. |
|
Yes, unfortunately where the DLL is loaded is up to Windows and not us 😄 |
|
Sorry I didnt explain enough. The reason why it took so long for me to get back, is that I learned how to do a compiled powershell module. So I just set the target and then did a dotnet build and process takes the dlls from %HOMEPATH%.nuget\packages\yubico.yubikey\1.7.0 etc.. And since my code is now a compiled DLL i think I need to understand bindingredirects as yubico.yubikey and its dependency:
|
|
My GOD, I Am so sorry.. I have replied to the wrong issue, I was reading this and then replied in the wrong.. So sorry for all involved. |
|
I'll leave the comments above, as they may actually be relevant to this thread as well. Just replace PowerShell and Python and the same investigations would be the same 😃 |
|
@GregDomzalski Nice trick with procmon! Tried it and it gave these clues: I copied the file to one of those locations and it worked! Even though it's not great idea to copy dll files around like that it would work as a workaround. We can see with procmon that it tries a bunch of paths. I'm not sure what is suggesting those. Do you know if this is a peculiarity of python or windows? And of course, do you know if there is a way to affect it somehow? |
|
I found that you can use import os
with os.add_dll_directory(r"C:\git\pythonnet_test\cs\testlib\out\runtimes\win-x64\native"):
from pythonnet import load
runtimeconfig_path = r"C:\git\pythonnet_test\cs\testlib\out\testlib.runtimeconfig.json"
load("coreclr", runtime_config=runtimeconfig_path)
import clr
import sys
sys.path.append(r"C:\git\pythonnet_test\cs\testlib\out")
clr.AddReference("testlib")
from testlib import Class1
n = Class1().CountDevices()
print(f"Found {n} devices") |
My team uses python for analytics purposes. We have a need to encrypt large datasets and to protect the one-time random symmetric encryption keys with a certificate on a YubiKey. We also have the requirement to do memory wipe of the memory that stores the symmetric encryption key. All the normal encryption packages for python store their encryption keys in a “bytes” object, and makes lots of internal copies of the encryption key. A bytes object in python is defined as immutable. I.e. it’s impossible to do memory wipe in python. My goal is to handle all encryption, certificate, and memory wipe related things with your C# SDK and package it in a dll that is called from python via the pythonnet package.
Unfortunately even the simplest thing results in this exception:
I have prepared a sample project to replicate the issue here:
https://github.com/johanrex/pythonnet_test
In order to replicate it:
This is the same problem that David Coons have reported here:
microsoft/vstest#4343
I have created a ticket for it (converted from issue to discussion by the pythonnet developer) here, but realistically I’m not getting any further on that:
pythonnet/pythonnet#2124
I have written to Yubico support (ticket 420093). Unfortunately, the proposed solution there can’t be applied to my contrived scenario with python->C# interop.
I do have odd requirements, which results in this contrived solution. One can have opinions on how reasonable the memory wipe requirement is, but the requirement will not go away. I really hope it's possible to find a way to use the C# SDK from Python. Seeing as you have the same issue with vstest perhaps there is something weird going on when publishing.
The text was updated successfully, but these errors were encountered: