Plugins do not function under Vault container

[ilsaloving]

Describe the bug
I have vault deployed via container and I am trying to add the plugin for Keeper Secrets Manager. When I try to register the plugin, I get this in the logs:

vault-server_1  | 2024-04-19T16:05:03.939Z [DEBUG] core: attempting to load backend plugin: name=vault-plugin-secrets-ksm
vault-server_1  | 2024-04-19T16:05:03.943Z [DEBUG] core: spawning a new plugin process: plugin_name=vault-plugin-secrets-ksm id=Q9KS9977Oc
vault-server_1  | 2024-04-19T16:05:04.062Z [DEBUG] core: failed to dispense v5 backend plugin: name=vault-plugin-secrets-ksm error="fork/exec /vault/plugins/vault-plugin-secrets-ksm: no such file or directory"
vault-server_1  | 2024-04-19T16:05:04.174Z [DEBUG] core: failed to dispense v4 backend plugin: name=vault-plugin-secrets-ksm error="fork/exec /vault/plugins/vault-plugin-secrets-ksm: no such file or directory"
vault-server_1  | 2024-04-19T16:05:04.175Z [WARN]  core: Error determining plugin version:
vault-server_1  |   error=
vault-server_1  |   | 1 error occurred:
vault-server_1  |   | \t* fork/exec /vault/plugins/vault-plugin-secrets-ksm: no such file or directory

If I try to execute it from shell, I get
/vault/plugins # ./vault-plugin-secrets-ksm
/bin/sh: ./vault-plugin-secrets-ksm: not found

To Reproduce
Steps to reproduce the behavior:

1. Create and configure a new container using the latest hashicorp vault image
2. Initialize and unseal as per documentation
3. Copy the plugin (https://github.com/Keeper-Security/secrets-manager/releases/tag/vault-plugin-secrets-ksm%2Fv1.0.2) to the configured plugin directory
4. Attempt to register the plugin
5. See error

Expected behavior
I expected it to register

Environment:
Container created via docker-compose, using hashicorp/vault:latest image

• Vault Server Version (retrieve with vault status): v1.16.1
• Vault CLI Version (retrieve with vault version): v1.16.1
• Server Operating System/Architecture: Ubuntu host, docker v24.0.5

Vault server configuration file(s):

{
    "listener": [
        {
            "tcp": {
                "address": "0.0.0.0:8200",
                "tls_disable": 1
            }
        }
    ],
    "storage": {
        "file": {
            "path": "/vault/file"
        }
    },
    "max_lease_ttl": "10h",
    "default_lease_ttl": "10h",
    "ui": true,
    "plugin_directory": "/vault/plugins",
    "log_level": "trace"
}

Additional context
I have attempted to repeat the process on my mac using arm64 images, and it gives different errors:

vault-server-1  | 2024-04-19T16:15:14.552Z [DEBUG] core: attempting to load backend plugin: name=ksm
vault-server-1  | 2024-04-19T16:15:14.552Z [DEBUG] core: spawning a new plugin process: plugin_name=ksm id=pajAbRvGuc
vault-server-1  | 2024-04-19T16:15:14.569Z [DEBUG] core: failed to dispense v5 backend plugin: name=ksm
vault-server-1  |   error=
vault-server-1  |   | Unrecognized remote plugin message:
vault-server-1  |   | Failed to read any lines from plugin's stdout
vault-server-1  |   | This usually means
vault-server-1  |   |   the plugin was not compiled for this architecture,
vault-server-1  |   |   the plugin is missing dynamic-link libraries necessary to run,
vault-server-1  |   |   the plugin is not executable by this process due to file permissions, or
vault-server-1  |   |   the plugin failed to negotiate the initial go-plugin protocol handshake
vault-server-1  |   |
vault-server-1  |   | Additional notes about plugin:
vault-server-1  |   |   Path: /vault/plugins/ksm
vault-server-1  |   |   Mode: -rwxr-xr-x
vault-server-1  |   |   Owner: 0 [root] (current: 100 [vault])
vault-server-1  |   |   Group: 0 [root] (current: 1000 [vault])
vault-server-1  |   |   ELF architecture: EM_AARCH64 (current architecture: arm64)
vault-server-1  |
vault-server-1  | 2024-04-19T16:15:14.581Z [DEBUG] core: successfully dispensed v4 backend plugin: name=ksm

I suspect that it is still the same issue, just with a different symptom.

Based on the research I did, I believe the issue has to do with using Alpine as the base of the container.
#8009
#17250
https://megamorf.gitlab.io/2019/09/08/alpine-go-builds-with-cgo-enabled/

While one possible option is for the developer to not use dynamic linking, I don't think this is reasonable. If the plugin works fine on a regular host, there should be no reason for it to fail in a container.
The 3rd link I provided suggests a solution. If that is not possible, then an alternative image should be provided using a different base distro.

[ilsaloving]

As an experiment, I created my own image using ubuntu. I got this when I registered the plugin:

vault-server_1  | 2024-04-19T21:48:25.565Z [DEBUG] core: attempting to load backend plugin: name=vault-plugin-secrets-ksm
vault-server_1  | 2024-04-19T21:48:25.565Z [DEBUG] core: spawning a new plugin process: plugin_name=vault-plugin-secrets-ksm id=2HcWuiKynM
vault-server_1  | 2024-04-19T21:48:25.929Z [DEBUG] core: failed to dispense v5 backend plugin: name=vault-plugin-secrets-ksm
vault-server_1  |   error=
vault-server_1  |   | Unrecognized remote plugin message:
vault-server_1  |   | Failed to read any lines from plugin's stdout
vault-server_1  |   | This usually means
vault-server_1  |   |   the plugin was not compiled for this architecture,
vault-server_1  |   |   the plugin is missing dynamic-link libraries necessary to run,
vault-server_1  |   |   the plugin is not executable by this process due to file permissions, or
vault-server_1  |   |   the plugin failed to negotiate the initial go-plugin protocol handshake
vault-server_1  |   |
vault-server_1  |   | Additional notes about plugin:
vault-server_1  |   |   Path: /opt/vault/plugins/vault-plugin-secrets-ksm
vault-server_1  |   |   Mode: -rwxr-xr-x
vault-server_1  |   |   Owner: 0 [root] (current: 0 [root])
vault-server_1  |   |   Group: 0 [root] (current: 0 [root])
vault-server_1  |   |   ELF architecture: EM_X86_64 (current architecture: amd64)
vault-server_1  |
vault-server_1  | 2024-04-19T21:48:26.038Z [DEBUG] core: successfully dispensed v4 backend plugin: name=vault-plugin-secrets-ksm

But apart from that, everything appears to be working. I'm attaching my Dockerfile.txt to this comment for anyone who is interested.
Note that this embeds my own config file that I've been using to triage this whole thing, so you should make sure to remove/modify it before building an image for production.