linux binary (0.29.0+) depends on glibc, won't run on alpine as-is #3499

johanneslarsson · 2021-05-28T08:23:26Z

Expected Behavior

./opa-v0.29.1 version
Version: 0.29.1
Build Commit: 3155647
Build Timestamp: 2021-05-27T20:48:51Z
Build Hostname: df4767d358f5
Go Version: go1.16.3
WebAssembly: available
root@9f838d2f1a0f:/# uname -a
Linux 9f838d2f1a0f 5.10.25-linuxkit #1 SMP Tue Mar 23 09:27:39 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Actual Behavior

/bin/sh: ./opa-v0.29.1: not found

Steps to Reproduce the Problem

docker run -it alpine:3.13.2
curl -L -o opa-v0.29.1 https://openpolicyagent.org/downloads/v0.29.1/opa_linux_amd64
chmod +x opa-v0.29.1

Additional Info

Works fine in latest Ubuntu for example

docker run -it ubuntu
root@9f838d2f1a0f:/# apt update
root@9f838d2f1a0f:/# apt upgrade
root@9f838d2f1a0f:/# apt install curl
root@9f838d2f1a0f:/# curl -L -o opa-v0.29.1 https://openpolicyagent.org/downloads/v0.29.1/opa_linux_amd64
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    81  100    81    0     0    100      0 --:--:-- --:--:-- --:--:--   100
100   621  100   621    0     0    466      0  0:00:01  0:00:01 --:--:--  4085
100 49.4M  100 49.4M    0     0  13.8M      0  0:00:03  0:00:03 --:--:-- 28.1M
root@9f838d2f1a0f:/# chmod +x opa-v0.29.1 
root@9f838d2f1a0f:/# ./opa-v0.29.1 version
Version: 0.29.1
Build Commit: 3155647
Build Timestamp: 2021-05-27T20:48:51Z
Build Hostname: df4767d358f5
Go Version: go1.16.3
WebAssembly: available
root@9f838d2f1a0f:/# uname -a
Linux 9f838d2f1a0f 5.10.25-linuxkit #1 SMP Tue Mar 23 09:27:39 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

The text was updated successfully, but these errors were encountered:

srenatus · 2021-05-28T08:25:18Z

When enabling Wasm for all the platform binaries, we've gotten ourselves a libc runtime dependency, it seems:

/ # ./src/opa_linux_amd64 version
/bin/sh: ./src/opa_linux_amd64: not found
/ # ldd /src/opa_linux_amd64
        /lib64/ld-linux-x86-64.so.2 (0x7f377d3af000)
        libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x7f377d3af000)
        libm.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f377d3af000)
        libdl.so.2 => /lib64/ld-linux-x86-64.so.2 (0x7f377d3af000)
Error loading shared library libgcc_s.so.1: No such file or directory (needed by /src/opa_linux_amd64)
        libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f377d3af000)
Error loading shared library ld-linux-x86-64.so.2: No such file or directory (needed by /src/opa_linux_amd64)
Error relocating /src/opa_linux_amd64: _Unwind_Resume: symbol not found
Error relocating /src/opa_linux_amd64: __register_atfork: symbol not found
Error relocating /src/opa_linux_amd64: __register_frame: symbol not found
Error relocating /src/opa_linux_amd64: _Unwind_GetIPInfo: symbol not found
Error relocating /src/opa_linux_amd64: __res_init: symbol not found
Error relocating /src/opa_linux_amd64: _Unwind_GetDataRelBase: symbol not found
Error relocating /src/opa_linux_amd64: _Unwind_GetRegionStart: symbol not found
Error relocating /src/opa_linux_amd64: _Unwind_SetGR: symbol not found
Error relocating /src/opa_linux_amd64: _Unwind_FindEnclosingFunction: symbol not found
Error relocating /src/opa_linux_amd64: _Unwind_GetTextRelBase: symbol not found
Error relocating /src/opa_linux_amd64: _Unwind_DeleteException: symbol not found
Error relocating /src/opa_linux_amd64: __deregister_frame: symbol not found
Error relocating /src/opa_linux_amd64: _Unwind_RaiseException: symbol not found
Error relocating /src/opa_linux_amd64: _Unwind_GetIP: symbol not found
Error relocating /src/opa_linux_amd64: _Unwind_Backtrace: symbol not found
Error relocating /src/opa_linux_amd64: _Unwind_GetCFA: symbol not found
Error relocating /src/opa_linux_amd64: _Unwind_GetLanguageSpecificData: symbol not found
Error relocating /src/opa_linux_amd64: _Unwind_SetIP: symbol not found
/ #

(this is using alpine:latest)

srenatus · 2021-05-28T08:26:55Z

If this blocks you, you should be able to build a binary without all that using make build-linux, or make ci-go-build-linux if you're not on linux but have docker available.

srenatus · 2021-05-28T08:42:54Z

👉 bytecodealliance/wasmtime-go#21

So, here's a cross road:

North: Wait for wasmtime to build a musl version, and use that for extra musl binaries (stretch: do the same thing ourselves)
East: Disable wasm for the linux binaries
West: Publish extra linux binaries without wasm

@johanneslarsson this is probably a problem for a few folks, but since you've come across it first: How strongly are you attached to using the alpine image? The one we're using to build the docker container is pretty slim, too: gcr.io/distroless/cc.

There seems to be an alpine-glibc image, too, as another workaround:

$ docker run -it -v $(pwd):/src frolvlad/alpine-glibc
Unable to find image 'frolvlad/alpine-glibc:latest' locally
latest: Pulling from frolvlad/alpine-glibc
540db60ca938: Already exists
d777323df2e1: Pull complete
Digest: sha256:2435a13068b6e9331b91c200dcba33e0f0c06cdbbc689fa856a753e3f093246a
Status: Downloaded newer image for frolvlad/alpine-glibc:latest
/ # /src/_release/0.30.0-dev/opa_linux_amd64
An open source project to policy-enable your service.

Usage:
  opa_linux_amd64 [command]

Available Commands:
  bench       Benchmark a Rego query
  build       Build an OPA bundle
  check       Check Rego source files
  deps        Analyze Rego query dependencies
  eval        Evaluate a Rego query
  fmt         Format Rego source files
  help        Help about any command
  parse       Parse Rego source file
  run         Start OPA in interactive or server mode
  sign        Generate an OPA bundle signature
  test        Execute Rego test cases
  version     Print the version of OPA

Flags:
  -h, --help   help for opa_linux_amd64

Use "opa_linux_amd64 [command] --help" for more information about a command.
/ #

johanneslarsson · 2021-05-28T08:44:16Z

If this blocks you, you should be able to build a binary without all that using make build-linux, or make ci-go-build-linux if you're not on linux but have docker available.

It doesn't really block us. Just trying to stay up to date 🥇 . I probably want to avoid building it ourselves for now.

johanneslarsson · 2021-05-28T08:47:40Z

👉 bytecodealliance/wasmtime-go#21

So, here's a cross road:

North: Wait for wasmtime to build a musl version, and use that for extra musl binaries (stretch: do the same thing ourselves)
East: Disable wasm for the linux binaries
West: Publish extra linux binaries without wasm

@johanneslarsson this is probably a problem for a few folks, but since you've come across it first: How strongly are you attached to using the alpine image? The one we're using to build the docker container is pretty slim, too: gcr.io/distroless/cc.

There seems to be an alpine-glibc image, too, as another workaround:
$ docker run -it -v $(pwd):/src frolvlad/alpine-glibc
Unable to find image 'frolvlad/alpine-glibc:latest' locally
latest: Pulling from frolvlad/alpine-glibc
540db60ca938: Already exists
d777323df2e1: Pull complete
Digest: sha256:2435a13068b6e9331b91c200dcba33e0f0c06cdbbc689fa856a753e3f093246a
Status: Downloaded newer image for frolvlad/alpine-glibc:latest
/ # /src/_release/0.30.0-dev/opa_linux_amd64
An open source project to policy-enable your service.

Usage:
  opa_linux_amd64 [command]

Available Commands:
  bench       Benchmark a Rego query
  build       Build an OPA bundle
  check       Check Rego source files
  deps        Analyze Rego query dependencies
  eval        Evaluate a Rego query
  fmt         Format Rego source files
  help        Help about any command
  parse       Parse Rego source file
  run         Start OPA in interactive or server mode
  sign        Generate an OPA bundle signature
  test        Execute Rego test cases
  version     Print the version of OPA

Flags:
  -h, --help   help for opa_linux_amd64

Use "opa_linux_amd64 [command] --help" for more information about a command.
/ #

Since I haven't got the sales pitch for WASM, (yet). I would just say: https://www.youtube.com/watch?v=LNBjMRvOB5M&ab_channel=PetShopBoysParlophone

srenatus · 2021-05-28T09:25:13Z

So, I think the only sweet spot is building the OPA binary with musl linked in. It'll run on any linux system (🤞), and we can still keep wasm enabled. A prerequisite is musl support in wasmtime and wasmtime-go. I'm looking into that... 🔍

johanneslarsson · 2021-05-28T10:38:50Z

@srenatus I see that I forgot to answer the image question, without going into details, let's say that it would be more work than replacing the FROM in the Dockerfile.

srenatus · 2021-05-28T12:08:46Z

I'll head north for a bit and see how far it is... (😉 bytecodealliance/wasmtime#2951)

johanneslarsson · 2021-05-31T07:08:49Z

I'll head north for a bit and see how far it is... (😉 bytecodealliance/wasmtime#2951)

Thanks. Let me know if you reach Oslo and I'll offer you a cold one (🍺).

srenatus · 2021-05-31T08:15:55Z

OK, it's windy and cold up north. I'll try going west today 🚂

anderseknert · 2021-05-31T08:34:19Z

Let me know if you reach Oslo and I'll offer you a cold one (🍺).

Considering the price of a beer in Oslo, that's an extremely generous offer @johanneslarsson 😆

Fixes open-policy-agent#3499. Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>

* build: add static (wasm-disabled) linux build Fixes #3499. Also: * build: deprecate 'release' and 'release-local' targets that aren't used in our build anymore, and will go away eventually. Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>

srenatus · 2021-05-31T09:19:08Z

I'll get a 0.29.4 out with these binaries included.

srenatus · 2021-05-31T10:16:09Z

☝️ There's an extra binary for your use case now, in v0.29.4

johanneslarsson · 2021-05-31T12:18:40Z

Yay, thanks will try it out.

srenatus · 2021-06-01T13:18:49Z

Similar things happen on Centos 7, it seems:

$ docker run -it -v $(pwd):/src centos:7
Unable to find image 'centos:7' locally
7: Pulling from library/centos
2d473b07cdd5: Pull complete
Digest: sha256:0f4ec88e21daf75124b8a9e5ca03c37a5e937e0e108a255d890492430789b60e
Status: Downloaded newer image for centos:7
[root@e441d40aa548 /]# cd /src
[root@e441d40aa548 src]# ls -a
.  ..  opa_darwin_amd64  opa_linux_amd64  opa_linux_amd64_static  opa_windows_amd64.exe
[root@e441d40aa548 src]# ldd opa_linux_amd64
./opa_linux_amd64: /lib64/libc.so.6: version `GLIBC_2.18' not found (required by ./opa_linux_amd64)
        linux-vdso.so.1 =>  (0x00007ffc3d5c4000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fb6ddd43000)
        libm.so.6 => /lib64/libm.so.6 (0x00007fb6dda41000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007fb6dd83d000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fb6dd627000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fb6dd259000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fb6ddf5f000)

srenatus · 2021-10-21T08:28:26Z

Experimented with it a bit more here: srenatus#9

It looks like it works, except there are a few warnings at build time:

CGO_ENABLED=1 GOFLAGS="-buildmode=exe" go build -tags=opa_wasm -o opa_linux_amd64 -ldflags "-X github.com/open-policy-agent/opa/version.Version=0.34.0-dev -X github.com/open-policy-agent/opa/version.Vcs=4dafd36 -X github.com/open-policy-agent/opa/version.Timestamp=2021-10-21T08:20:37Z -X github.com/open-policy-agent/opa/version.Hostname=e8e714829ae0 -extldflags=-static"
# github.com/open-policy-agent/opa
/usr/bin/ld: /tmp/go-link-1278712680/000007.o: in function `mygetgrouplist':
/usr/local/go/src/os/user/getgrouplist_unix.go:18: warning: Using 'getgrouplist' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: /tmp/go-link-1278712680/000006.o: in function `mygetgrgid_r':
/usr/local/go/src/os/user/cgo_lookup_unix.go:40: warning: Using 'getgrgid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: /tmp/go-link-1278712680/000006.o: in function `mygetgrnam_r':
/usr/local/go/src/os/user/cgo_lookup_unix.go:45: warning: Using 'getgrnam_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: /tmp/go-link-1278712680/000006.o: in function `mygetpwnam_r':
/usr/local/go/src/os/user/cgo_lookup_unix.go:35: warning: Using 'getpwnam_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: /tmp/go-link-1278712680/000006.o: in function `mygetpwuid_r':
/usr/local/go/src/os/user/cgo_lookup_unix.go:30: warning: Using 'getpwuid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: /tmp/go-link-1278712680/000004.o: in function `_cgo_3c1cec0c9a4e_C2func_getaddrinfo':
/tmp/go-build/cgo-gcc-prolog:58: warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
make[1]: Leaving directory '/src'

...I can't tell if they're a show-stopper for this.

srenatus changed the title ~~Binary file opa_linux_amd64 of versions 0.29.0 / 0.29.1 don't run in Alpine Linux (3.13.2, 3.13.5)~~ linux binary (0.29.0, 0.29.1) depends on glibc, won't run on alpine as-is May 28, 2021

srenatus changed the title ~~linux binary (0.29.0, 0.29.1) depends on glibc, won't run on alpine as-is~~ linux binary (0.29.0+) depends on glibc, won't run on alpine as-is May 28, 2021

This was referenced May 30, 2021

Update opa to 0.29.1 cloudposse/packages#1330

Merged

Update opa to 0.29.2 cloudposse/packages#1335

Merged

Update opa to 0.29.3 cloudposse/packages#1338

Merged

srenatus mentioned this issue May 31, 2021

build: add static (wasm-disabled) linux build #3511

Merged

srenatus self-assigned this May 31, 2021

srenatus added this to In Progress in Open Policy Agent May 31, 2021

srenatus added a commit to srenatus/opa that referenced this issue May 31, 2021

build: add static (wasm-disabled) linux build

ac04d3c

Fixes open-policy-agent#3499. Signed-off-by: Stephan Renatus <stephan.renatus@gmail.com>

srenatus closed this as completed in #3511 May 31, 2021

Open Policy Agent automation moved this from In Progress to Done May 31, 2021

Nuru mentioned this issue May 31, 2021

Update opa to 0.29.4 cloudposse/packages#1342

Merged

srenatus mentioned this issue Jun 1, 2021

linux build depends on glibc 2.18+ #3524

Closed

tsandall mentioned this issue Jun 4, 2021

/bin/sh: ./opa: not found in Docker when upgrading to v0.29.4 #3532

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

linux binary (0.29.0+) depends on glibc, won't run on alpine as-is #3499

linux binary (0.29.0+) depends on glibc, won't run on alpine as-is #3499

johanneslarsson commented May 28, 2021

srenatus commented May 28, 2021

srenatus commented May 28, 2021

srenatus commented May 28, 2021

johanneslarsson commented May 28, 2021

johanneslarsson commented May 28, 2021

srenatus commented May 28, 2021

johanneslarsson commented May 28, 2021

srenatus commented May 28, 2021

johanneslarsson commented May 31, 2021

srenatus commented May 31, 2021

anderseknert commented May 31, 2021

srenatus commented May 31, 2021

srenatus commented May 31, 2021

johanneslarsson commented May 31, 2021

srenatus commented Jun 1, 2021

srenatus commented Oct 21, 2021

linux binary (0.29.0+) depends on glibc, won't run on alpine as-is #3499

linux binary (0.29.0+) depends on glibc, won't run on alpine as-is #3499

Comments

johanneslarsson commented May 28, 2021

Expected Behavior

Actual Behavior

Steps to Reproduce the Problem

Additional Info

srenatus commented May 28, 2021

srenatus commented May 28, 2021

srenatus commented May 28, 2021

johanneslarsson commented May 28, 2021

johanneslarsson commented May 28, 2021

srenatus commented May 28, 2021

johanneslarsson commented May 28, 2021

srenatus commented May 28, 2021

johanneslarsson commented May 31, 2021

srenatus commented May 31, 2021

anderseknert commented May 31, 2021

srenatus commented May 31, 2021

srenatus commented May 31, 2021

johanneslarsson commented May 31, 2021

srenatus commented Jun 1, 2021

srenatus commented Oct 21, 2021