openssl.so fails to find EC_GROUP_new_curve_GF2m in libcrypto.so.1.1

[natedogith1]

I created an image of a machine modified from GCP image sles-15-sp2-sap-v20220718-x86-64, and google-fluentd is having library issues. I've managed to reproduce it once with just the base image + installing google-fluentd, but that instance has since spontaneously fixed itself. The original image is still having problems. As a workaround, I've downgraded to google-fluentd=1.9.8-1.sles15

OS info:

> cat /etc/os-release
NAME="SLES"
VERSION="15-SP2"
VERSION_ID="15.2"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP2"
ID="sles"
ID_LIKE="suse"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:15:sp2"
VARIANT_ID="sles-sap"

Here's the error message from journalctl:

Aug 29 13:36:46 n-sles15-03 google-fluentd[13586]: Starting google-fluentd 1.9.9: /opt/google-fluentd/embedded/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require': /opt/google-fluentd/embedded/lib/ruby/2.6.0/x86_64-linux/openssl.so: symbol EC_GROUP_new_curve_GF2m, version OPENSSL_1_1_0 not defined in file libcrypto.so.1.1 with link time reference - /opt/google-fluentd/embedded/lib/ruby/2.6.0/x86_64-linux/openssl.so (LoadError)
Aug 29 13:36:46 n-sles15-03 google-fluentd[13586]:         from /opt/google-fluentd/embedded/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
Aug 29 13:36:46 n-sles15-03 google-fluentd[13586]:         from /opt/google-fluentd/embedded/lib/ruby/2.6.0/openssl.rb:13:in `<top (required)>'
Aug 29 13:36:46 n-sles15-03 google-fluentd[13586]:         from /opt/google-fluentd/embedded/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
Aug 29 13:36:46 n-sles15-03 google-fluentd[13586]:         from /opt/google-fluentd/embedded/lib/ruby/gems/2.6.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/server.rb:24:in `<top (required)>'
Aug 29 13:36:46 n-sles15-03 google-fluentd[13586]:         from /opt/google-fluentd/embedded/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
Aug 29 13:36:46 n-sles15-03 google-fluentd[13586]:         from /opt/google-fluentd/embedded/lib/ruby/gems/2.6.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/http_server.rb:26:in `<top (required)>'
Aug 29 13:36:46 n-sles15-03 google-fluentd[13586]:         from /opt/google-fluentd/embedded/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
Aug 29 13:36:46 n-sles15-03 google-fluentd[13586]:         from /opt/google-fluentd/embedded/lib/ruby/gems/2.6.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper.rb:25:in `<top (required)>'
Aug 29 13:36:46 n-sles15-03 google-fluentd[13586]:         from /opt/google-fluentd/embedded/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
Aug 29 13:36:46 n-sles15-03 google-fluentd[13586]:         from /opt/google-fluentd/embedded/lib/ruby/gems/2.6.0/gems/fluentd-1.13.3/lib/fluent/plugin/filter.rb:22:in `<top (required)>'
Aug 29 13:36:46 n-sles15-03 google-fluentd[13586]:         from /opt/google-fluentd/embedded/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
Aug 29 13:36:46 n-sles15-03 google-fluentd[13586]:         from /opt/google-fluentd/embedded/lib/ruby/gems/2.6.0/gems/fluentd-1.13.3/lib/fluent/compat/filter.rb:18:in `<top (required)>'
Aug 29 13:36:46 n-sles15-03 google-fluentd[13586]:         from /opt/google-fluentd/embedded/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
Aug 29 13:36:46 n-sles15-03 google-fluentd[13586]:         from /opt/google-fluentd/embedded/lib/ruby/gems/2.6.0/gems/fluentd-1.13.3/lib/fluent/filter.rb:17:in `<top (required)>'
Aug 29 13:36:46 n-sles15-03 google-fluentd[13586]:         from /opt/google-fluentd/embedded/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
Aug 29 13:36:46 n-sles15-03 google-fluentd[13586]:         from /opt/google-fluentd/embedded/lib/ruby/gems/2.6.0/gems/fluentd-1.13.3/lib/fluent/event_router.rb:19:in `<top (required)>'
Aug 29 13:36:46 n-sles15-03 google-fluentd[13586]:         from /opt/google-fluentd/embedded/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
Aug 29 13:36:46 n-sles15-03 google-fluentd[13586]:         from /opt/google-fluentd/embedded/lib/ruby/gems/2.6.0/gems/fluentd-1.13.3/lib/fluent/engine.rb:19:in `<top (required)>'
Aug 29 13:36:46 n-sles15-03 google-fluentd[13586]:         from /opt/google-fluentd/embedded/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
Aug 29 13:36:46 n-sles15-03 google-fluentd[13586]:         from /opt/google-fluentd/embedded/lib/ruby/gems/2.6.0/gems/fluentd-1.13.3/lib/fluent/supervisor.rb:23:in `<top (required)>'
Aug 29 13:36:46 n-sles15-03 google-fluentd[13586]:         from /opt/google-fluentd/embedded/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
Aug 29 13:36:46 n-sles15-03 google-fluentd[13586]:         from /opt/google-fluentd/embedded/lib/ruby/gems/2.6.0/gems/fluentd-1.13.3/lib/fluent/command/fluentd.rb:19:in `<top (required)>'
Aug 29 13:36:46 n-sles15-03 google-fluentd[13586]:         from /opt/google-fluentd/embedded/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
Aug 29 13:36:46 n-sles15-03 google-fluentd[13586]:         from /opt/google-fluentd/embedded/lib/ruby/gems/2.6.0/gems/fluentd-1.13.3/bin/fluentd:15:in `<top (required)>'
Aug 29 13:36:46 n-sles15-03 google-fluentd[13586]:         from /opt/google-fluentd/embedded/bin/fluentd:23:in `load'
Aug 29 13:36:46 n-sles15-03 google-fluentd[13586]:         from /opt/google-fluentd/embedded/bin/fluentd:23:in `<top (required)>'
Aug 29 13:36:46 n-sles15-03 google-fluentd[13586]:         from /usr/sbin/google-fluentd:7:in `load'
Aug 29 13:36:46 n-sles15-03 google-fluentd[13586]:         from /usr/sbin/google-fluentd:7:in `<main>'

I set the environment variables "LD_DEBUG=all" and "LD_DEBUG_OUTPUT=/tmp/fluentd_ld" to see what library was being loaded, these are the lines for the failure:

     23929:     symbol=EC_GROUP_new_curve_GF2m;  lookup in file=/opt/google-fluentd/embedded/bin/ruby [0]
     23929:     symbol=EC_GROUP_new_curve_GF2m;  lookup in file=/opt/google-fluentd/embedded/lib/libruby.so.2.6 [0]
     23929:     symbol=EC_GROUP_new_curve_GF2m;  lookup in file=/opt/google-fluentd/embedded/lib/libz.so.1 [0]
     23929:     symbol=EC_GROUP_new_curve_GF2m;  lookup in file=/lib64/libpthread.so.0 [0]
     23929:     symbol=EC_GROUP_new_curve_GF2m;  lookup in file=/lib64/librt.so.1 [0]
     23929:     symbol=EC_GROUP_new_curve_GF2m;  lookup in file=/lib64/libdl.so.2 [0]
     23929:     symbol=EC_GROUP_new_curve_GF2m;  lookup in file=/lib64/libcrypt.so.1 [0]
     23929:     symbol=EC_GROUP_new_curve_GF2m;  lookup in file=/lib64/libm.so.6 [0]
     23929:     symbol=EC_GROUP_new_curve_GF2m;  lookup in file=/lib64/libc.so.6 [0]
     23929:     symbol=EC_GROUP_new_curve_GF2m;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
     23929:     symbol=EC_GROUP_new_curve_GF2m;  lookup in file=/opt/google-fluentd/embedded/lib/ruby/2.6.0/x86_64-linux/enc/encdb.so [0]
     23929:     symbol=EC_GROUP_new_curve_GF2m;  lookup in file=/opt/google-fluentd/embedded/lib/ruby/2.6.0/x86_64-linux/enc/trans/transdb.so [0]
     23929:     symbol=EC_GROUP_new_curve_GF2m;  lookup in file=/opt/google-fluentd/embedded/lib/ruby/2.6.0/x86_64-linux/stringio.so [0]
     23929:     symbol=EC_GROUP_new_curve_GF2m;  lookup in file=/opt/google-fluentd/embedded/lib/ruby/2.6.0/x86_64-linux/etc.so [0]
     23929:     symbol=EC_GROUP_new_curve_GF2m;  lookup in file=/opt/google-fluentd/embedded/lib/ruby/gems/2.6.0/gems/json-2.2.0/lib/json/ext/parser.so [0]
     23929:     symbol=EC_GROUP_new_curve_GF2m;  lookup in file=/opt/google-fluentd/embedded/lib/ruby/gems/2.6.0/gems/json-2.2.0/lib/json/ext/generator.so [0]
     23929:     symbol=EC_GROUP_new_curve_GF2m;  lookup in file=/opt/google-fluentd/embedded/lib/ruby/gems/2.6.0/gems/yajl-ruby-1.3.1/lib/yajl/yajl.so [0]
     23929:     symbol=EC_GROUP_new_curve_GF2m;  lookup in file=/opt/google-fluentd/embedded/lib/ruby/2.6.0/x86_64-linux/socket.so [0]
     23929:     symbol=EC_GROUP_new_curve_GF2m;  lookup in file=/opt/google-fluentd/embedded/lib/ruby/2.6.0/x86_64-linux/io/wait.so [0]
     23929:     symbol=EC_GROUP_new_curve_GF2m;  lookup in file=/opt/google-fluentd/embedded/lib/ruby/2.6.0/x86_64-linux/ripper.so [0]
     23929:     symbol=EC_GROUP_new_curve_GF2m;  lookup in file=/opt/google-fluentd/embedded/lib/ruby/2.6.0/x86_64-linux/strscan.so [0]
     23929:     symbol=EC_GROUP_new_curve_GF2m;  lookup in file=/opt/google-fluentd/embedded/lib/ruby/gems/2.6.0/gems/cool.io-1.6.0/lib/iobuffer_ext.so [0]
     23929:     symbol=EC_GROUP_new_curve_GF2m;  lookup in file=/opt/google-fluentd/embedded/lib/ruby/gems/2.6.0/gems/cool.io-1.6.0/lib/cool.io_ext.so [0]
     23929:     symbol=EC_GROUP_new_curve_GF2m;  lookup in file=/opt/google-fluentd/embedded/lib/ruby/gems/2.6.0/gems/msgpack-1.5.4/lib/msgpack/msgpack.so [0]
     23929:     symbol=EC_GROUP_new_curve_GF2m;  lookup in file=/opt/google-fluentd/embedded/lib/ruby/2.6.0/x86_64-linux/date_core.so [0]
     23929:     symbol=EC_GROUP_new_curve_GF2m;  lookup in file=/opt/google-fluentd/embedded/lib/ruby/gems/2.6.0/gems/strptime-0.2.5/lib/strptime/strptime.so [0]
     23929:     symbol=EC_GROUP_new_curve_GF2m;  lookup in file=/opt/google-fluentd/embedded/lib/ruby/2.6.0/x86_64-linux/bigdecimal.so [0]
     23929:     symbol=EC_GROUP_new_curve_GF2m;  lookup in file=/opt/google-fluentd/embedded/lib/ruby/gems/2.6.0/gems/oj-3.3.10/lib/oj/oj.so [0]
     23929:     symbol=EC_GROUP_new_curve_GF2m;  lookup in file=/opt/google-fluentd/embedded/lib/ruby/2.6.0/x86_64-linux/zlib.so [0]
     23929:     symbol=EC_GROUP_new_curve_GF2m;  lookup in file=/opt/google-fluentd/embedded/lib/ruby/2.6.0/x86_64-linux/io/nonblock.so [0]
     23929:     symbol=EC_GROUP_new_curve_GF2m;  lookup in file=/opt/google-fluentd/embedded/lib/ruby/2.6.0/x86_64-linux/cgi/escape.so [0]
     23929:     symbol=EC_GROUP_new_curve_GF2m;  lookup in file=/opt/google-fluentd/embedded/lib/ruby/2.6.0/x86_64-linux/digest/md5.so [0]
     23929:     symbol=EC_GROUP_new_curve_GF2m;  lookup in file=/usr/lib64/libssl.so.1.1 [0]
     23929:     symbol=EC_GROUP_new_curve_GF2m;  lookup in file=/usr/lib64/libcrypto.so.1.1 [0]
     23929:     /opt/google-fluentd/embedded/lib/ruby/2.6.0/x86_64-linux/openssl.so: error: relocation error: symbol EC_GROUP_new_curve_GF2m, version OPENSSL_1_1_0 not defined in file libcrypto.so.1.1 with link time reference (fatal)

Looking through the file for why libcrypto.so.1.1 was loaded, I found these lines:

     23929:     symbol=initgroups;  lookup in file=/sbin/start_daemon [0]
     23929:     symbol=initgroups;  lookup in file=/usr/lib64/libblogger.so.2 [0]
     23929:     symbol=initgroups;  lookup in file=/lib64/libpthread.so.0 [0]
     23929:     symbol=initgroups;  lookup in file=/lib64/libc.so.6 [0]
     23929:     binding file /sbin/start_daemon [0] to /lib64/libc.so.6 [0]: normal symbol `initgroups' [GLIBC>
     23929:     symbol=_nss_files_initgroups_dyn;  lookup in file=/lib64/libnss_files.so.2 [0]
     23929:     binding file /lib64/libnss_files.so.2 [0] to /lib64/libnss_files.so.2 [0]: normal symbol `_nss>
     23929:
     23929:     file=libnss_cache_oslogin.so.2 [0];  dynamically loaded by /lib64/libc.so.6 [0]
     23929:     find library=libnss_cache_oslogin.so.2 [0]; searching
     23929:      search cache=/etc/ld.so.cache
     23929:       trying file=/usr/lib64/libnss_cache_oslogin.so.2
...
     23929:     file=libcurl.so.4 [0];  needed by /usr/lib64/libnss_cache_oslogin.so.2 [0]
     23929:     find library=libcurl.so.4 [0]; searching
     23929:      search cache=/etc/ld.so.cache
     23929:       trying file=/usr/lib64/libcurl.so.4
...
     23929:     file=libcrypto.so.1.1 [0];  needed by /usr/lib64/libcurl.so.4 [0]
     23929:     find library=libcrypto.so.1.1 [0]; searching
     23929:      search cache=/etc/ld.so.cache
     23929:       trying file=/usr/lib64/libcrypto.so.1.1

As far as I can tell, it seems like google-fluentd is using the system libcrypto.so.1.1 instead of the one it ships with, and the one it ships with has extra symbols:

> readelf --wide -s /opt/google-fluentd/embedded/lib/libcrypto.so.1.1 | grep EC_GROUP_new_curve
  3718: 000000000010a6a0    92 FUNC    GLOBAL DEFAULT   14 EC_GROUP_new_curve_GFp@@OPENSSL_1_1_0
  3977: 000000000010a700    92 FUNC    GLOBAL DEFAULT   14 EC_GROUP_new_curve_GF2m@@OPENSSL_1_1_0
  7024: 000000000010a6a0    92 FUNC    GLOBAL DEFAULT   14 EC_GROUP_new_curve_GFp
  7075: 000000000010a700    92 FUNC    GLOBAL DEFAULT   14 EC_GROUP_new_curve_GF2m
> readelf --wide -s /usr/lib64/libcrypto.so.1.1 | grep EC_GROUP_new_curve
  3743: 000000000010f100    92 FUNC    GLOBAL DEFAULT   14 EC_GROUP_new_curve_GFp@@OPENSSL_1_1_0

I think this wasn't a problem in google-fluentd=1.9.8-1.sles15 because the google-fluentd's libcrypto was libcrypto.so.1.0.0, which the system doesn't have.

Checking the LD_DEBUG=all output for google-fluentd=1.9.9-1.sles15 on a system that isn't having problems, it seems like libnss isn't being loaded. I'm not sure why, I ensured /etc/nsswitch.conf was the same on the passing and failing systems, and they continued to pass/fail.