[BUG] openssl core dumps under Ubuntu 22.04 when specifying the network service name instead of the port

[sanderjo]

Wrong input, so PEBKAC, but core dumped, so worth reporting?

Before you open an issue please check which version you are running and whether it is the latest in stable / dev branch
I am running version

commit 1a9a486474cacd377c444b6fc08a9e754ef95480
testssl.sh       3.2rc3 from https://testssl.sh/dev/

Before you open an issue please whether this is a known problem by searching the issues

Probably not related to #1934

Command line / docker command to reproduce

./testssl.sh news.newshosting.com:nntps

No problem when I use:

./testssl.sh news.newshosting.com:563

Expected behavior
Accept the :nntps, or reject it without core dump.

Your system (please complete the following information):

• OS: Ubuntu 22.04.3 LTS
• Platform: Linux 6.2.0-37-generic x86_64
• OpenSSL: ./bin/openssl.Linux.x86_64

Additional context

sander@zwart2204:~/git/poging2_testssl$ ./testssl.sh news.newshosting.com:nntps

###########################################################
    testssl.sh       3.2rc3 from https://testssl.sh/dev/
    (1a9a486 2023-11-10 19:41:08)

      This program is free software. Distribution and
             modification under GPLv2 permitted.
      USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!

       Please file bugs @ https://testssl.sh/bugs/

###########################################################

 Using "OpenSSL 1.0.2-bad (1.0.2k-dev)" [~179 ciphers]
 on zwart2204:./bin/openssl.Linux.x86_64
 (built: "Sep  1 14:03:44 2022", platform: "linux-x86_64")


Testing all IPv4 addresses (port nntps): 85.12.62.225 185.90.196.65 85.12.62.251 185.90.196.97 185.90.196.129
--------------------------------------------------------------------------------------------------------------
 Start 2023-12-01 21:32:04                -->> 85.12.62.225:nntps (news.newshosting.com) <<--

 Further IP addresses:   185.90.196.97 185.90.196.65 185.90.196.129 85.12.62.251 
 rDNS (85.12.62.225):    --
./testssl.sh: line 21817: 131463 Segmentation fault      (core dumped) $OPENSSL s_client $(s_client_options "$proto $BUGS -connect "$NODEIP:$PORT" -msg $PROXY $SNI") < /dev/null > $TMPFILE 2>> $ERRFILE
./testssl.sh: line 21817: 131467 Segmentation fault      (core dumped) $OPENSSL s_client $(s_client_options "$proto $BUGS -connect "$NODEIP:$PORT" -msg $PROXY $SNI") < /dev/null > $TMPFILE 2>> $ERRFILE
./testssl.sh: line 21817: 131471 Segmentation fault      (core dumped) $OPENSSL s_client $(s_client_options "$proto $BUGS -connect "$NODEIP:$PORT" -msg $PROXY $SNI") < /dev/null > $TMPFILE 2>> $ERRFILE
./testssl.sh: line 21817: 131475 Segmentation fault      (core dumped) $OPENSSL s_client $(s_client_options "$proto $BUGS -connect "$NODEIP:$PORT" -msg $PROXY $SNI") < /dev/null > $TMPFILE 2>> $ERRFILE
./testssl.sh: line 21817: 131479 Segmentation fault      (core dumped) $OPENSSL s_client $(s_client_options "$proto $BUGS -connect "$NODEIP:$PORT" -msg $PROXY $SNI") < /dev/null > $TMPFILE 2>> $ERRFILE
./testssl.sh: line 21817: 131483 Segmentation fault      (core dumped) $OPENSSL s_client $(s_client_options "$proto $BUGS -connect "$NODEIP:$PORT" -msg $PROXY $SNI") < /dev/null > $TMPFILE 2>> $ERRFILE
 Your OpenSSL cannot connect to 85.12.62.225:nntps
 The results might look ok but they could be nonsense. Really proceed ? ("yes" to continue) --> n

sander@zwart2204:~/git/poging2_testssl$

[sanderjo]

I think the PORT is determined here:

PORT=$(sed 's/^.*\://' <<< "$NODE") && NODE=$(sed 's/\:.*$//' <<< "$NODE")

Let's check

$ sed 's/^.*\://' <<< "news.bla.com:563"
563
$ sed 's/^.*\://' <<< "news.bla.com:NNTPS"
NNTPS

Yes. So ... some checking if the PORT is a number? Or: remove non-numeric:

$ echo "news.bla.com:563" | sed 's/^.*\://' |  sed 's/[^0-9]*//g'
563
$ echo "news.bla.com:NNTP" | sed 's/^.*\://' |  sed 's/[^0-9]*//g'

$

In one sed command:

$ echo "news.bla.com:563" | sed 's/^.*\:[^0-9]*//' 
563
$ echo "news.bla.com:NNTPS" | sed 's/^.*\:[^0-9]*//' 

$ 

[sanderjo]

OK ... an escape forward: accept and work with "nntps" and other descriptions:

A few lines hacked together to get it working ... from the if :

          # determine v4 port, supposed it was supplied additionally
          grep -q ':' <<< "$NODE" && \
               #PORT=$(sed 's/^.*\:[^0-9]*//'  <<< "$NODE") && NODE=$(sed 's/\:.*$//' <<< "$NODE")
               PORT=$(sed 's/^.*\://'  <<< "$NODE") && NODE=$(sed 's/\:.*$//' <<< "$NODE")
               #PORT=$(sed 's/[^0-9]*//g' <<< "$PORT")
               if [ -n "$PORT" ] && [ "$PORT" -eq "$PORT" ] 2>/dev/null; then
                 echo "NUMBER, thus good" 
               else
                 echo "not a number"
                 PORT=`getent services $PORT | awk '{ print $2 }' | awk -F/ '{ print $1 }'`
                 echo $PORT
               fi

[drwetter]

That's a strange behavior from the shipped openssl binary. It works though when using an external new binary which is the default on Mac Silicon.

The code looks different than I would write it today. I am terribly busy with other things atm, give me a ~week.

[sanderjo]

I'll send a PR with cleaner code than my hacking above.

[sanderjo]

Oh, wait: openssl does accept ":nntps". So why does it not work within testssl? Maybe I should find that out first?

$ echo "QUIT\r\n\r\n" | openssl s_client -connect news.newshosting.com:nntps

[drwetter]

As indicated, it depends on the openssl version.

[sanderjo]

[drwetter]

I looked into the sources of openssl and I couldn´t find translation or mapping of service --> port

We were chasing a ghost, kind of:

Same git commit as above. Also echo "QUIT\r\n\r\n" | OPENSSL_CONF='' ./bin/openssl.Linux.x86_64 s_client -connect news.newshosting.com:nntps worked.

That's for Debian 11 and Opensuse Leap 15.5.

Looks to me the OS tries to resolve that but not in your case (strace output:)

close(3)                                = 0
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
connect(3, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = 0
sendto(3, "\2\0\0\0\22\0\0\0\t\0\0\0services\0", 21, MSG_NOSIGNAL, NULL, 0) = 21
poll([{fd=3, events=POLLIN|POLLERR|POLLHUP}], 1, 5000) = 1 ([{fd=3, revents=POLLIN}])
recvmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="services\0", iov_len=9}, {iov_base="\310O\3\0\0\0\0\0", iov_len=8}], m
sg_iovlen=2, msg_control=[{cmsg_len=20, cmsg_level=SOL_SOCKET, cmsg_type=SCM_RIGHTS, cmsg_data=[4]}], msg_controllen=20, msg_flags=M
SG_CMSG_CLOEXEC}, MSG_CMSG_CLOEXEC) = 17
mmap(NULL, 217032, PROT_READ, MAP_SHARED, 4, 0) = 0x7fd6eb61a000
close(4)                                = 0
close(3)                                = 0
open("/etc/gai.conf", O_RDONLY|O_CLOEXEC) = -1 ENOENT (Datei oder Verzeichnis nicht gefunden)
socket(AF_INET, SOCK_DGRAM, IPPROTO_IP) = 3
connect(3, {sa_family=AF_INET, sin_port=htons(563), sin_addr=inet_addr("85.12.62.251")}, 16) = 0
getsockname(3, {sa_family=AF_INET, sin_port=htons(47804), sin_addr=inet_addr("192.168.33.29")}, [28 => 16]) = 0
connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0
connect(3, {sa_family=AF_INET, sin_port=htons(563), sin_addr=inet_addr("85.12.62.225")}, 16) = 0
getsockname(3, {sa_family=AF_INET, sin_port=htons(52637), sin_addr=inet_addr("192.168.33.29")}, [28 => 16]) = 0
connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0
connect(3, {sa_family=AF_INET, sin_port=htons(563), sin_addr=inet_addr("185.90.196.65")}, 16) = 0
getsockname(3, {sa_family=AF_INET, sin_port=htons(33275), sin_addr=inet_addr("192.168.33.29")}, [28 => 16]) = 0
connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0
connect(3, {sa_family=AF_INET, sin_port=htons(563), sin_addr=inet_addr("185.90.196.97")}, 16) = 0
getsockname(3, {sa_family=AF_INET, sin_port=htons(50612), sin_addr=inet_addr("192.168.33.29")}, [28 => 16]) = 0
connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0
connect(3, {sa_family=AF_INET, sin_port=htons(563), sin_addr=inet_addr("185.90.196.129")}, 16) = 0
getsockname(3, {sa_family=AF_INET, sin_port=htons(60977), sin_addr=inet_addr("192.168.33.29")}, [28 => 16]) = 0

Sooo the best would be to find out why that fails on your machine and then look for the correct fix.

[sanderjo]

Looks like a ghost / PEBKAC indeed: on another Ubuntu machine (Ubuntu 20.04, not 22.04, and another OpenSSL / another testssl version?) it does work.

Sorry. I should have tested different machine before.

[drwetter]

It would be great to know the cause. I assume testssl.sh -p -t smtp somehost:smtp doesn't work either?

[sanderjo]

It would be great to know the cause. I assume testssl.sh -p -t smtp somehost:smtp doesn't work either?

Correct: doesn't work either ... coredump too

sander@zwart2204:~/git/testssl.sh$ ./testssl.sh -p -t smtp  gmail-smtp-in.l.google.com:smtp

###########################################################
    testssl.sh       3.2rc2 from https://testssl.sh/dev/
    (7670275 2022-12-27 22:06:12)

      This program is free software. Distribution and
             modification under GPLv2 permitted.
      USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!

       Please file bugs @ https://testssl.sh/bugs/

###########################################################

 Using "OpenSSL 1.0.2-bad (1.0.2k-dev)" [~179 ciphers]
 on zwart2204:./bin/openssl.Linux.x86_64
 (built: "Sep  1 14:03:44 2022", platform: "linux-x86_64")


 Start 2023-12-10 20:44:33        -->> 142.250.27.27:smtp (gmail-smtp-in.l.google.com) <<--

 Further IP addresses:   2a00:1450:4025:401::1a 
 rDNS (142.250.27.27):   ra-in-f27.1e100.net.
./testssl.sh: line 21619: 152740 Segmentation fault      (core dumped) $OPENSSL s_client $(s_client_options "$STARTTLS_OPTIMAL_PROTO $BUGS -connect "$NODEIP:$PORT" $PROXY -msg $STARTTLS $SNI") < /dev/null > $TMPFILE 2>> $ERRFILE
./testssl.sh: line 21619: 152744 Segmentation fault      (core dumped) $OPENSSL s_client $(s_client_options "$STARTTLS_OPTIMAL_PROTO $BUGS -connect "$NODEIP:$PORT" $PROXY -msg $STARTTLS $SNI") < /dev/null > $TMPFILE 2>> $ERRFILE
./testssl.sh: line 21619: 152748 Segmentation fault      (core dumped) $OPENSSL s_client $(s_client_options "$STARTTLS_OPTIMAL_PROTO $BUGS -connect "$NODEIP:$PORT" $PROXY -msg $STARTTLS $SNI") < /dev/null > $TMPFILE 2>> $ERRFILE
./testssl.sh: line 21619: 152752 Segmentation fault      (core dumped) $OPENSSL s_client $(s_client_options "$STARTTLS_OPTIMAL_PROTO $BUGS -connect "$NODEIP:$PORT" $PROXY -msg $STARTTLS $SNI") < /dev/null > $TMPFILE 2>> $ERRFILE
./testssl.sh: line 21619: 152756 Segmentation fault      (core dumped) $OPENSSL s_client $(s_client_options "$STARTTLS_OPTIMAL_PROTO $BUGS -connect "$NODEIP:$PORT" $PROXY -msg $STARTTLS $SNI") < /dev/null > $TMPFILE 2>> $ERRFILE
 Your OpenSSL cannot connect to 142.250.27.27:smtp
 The results might look ok but they could be nonsense. Really proceed ? ("yes" to continue) --> n

sander@zwart2204:~/git/testssl.sh$ 

[drwetter]

Was able to reproduce this with a ootb Ubuntu Jammy Jellyfish:

It core dumps when/after reading /etc/services 🤷🏽. Doesn't happen when the port is specified instead. Seems to be more a Ubuntu problem.

As it is not an edge case we need a good workaround. Emphasis is on the work 'good'. Need to sleep on it.

Recommended workaround is using --openssl=/usr/bin/openssl or to specify the port instead.

[sanderjo]

Recommended workaround is using --openssl=/usr/bin/openssl

./testssl.sh --openssl=/usr/bin/openssl news.newshosting.com:nntps
Beautiful! Works on my Ubuntu 22.04.3 LTS aka Jammy.

[mzpqnxow]

Recommended workaround is using --openssl=/usr/bin/openssl

`./testssl.sh --openssl=/usr/bin/openssl news.newshosting.com:nntps

`

Beautiful! Works on my Ubuntu 22.04.3 LTS aka Jammy.

Is the problematic openssl exe statically linked with glibc? If so, it could be symptomatic of known issues with glibc and its dependence upon libnss, which can not be statically linked. See this to get you started if you're not familiar with it. It seems plausible since getprotoent() depends upon NSS, to parse and adhere to nsswitch.conf

If that is in fact the root of the issue (I'm not necessarily saying it is, but I might as well complete the thought at this point) then the "good" solution in my opinion would be to replace that statically linked exe with one built using a toolchain that is friendly towards static linking (and cross-compiling, for that matter)

musl toolchains would be the obvious choice here

Again, though- I'm making two assumptions here- that the toolchain used to build that exe used glibc, and that the NSS issue could in fact manifest as memory corruption/dumping core

I didn't look if there was a "backtrace" and "info registers" from gdb posted, but it would probably be helpful if someone could take a look. Depending in your system you may be able to use "ulimit -c unlimited" to allow for a full core file

[drwetter]

Hey @mzpqnxow ,

this is a different problem, see my strace above, Did you see somewhere here getprotoent()?

musl libc has performance issues and has also problems with NSS, at least in a container.

Other then that: The toolchain I was using when compiling openssl the last time is very old. I did that deliberately because I wanted folks using old distros to use those binaries . That might contribute to the problem. In any case I'll try next time to upgrade the compile environment. One of the links you referred to was pointing to https://stackoverflow.com/questions/2725255/create-statically-linked-binary-that-uses-getaddrinfo . That might be helpful when recompiling.

[mzpqnxow]

Hey @mzpqnxow ,

this is a different problem, see my strace above, Did you see somewhere here getprotoent()?

Assuming you mean do I see any sequences of system calls that may be in line with what glibc getprotoent() does - no, I don't. Though I imagine if it was an NSS issue, it might be triggered by any number of libc functions that reference nsswitch.conf (getservbyname, getaddrinfo, etc.)

Regardless, if you're fairly certain it's not an NSS issue, I believe you

musl libc has performance issues
and has also problems with NSS, at least in a container

Pardon me as I hijack the issue for a moment 😀

I'm very surprised to hear of performance issues. I'm even more surprised to hear of issues manifesting in containers, especially given one of the most common bases for Docker containers (for services/applications at least) is Alpine with musl. If you have details on either I would be happy to raise them on the musl development list, I've had great experiences there

EDIT: I see #2354 has some information on this, I may take a look out of curiosity

I wonder if maybe you haven't given musl a fair shot recently? I'm being so insistent because I use it specifically for statically linked executables very regularly on all sorts of kernels (2.6, 3.x, 4.x, 5.x) and architectures (x64, x86, arm, mips) and haven't had any issues in as long as I can remember

Other then that: The toolchain I was using when compiling openssl the last time is very old. I did that deliberately because I wanted folks using old distros to use those binaries . That might contribute to the problem. In any case I'll try next time to upgrade the compile environment. One of the links you referred to was pointing to https://stackoverflow.com/questions/2725255/create-statically-linked-binary-that-uses-getaddrinfo . That might be helpful when recompiling.

Happy it may be helpful

My last words on this, if you didn't see them coming... I can't recommend strongly enough that you reconsider using musl for this. Aside from the fact that glibc hasn't been intended for/supported with static linking for a long time (you know that already) musl just makes cross-compiling so much easier

I will cease and desist now 😊

As always, thanks for the work you do on this project