OpenSSL error with Ubuntu 22.04 on Arm32 architecture

[mthalman]

When attempting to run a .NET app that makes an HTTPS connection with the upcoming release of Ubuntu 22.04 on the Arm32 architecture, it results in an OpenSSL error. This issue will block our upcoming support for Ubuntu 22.04 (see dotnet/core#7038).

This only happens in Arm32. Using Arm64 works fine. The easiest way to reproduce this is with a Docker container. But because .NET doesn't work with QEMU, emulation from an x64 machine won't allow you to repro this. You'll need an Arm machine.

I've repro'd this on my Raspberry Pi 4 machine. But it also repros on the .NET Docker team's Jetson build machines which are Arm64.

Repro

1. Get an Arm machine with Docker installed.

2. Save the following contents to a file named Dockerfile in an empty directory:

   FROM mcr.microsoft.com/dotnet/sdk:6.0-focal AS build
   
   WORKDIR dotnetapp
   
   # Create a simple project that makes an HTTPS connection
   RUN dotnet new console --no-restore
   RUN echo 'var response = await new System.Net.Http.HttpClient().GetAsync("https://www.microsoft.com"); response.EnsureSuccessStatusCode(); System.Console.WriteLine("Hello World!");' > Program.cs
   
   RUN dotnet publish -c release -o /app -r linux-arm --self-contained /p:PublishSingleFile=true
   
   
   FROM arm32v7/ubuntu:jammy
   
   # Install .NET dependencies
   RUN apt-get update \
       && apt-get install -y --no-install-recommends \
           ca-certificates \
           \
           libc6 \
           libgcc1 \
           libgssapi-krb5-2 \
           libicu67 \
           libssl1.1 \
           libstdc++6 \
           zlib1g \
       && rm -rf /var/lib/apt/lists/*
   
   WORKDIR /app
   COPY --from=build /app .
   ENTRYPOINT ["./dotnetapp"]

3. Set the current directory to the directory where the Dockerfile is located.

4. $ docker build -t test

5. $ docker run --rm test

Expected Results:

Hello World!

Actual Results:

Unhandled exception. System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
 ---> Interop+OpenSsl+SslException: SSL Handshake failed with OpenSSL error - SSL_ERROR_SSL.
 ---> Interop+Crypto+OpenSslCryptographicException: error:0A0000BF:SSL routines::no protocols available
   --- End of inner exception stack trace ---
   at Interop.OpenSsl.DoSslHandshake(SafeSslHandle context, ReadOnlySpan`1 input, Byte[]& sendBuf, Int32& sendCount)
   at System.Net.Security.SslStreamPal.HandshakeInternal(SafeFreeCredentials credential, SafeDeleteSslContext& context, ReadOnlySpan`1 inputBuffer, Byte[]& outputBuffer, SslAuthenticationOptions sslAuthenticationOptions)
   --- End of inner exception stack trace ---
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request)
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Program.<Main>$(String[] args) in /dotnetapp/Program.cs:line 1
   at Program.<Main>(String[] args)

[ghost]

Tagging subscribers to this area: @dotnet/ncl, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

---

When attempting to run a .NET app that makes an HTTPS connection with the upcoming release of Ubuntu 22.04 on the Arm32 architecture, it results in an OpenSSL error. This issue will block our upcoming support for Ubuntu 22.04 (see dotnet/core#7038).

This only happens in Arm32. Using Arm64 works fine. The easiest way to reproduce this is with a Docker container. But because .NET doesn't work with QEMU, emulation from an x64 machine won't allow you to repro this. You'll need an Arm machine.

I've repro'd this on my Raspberry Pi 4 machine. But it also repros on the .NET Docker team's Jetson build machines which are Arm64.

Repro

1. Get an Arm machine with Docker installed.

2. Save the following contents to a file named Dockerfile in an empty directory:

   FROM mcr.microsoft.com/dotnet/sdk:6.0-focal AS build
   
   WORKDIR dotnetapp
   
   # Create a simple project that makes an HTTPS connection
   RUN dotnet new console --no-restore
   RUN echo 'var response = await new System.Net.Http.HttpClient().GetAsync("https://www.microsoft.com"); response.EnsureSuccessStatusCode(); System.Console.WriteLine("Hello World!");' > Program.cs
   
   RUN dotnet publish -c release -o /app -r linux-arm --self-contained /p:PublishSingleFile=true
   
   
   FROM arm32v7/ubuntu:jammy
   
   # Install .NET dependencies
   RUN apt-get update \
       && apt-get install -y --no-install-recommends \
           ca-certificates \
           \
           libc6 \
           libgcc1 \
           libgssapi-krb5-2 \
           libicu67 \
           libssl1.1 \
           libstdc++6 \
           zlib1g \
       && rm -rf /var/lib/apt/lists/*
   
   WORKDIR /app
   COPY --from=build /app .
   ENTRYPOINT ["./dotnetapp"]

3. Set the current directory to the directory where the Dockerfile is located.

4. $ docker build -t test

5. $ docker run --rm test

Expected Results:

Hello World!

Actual Results:

Unhandled exception. System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
 ---> Interop+OpenSsl+SslException: SSL Handshake failed with OpenSSL error - SSL_ERROR_SSL.
 ---> Interop+Crypto+OpenSslCryptographicException: error:0A0000BF:SSL routines::no protocols available
   --- End of inner exception stack trace ---
   at Interop.OpenSsl.DoSslHandshake(SafeSslHandle context, ReadOnlySpan`1 input, Byte[]& sendBuf, Int32& sendCount)
   at System.Net.Security.SslStreamPal.HandshakeInternal(SafeFreeCredentials credential, SafeDeleteSslContext& context, ReadOnlySpan`1 inputBuffer, Byte[]& outputBuffer, SslAuthenticationOptions sslAuthenticationOptions)
   --- End of inner exception stack trace ---
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request)
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Program.<Main>$(String[] args) in /dotnetapp/Program.cs:line 1
   at Program.<Main>(String[] args)

Author:	mthalman
Assignees:	-
Labels:	area-System.Net.Security, untriaged
Milestone:	-

[mthalman]

@dotnet/ncl, @vcsjones - This issue is time-sensitive in order to have support for the upcoming release of Ubuntu 22.04. I'd like to make sure it's triaged soon.

[vcsjones]

@mthalman I can reproduce it using the steps you provided on Graviton2 ARM64 instance.

I can run openssl s_client -connect www.microsoft.com:443 in the container, I end up negotiating TLSv1.3 TLS_AES_256_GCM_SHA384 with an X25519 key exchange, so I don't think it's a SECLEVEL issue.

Attached is a tcpdump (opens in Wireshark just fine). dump.bin.zip

(Learning how to get a tcpdump from a docker container, to a remote VM, to my host was a fun exercise).

[vcsjones]

Possibly worth pointing out: this docker image has OpenSSL 3.0.1 on it.

[karelz]

@mthalman is there a repro machine (the ARM64 one) we could use? I don't think we have arm32 Raspberry Pis handy for investigation :(

[mthalman]

We have build machines at https://dev.azure.com/dnceng/public/_settings/agentqueues?queueId=125&view=agents. I don't have info on how to SSH into these machines. If you need to SSH, you'll need to contact DDFun.

[rzikm]

It is very weird that the first (outgoing) TLS packet in capture is TLS Alert

so far, getting local build to run inside the docker image has been rather difficult, but I wonder what I find once I get it working.

[ilyas1974]

Information on available Dev systems can be found at https://github.com/dotnet/core-eng/wiki/%5BFR-Operations%5D---Tracking-of-Hardware-not-in-Helix.

[rzikm]

So the problem is that between OpenSSL 1.1 and OpenSSL 3 there has been change in the type of the argument to SSL(_CTX)?_set_options(SSL(_CTX)? *s, options) from uint32_t to uint64_t. Our source code still uses signatures with uint32_t. On 64-bit platforms, this makes no difference, as the arguments are passed the same way (register), but on arm32, the 64-bit int obviously does not fit into a register (where we put our 32-bit version of it) so OpenSSL looks for it elsewhere and reads garbage.

Hypotesis confirmed with following crude fix (which in turn, breaks arm32 with OpenSSL 1.1)

static void SSL_CTX_set_options_fixed(SSL_CTX* ctx, u_int32_t op)
{
    void (*func)(SSL_CTX* ctx, u_int64_t op) = (void(*)(SSL_CTX*, u_int64_t))SSL_CTX_set_options;
    func(ctx, op);
}

[am11]

Probably this can be handled in apibridge?

runtime/src/native/libs/System.Security.Cryptography.Native/apibridge.c

Lines 781 to 786 in 7414af2

	unsigned long local_SSL_CTX_set_options(SSL_CTX* ctx, unsigned long options)
	{
	// SSL_CTX_ctrl is signed long in and signed long out; but SSL_CTX_set_options,
	// which was a macro call to SSL_CTX_ctrl in 1.0, is unsigned/unsigned.
	return (unsigned long)SSL_CTX_ctrl(ctx, SSL_CTRL_OPTIONS, (long)options, NULL);
	}

[wfurt]

that seems like good place. cc: @bartonjs

[bartonjs]

apibridge would make it easy for 1.0.2 to call the 3.0 API, but I'm not sure what we'd do for 1.1.x calling it the 3.0 way.

We'd need to do something like @rzikm suggested, but detecting the runtime environment to know how to target the call.

static uint64_t SSL_CTX_set_options_dynamic(SSL_CTX* ctx, uint32_t options)
{
#ifdef WHATEVER_THE_PORTABLE_DEFINE_IS
    if (some 3.0 sentinel function)
    {
        uint64_t (*func)(SSL_CTX* ctx, u_int64_t op) = (uint64_t(*)(SSL_CTX*, u_int64_t))SSL_CTX_set_options;
        return func(ctx, options);
    }
    else
    {
        uint32_t (*func)(SSL_CTX* ctx, u_int32_t op) = (uint32_t(*)(SSL_CTX*, u_int32_t))SSL_CTX_set_options;
        return func(ctx, options);
    }
#else
    return SSL_CTX_set_options(ctx, options);
#endif
}

[rzikm]

@mthalman Do you think this is worth backporting to 6.0 so that we can support Ubuntu 22.04 on arm32 earlier than .NET 7 release?

[mthalman]

@mthalman Do you think this is worth backporting to 6.0 so that we can support Ubuntu 22.04 on arm32 earlier than .NET 7 release?

It's a requirement. And not only for 6.0 but for 5.0 and 3.1 as well. Each of these versions is specified in dotnet/core#7038.

[rzikm]

Opening for backporting

[am11]

Release date of Ubuntu 22.04 is April 21, 2022. .NET 5 will reach End of support on May 8, 2022. Do we need backport to .NET 5?

[mthalman]

Release date of Ubuntu 22.04 is April 21, 2022. .NET 5 will reach End of support on May 8, 2022. Do we need backport to .NET 5?

Probably not. But it'd be worthwhile to post to dotnet/core#7038 indicating what the intention is.

[am11]

I think if there is not going to be another release of .NET 5 in those 17 days, we can skip the backport. Just wanted to point out the short window that we have there.

[bartonjs]

3.1 is also not relevant, as we don't support OpenSSL 3 with that release.

[wfurt]

We also don't need to fix every bug IMHO. This decision should come from ship room.

[mthalman]

We also don't need to fix every bug IMHO. This decision should come from ship room.

True but this bug effectively makes .NET unusable on this platform for all but the most basic applications.

[skytech-cyber]

amd64 has same issue. "The SSL connection could not be established"

.NET SDK (reflecting any global.json):
Version: 6.0.201
Commit: ef40e6aa06

Runtime Environment:
OS Name: ubuntu
OS Version: 22.04
OS Platform: Linux
RID: linux-x64
Base Path: /usr/share/dotnet/sdk/6.0.201/

[rzikm]

@skytech-cyber Is the issue on the same callstack and with the same message? I was using OpenSSL 3 locally several times (although on different distro) and have not run into this issue.

[rzikm]

The bug has been fixed in main and in 6.0.5. Backporting to other versions is not needed: .NET 5 is near EOL and .NET 3.1 does not support OpenSSL 3.0.

[leopignataro]

Dear @bartonjs and @rzikm, you both mentioned .NET Core 3.1 does not support OpenSSL 3.0. What does that mean in terms of the future support (or lack of) for .NET Core Runtime 3.1 on Ubuntu 22.04?

Thanks!

[mthalman]

Dear @bartonjs and @rzikm, you both mentioned .NET Core 3.1 does not support OpenSSL 3.0. What does that mean in terms of the future support (or lack of) for .NET Core Runtime 3.1 on Ubuntu 22.04?

See dotnet/core#7038 (comment)