Native Sockets TLS Client/Server support for linux #2939

olme04 · 2022-04-09T08:57:30Z

Based on: #2929

Subsystem
Network, CIO

Motivation
partially fixes https://youtrack.jetbrains.com/issue/KTOR-1181 (no darwin support yet)

Solution

Use cinterop with openssl on linux to handle TLS
Support both client/server authentication and verification
Added support for multiplatform configuration of TLS via PKCS12 certificates
Update JVM implementation of TLS to use SSLEngine with new API (server TLS, authentication for client and so on), but use old implementation if new API isn't used.

Note:

I have no ability to make it working on darwin, as I have no mac, so implementation is fully in linux sourceset
on Darwin, we can use SecureTransport api, which is out of the box coming in kotlin-native via Foundation. openssl isn't needed, which will simplify adopting it. SecureTransport has similar API to both openssl and JVM SSLEngine

TODO:

support certificate generation via openssl - for now, I created 2 certificates with 1000 days validity for tests via tls-certificates module
Implementations of both SSLEngine and via openssl is behaving differently when working with certificates - f.e. JVM don't check certificate validity and server name validity.
May be we can introduce such an API with ExperimentalTLSApi annotation, because of possible bugs, and get more feedback on cases, where it can have unexpected behavior?
More tests for TLS specifics is needed: different certificates (expired, wrong data), passwords validation, TLS session graceful closing
For now, cinterop on linux points to brew location, because, at least on my machine, if to use same paths as in curl, it will fail too link binary because of GLIBC issues (related: https://youtrack.jetbrains.com/issue/KT-43501, https://youtrack.jetbrains.com/issue/KT-47061#focus=Comments-27-4947040.0-0)
Need to restore binary compatibility on JVM - need to add some hidden declarations, from source perspective, changes are compatible

TODO: test failed - HttpServerCommonTestSuite.testProxyHeaders

sonatype-lift · 2022-04-09T09:18:09Z

ktor-network/ktor-network-tls/jvm/src/io/ktor/network/tls/SSLEngineSocket.kt

+                    handleResult(result)
+                    if (result.status == SSLEngineResult.Status.CLOSED) break@loop
+                }
+            } catch (cause: Throwable) {


TooGenericExceptionCaught: Caught exception is too generic. Prefer catching specific exceptions to the case that is currently handled.

(at-me in a reply with help or ignore)

sonatype-lift · 2022-04-09T09:18:10Z

ktor-network/ktor-network-tls/jvm/src/io/ktor/network/tls/SSLEngineUnwrapper.kt

+            if (unwrapDestination !== initialUnwrapDestination) updateUnwrapDestination(unwrapDestination)
+            resumeUnwrapContinuation(result)
+            return result
+        } catch (cause: Throwable) {


TooGenericExceptionCaught: Caught exception is too generic. Prefer catching specific exceptions to the case that is currently handled.

(at-me in a reply with help or ignore)

sonatype-lift · 2022-04-09T09:18:11Z

ktor-network/ktor-network-tls/linuxX64/src/io/ktor/network/tls/BIOPipe.kt

+            }
+
+            return result
+        } catch (cause: Throwable) {


TooGenericExceptionCaught: Caught exception is too generic. Prefer catching specific exceptions to the case that is currently handled.

(at-me in a reply with help or ignore)

sonatype-lift · 2022-04-09T09:18:14Z

ktor-network/ktor-network-tls/linuxX64/src/io/ktor/network/tls/PKCS12Certificate.kt

+                "Failed to parse PKCS12 file"
+            }
+            return PKCS12Certificate(privateKey.value!!, x509Certificate.value!!)
+        } catch (cause: Throwable) {


TooGenericExceptionCaught: Caught exception is too generic. Prefer catching specific exceptions to the case that is currently handled.

(at-me in a reply with help or ignore)

sonatype-lift · 2022-04-09T09:18:15Z

ktor-network/ktor-network-tls/linuxX64/src/io/ktor/network/tls/SSLSocket.kt

+                    this.channel.flush()
+                    //println("[$debugString] READING STEP RESULT: $result")
+                } while (result > 0)
+            } catch (cause: Throwable) {


TooGenericExceptionCaught: Caught exception is too generic. Prefer catching specific exceptions to the case that is currently handled.

(at-me in a reply with help or ignore)

e5l · 2022-05-20T05:33:47Z

ktor-network/ktor-network-tls/build.gradle.kts

-            api(project(":ktor-network"))
-            api(project(":ktor-utils"))
+kotlin {
+    linuxX64 {


could we try adding other platofrms?

e5l · 2022-05-20T05:33:57Z

ktor-network/ktor-network-tls/darwin/src/io/ktor/network/tls/PKCS12Certificate.kt

+
+package io.ktor.network.tls
+
+public class PKCS12Certificate(


kdocs are missing

e5l · 2022-05-20T05:34:19Z

ktor-network/ktor-network-tls/darwin/src/io/ktor/network/tls/PKCS12Certificate.kt

+
+public class PKCS12Certificate(
+    public val path: String,
+    public val password: (() -> CharArray)?


consider adding default if password really can be null

e5l · 2022-05-20T05:34:56Z

ktor-network/ktor-network-tls/darwin/src/io/ktor/network/tls/TLSConfigDarwin.kt

+    public actual val verification: TLSVerificationConfig?,
+) : Closeable {
+    override fun close() {
+        //NOOP


e5l · 2022-05-20T05:35:36Z

ktor-network/ktor-network-tls/jvmAndNix/src/io/ktor/network/tls/TLSConfig.kt

+import io.ktor.utils.io.core.*
+
+//TODO: make it closeable
+public expect class TLSConfig: Closeable {


Could you tell me why the config is closeable?

e5l · 2022-05-20T05:36:01Z

ktor-network/ktor-network-tls/jvmAndNix/test/io/ktor/network/tls/TlsClientSocketTest.kt

+                        writeStringUtf8("Connection: close\r\n\r\n")
+                        flush()
+                    }
+                    println(socket.openReadChannel().readRemaining().readText())


please consider dropping println

Tiebe · 2023-08-02T22:59:36Z

@olme04 are you still working on this?

EDIT: ah, zero contributions the past year, on any github repo :(

whyoleg · 2023-08-03T07:56:03Z

@Tiebe just in case, while using openssl here could be a good idea, the main problem with this approach is how to distribute it. I mean, we can do or dynamic or static linking to openssl or 2 separate artefacts for each, but it will be not so straightforward for end-users and library developers what to choose. Also there could be issues with how cinterop for curl (client engine) works with this cinterop for openssl. While this could be now implemented, there are a lot of questions and issues which should be done to make this production ready. As you see, there are a lot of other things in TODO column.

Tiebe · 2023-08-03T22:47:00Z

@whyoleg Fair enough. Would just really love to have a websocket client with TLS support in Kotlin Native. Would you have any recommendations?

whyoleg · 2023-08-04T07:12:47Z

@Tiebe I think, that for Apple platforms you can use darwin engine, for Windows - winhttp, for linux - there is really nothing now, because there are 2 ways: write TLS impl from scratch (you will still need cryptography support, from scratch or library, which have similar limitations) or use openssl/borinssl etc and leave with inconveniences :(
So, if you need linux support - you have no good ways for now

olme04 added 9 commits April 8, 2022 14:38

implement JVM tls support via SSLEngine

861fed4

implement JVM server CIO TLS support via SSLEngine

fb91fd6

TODO: test failed - HttpServerCommonTestSuite.testProxyHeaders

add apiDump

b6a2f3c

fix formatting

a6c32fe

remove printlns and add comment

b8266d7

add TLS support to native linux via openssl

ade156a

add TLS verification support

b7425fc

update CIO server TLS usage

3772f17

apidump

433a829

sonatype-lift bot reviewed Apr 9, 2022

View reviewed changes

olme04 mentioned this pull request Apr 12, 2022

CIO server HTTPS support on Native #2942

Open

e5l reviewed May 20, 2022

View reviewed changes

This was referenced Jun 27, 2022

Native websocket client support #1215

Closed

Add WebSocket support for the iOS #1894

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Native Sockets TLS Client/Server support for linux #2939

Native Sockets TLS Client/Server support for linux #2939

olme04 commented Apr 9, 2022

sonatype-lift bot Apr 9, 2022

sonatype-lift bot Apr 9, 2022

sonatype-lift bot Apr 9, 2022

sonatype-lift bot Apr 9, 2022

sonatype-lift bot Apr 9, 2022

e5l May 20, 2022

e5l May 20, 2022

e5l May 20, 2022

e5l May 20, 2022

e5l May 20, 2022

e5l May 20, 2022

Tiebe commented Aug 2, 2023 •

edited

whyoleg commented Aug 3, 2023

Tiebe commented Aug 3, 2023

whyoleg commented Aug 4, 2023

Native Sockets TLS Client/Server support for linux #2939

Are you sure you want to change the base?

Native Sockets TLS Client/Server support for linux #2939

Conversation

olme04 commented Apr 9, 2022

sonatype-lift bot Apr 9, 2022

Choose a reason for hiding this comment

sonatype-lift bot Apr 9, 2022

Choose a reason for hiding this comment

sonatype-lift bot Apr 9, 2022

Choose a reason for hiding this comment

sonatype-lift bot Apr 9, 2022

Choose a reason for hiding this comment

sonatype-lift bot Apr 9, 2022

Choose a reason for hiding this comment

e5l May 20, 2022

Choose a reason for hiding this comment

e5l May 20, 2022

Choose a reason for hiding this comment

e5l May 20, 2022

Choose a reason for hiding this comment

e5l May 20, 2022

Choose a reason for hiding this comment

e5l May 20, 2022

Choose a reason for hiding this comment

e5l May 20, 2022

Choose a reason for hiding this comment

Tiebe commented Aug 2, 2023 • edited

whyoleg commented Aug 3, 2023

Tiebe commented Aug 3, 2023

whyoleg commented Aug 4, 2023

Tiebe commented Aug 2, 2023 •

edited