Getting TCP reset using tokio_boring + Hyper on the server when closing connections

[OldMansKid]

Hello Hyper devs,
I use the following code to drive the HTTPs server. During testing, the service forwards requests to Nginx listening at localhost:80,
but I'm getting TCP reset from curl https://www.my-domain.com, which is 127.0.0.2:9443, changed in /etc/hosts, and the certificate is valid too.
From the console log, I see "successfully disconnected with client", but in Wireshark, the server tries to send an encrypted alert even after receiving TCP [FIN, ACK] from the client. And this happens every time.

Does this look like a SSL configuration problem or a misuse of Hyper when shutting down?

fn start_client_service(
        &self,
        client_stream: tokio_boring::SslStream<TcpStream>,
        service_channel: ServiceChannel<ProxyBody>,
        shutdown_receiver: tokio::sync::oneshot::Receiver<i32>,
    ) {
        let (sender, receiver) = tokio::sync::oneshot::channel();
        let repetition_service = RepetitionService::new(service_channel, receiver);
        let client_io = TokioIo::new(client_stream);
        let client_conn = hyper::server::conn::http1::Builder::new();
        let conn_future = client_conn.serve_connection(client_io, repetition_service);
        let mut conn_future = Box::pin(conn_future);
        tokio::spawn(async move {
            tokio::select! {
                conn_result = conn_future.as_mut() => {
                    match conn_result {
                        Ok(()) => {log::info!("successfully disconnected with client"); let _ = sender.send(0);},
                        Err(e) => {log::warn!("error serving client: {}", e); let _ = sender.send(1);},
                    }
                },
                _ = shutdown_receiver => {
                    log::info!("shut down response received, gracefully shut down");
                    conn_future.as_mut().graceful_shutdown();
                    let result = conn_future.await;
                    log::info!("gracefully shutdown result: {:?}", result);
                }
            }
        });
    }

[seanmonstar]

Does this happen with other TLS implementations? Or without TLS? That could help identify where the problem is.

[OldMansKid]

Oh good idea, why didn't I think of that. It seems that the problem is with tokio_boring, with plain HTTP, Tokio + Hyper behaves normally:

[OldMansKid]

Hello Sean, I've spent the last few days to try to solve the TCP reset problem to no avail. Can the hyper-boring crate still be used with Hyper 1.0, or do your other higher level crates reqwest and warp support BoringSSL? I need to use BoringSSL to have the same TLS JA3 fingerprint as Chrome.

[seanmonstar]

I've never looked into BoringSSL crates, sorry.

[OldMansKid]

I solved this (mostly) through a hacky method. I created a wrapper over tokio's TcpStream with a cancellation token, before calling graceful_shutdown(), I cancel the token first to signal the wrapper to block all write attempts. Now I have this nice TCP closing process:

Thank you a lot for creating this library, I switched from C++ to Rust almost entirely because of Hyper.