Graceful shutdown for persistent connections

[zhu-he]

When an actix-web server receives a SIGTERM signal, it will stop accepting new connections and wait for all active connections to close. After the shutdown timeout configured by HttpServer::shutdown_timeout, the server will force close all active connections.

In our case, we have several servers behind a layer 4 load balancer. When we want to deploy a new version of our service, we will perform a rolling update. We will stop one old server, deploy the new version of the service, and start the new server. Before the old server is stopped, the load balancer will stop making new connections to the old server. However, the old server will still have some persistent connections. If some request is received before the old server is stopped, but not finished before the shutdown timeout, the client will receive a connection reset by peer error.

After some research, we found that other web frameworks have a more graceful shutdown strategy. For HTTP/1.1, the server will add a Connection: close header to all subsequent responses. For HTTP/2, the server will send a GOAWAY frame to the client. This will allow the client to close the connection gracefully.

For example, in the hyper crate, the above strategy is implemented in the code. All hyper-based web frameworks, such as axum, can benefit from this graceful shutdown strategy.

We have tried to implement this graceful shutdown strategy in actix-web using middleware. However, some difficulties are encountered. For example, the middleware cannot access the shutdown signal, but luckily signals are the only shutdown condition in our case, and we can directly watch them. Another difficulty is that the middleware can only solve the problem in HTTP/1.1, but not in HTTP/2, as the middleware cannot access the underlying connection.

We hope that actix-web can implement this graceful shutdown strategy. This will make actix-web more suitable for production environments.

[liketoeatcheese]

Hi @zhu-he ,

I'm using version 4.5.1 of Actix-web, and when I do kill -SIGTERM {actix server pid}, I got graceful shutdown logs:

[2024-03-01T15:31:06.361604665Z]  INFO: testapp/2721791 on archlinux: starting 32 workers (target=actix_server::builder, line=240)
    file: /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.3.0/src/builder.rs
[2024-03-01T15:31:06.361654984Z]  INFO: testapp/2721791 on archlinux: Actix runtime found; starting in Actix runtime (target=actix_server::server, line=196)
    file: /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.3.0/src/server.rs
[2024-03-01T15:49:20.526868842Z]  INFO: testapp/2721791 on archlinux: SIGTERM received; starting graceful shutdown (target=actix_server::server, line=328)
    file: /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.3.0/src/server.rs
[2024-03-01T15:49:20.526993052Z]  INFO: testapp/2721791 on archlinux: accept thread stopped (target=actix_server::accept, line=143)
    file: /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.3.0/src/accept.rs
[2024-03-01T15:49:20.527000752Z]  INFO: testapp/2721791 on archlinux: shutting down idle worker (target=actix_server::worker, line=595)
    file: /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.3.0/src/worker.rs
[2024-03-01T15:49:20.526999672Z]  INFO: testapp/2721791 on archlinux: shutting down idle worker (target=actix_server::worker, line=595)
    file: /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.3.0/src/worker.rs
[2024-03-01T15:49:20.526999202Z]  INFO: testapp/2721791 on archlinux: shutting down idle worker (target=actix_server::worker, line=595)
    file: /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.3.0/src/worker.rs
[2024-03-01T15:49:20.527008162Z]  INFO: testapp/2721791 on archlinux: shutting down idle worker (target=actix_server::worker, line=595)
    file: /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.3.0/src/worker.rs
[2024-03-01T15:49:20.527039163Z]  INFO: testapp/2721791 on archlinux: shutting down idle worker (target=actix_server::worker, line=595)
    file: /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.3.0/src/worker.rs
[2024-03-01T15:49:20.527040443Z]  INFO: testapp/2721791 on archlinux: shutting down idle worker (target=actix_server::worker, line=595)
    file: /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.3.0/src/worker.rs
[2024-03-01T15:49:20.527045863Z]  INFO: testapp/2721791 on archlinux: shutting down idle worker (target=actix_server::worker, line=595)
    file: /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.3.0/src/worker.rs
[2024-03-01T15:49:20.527062153Z]  INFO: testapp/2721791 on archlinux: shutting down idle worker (target=actix_server::worker, line=595)
    file: /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.3.0/src/worker.rs
[2024-03-01T15:49:20.527048713Z]  INFO: testapp/2721791 on archlinux: shutting down idle worker (target=actix_server::worker, line=595)
    file: /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.3.0/src/worker.rs
[2024-03-01T15:49:20.527065093Z]  INFO: testapp/2721791 on archlinux: shutting down idle worker (target=actix_server::worker, line=595)
    file: /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.3.0/src/worker.rs
[2024-03-01T15:49:20.527065673Z]  INFO: testapp/2721791 on archlinux: shutting down idle worker (target=actix_server::worker, line=595)
    file: /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/actix-server-2.3.0/src/worker.rs

Not sure which version you are using, maybe the maintainers did add this on the later version