Help : Server graceful shutdown

[ameykshirsagar]

Hi,
I have really liked the project for my microservices but how can I implement graceful shutdown of server instance. I have tried accessing server.close by implementing (*server.close).store(true, Ordering::Relaxed);
But it says that server.close is a private
Is there any other way to shutdown the server or I am doing anything wrong.

[Richienb]

This might work:

server.unblock();
std::mem::drop(server);

[kolbma]

See signal-hook.

I "catch" e.g. the 1st SIGTERM (or SIGINT), set an cond_term: AtomicBool, call unblock() for how many threads there are. In the spawned threads I check the AtomicBool, short before return in the spawned threads another AtomicU8 (th_stop_count_spawned) is counted up for stopping/exiting threads. (If you have more threads than u8::MAX, change the type) (s_num is my server thread number from the for-loop creating the threads)

let thread_builder = std::thread::Builder::new();
let th = thread_builder.spawn(move || {
    debug!("thread init {}", s_num);

    for req in server.incoming_requests() {
        // handle request
        let mut response = response_generate(&req, &req_cfg);
        if let Err(err) = req.respond(response) {
            error!("responding failed: {:?}", err);
        }

        if cond_term.load(Ordering::Relaxed) {
            debug!("thread {} stop graceful", s_num);
            // stop `incoming_requests()` by `signal_hook` condition
            break;
        }
    }

    // this here is called also after `unblock()` of `incoming_requests()`
    if cond_term.load(Ordering::Relaxed) {
        debug!("thread {} stop graceful", s_num);
        let _ = th_stop_count_spawned.fetch_add(1, Ordering::Relaxed);
    } else {
        panic!("unexpected thread abort");
    }
});

On program start I start a mostly sleeping thread like this...

let th = thread_builder.spawn(move || {
        debug!("graceful shutdown thread init");

        while !check_term.load(Ordering::Relaxed) {
            thread::sleep(Duration::from_millis(750));
        }

        info!(
            "graceful server shutdown in up to {} seconds",
            server_shutdown
        );

        for _ in 1..=server_num {
            server.unblock();
        }

        for _ in 0..server_shutdown {
            if th_stop_count_spawned.load(Ordering::Relaxed) >= server_num {
                return;
            }
            thread::sleep(Duration::from_secs(1));
        }

        exit_routine();
    });

server_shutdown is the graceful wait time period in seconds.
check_term is a different Arc::clone from cond_term above.
The return ends this extra thread and the program continues its code after the outside JoinHandle::join().
There I have a Vec with all JoinHandle from all threads and simply block that all threads are joinable.
The exit_routine() is a handler when the graceful wait time (in ~seconds) is over and there would be incoming_request threads left over after this wait time. So to cleanup also some stuff on not graceful shutdown and force a process::exit().