Graceful Shutdown

[geekerzhou]

Pre-check

• I am sure that all the content I provide is in English.

Search before asking

• I had searched in the issues and found no similar feature requirement.

Apache Dubbo Component

Java SDK (apache/dubbo)

Descriptions

2024-03-29 13:41:07.854 [DubboShutdownHook] INFO , o.a.dubbo.config.DubboShutdownHook - [DUBBO] Dubbo shutdown hooks execute now. Dubbo Application1.1, dubbo version: 3.2.9, current host: 10.10.1.33

2024-03-29 13:41:08.155 {"status":"UP","components":{"db":{"status":"UP","details":{"database":"PostgreSQL","validationQuery":"isValid()"}},"discoveryComposite":{"description":"Discovery Client not initialized","status":"UNKNOWN","components":{"discoveryClient":{"description":"Discovery Client not initialized","status":"UNKNOWN"}}},"diskSpace":{"status":"UP","details":{"total":250685575168,"free":15884992512,"threshold":10485760,"path":"","exists":true}},"dubbo":{"status":"UP","details":{"memory":{"source":"management.health.dubbo.status.defaults","status":{"level":"OK","message":"max:4096M,total:224M,used:188M,free:36M"}},"load":{"source":"management.health.dubbo.status.defaults","status":{"level":"OK","message":"load:2.67822265625,cpu:12"}},"registry":{"source":"management.health.dubbo.status.defaults","status":{"level":"OK","message":"10.0.2.251:8848(connected),10.0.2.251:8848(connected)"}},"spring":{"source":"management.health.dubbo.status.defaults","status":{"level":"OK","message":""}},"server":{"source":"management.health.dubbo.status.defaults","status":{"level":"OK","message":"/10.10.1.33:20880(clients:0)"}}}},"nacosConfig":{"status":"UP"},"ping":{"status":"UP"},"redis":{"status":"UP","details":{"version":"6.0.9"}},"refreshScope":{"status":"UP"}}}

2024-03-29 13:41:12.098 {"status":"UP","components":{"db":{"status":"UP","details":{"database":"PostgreSQL","validationQuery":"isValid()"}},"discoveryComposite":{"description":"Discovery Client not initialized","status":"UNKNOWN","components":{"discoveryClient":{"description":"Discovery Client not initialized","status":"UNKNOWN"}}},"diskSpace":{"status":"UP","details":{"total":250685575168,"free":15878217728,"threshold":10485760,"path":"","exists":true}},"dubbo":{"status":"UNKNOWN","details":{"memory":{"source":"management.health.dubbo.status.defaults","status":{"level":"OK","message":"max:4096M,total:238M,used:155M,free:83M"}},"load":{"source":"management.health.dubbo.status.defaults","status":{"level":"OK","message":"load:2.67822265625,cpu:12"}},"registry":{"source":"management.health.dubbo.status.defaults","status":{"level":"OK","message":"10.0.2.251:8848(connected),10.0.2.251:8848(connected)"}},"spring":{"source":"management.health.dubbo.status.defaults","status":{"level":"OK","message":""}},"server":{"source":"management.health.dubbo.status.defaults","status":{"level":"UNKNOWN"}}}},"nacosConfig":{"status":"UP"},"ping":{"status":"UP"},"redis":{"status":"UP","details":{"version":"6.0.9"}},"refreshScope":{"status":"UP"}}}

Dubbo has been taken offline, but the actuator health check information still shows Dubbo status as UP. It takes about 5 seconds for the server check item to change to UNKNOWN, yet the overall healthcheck remains UP. Using Nginx to request /actuator/health for determining whether to take the upstream offline is not feasible anymore. Since Dubbo has been taken offline, if the requested HTTP interface involves Dubbo-related logic, it throws an exception.

Related issues

No response

Are you willing to submit a pull request to fix on your own?

• Yes I am willing to submit a pull request on my own!

Code of Conduct

• I agree to follow this project's Code of Conduct

[AlbumenJ]

How can you let dubbo shutdown?

[geekerzhou]

docker restart

[AlbumenJ]

This is not a good graceful shutdown hook. It would recommend to add a PreHook to execute curl http://127.0.0.1:22222/offline

[geekerzhou]

my solution here:

@Log
public class ImmediateDownClosedEventListener implements ApplicationListener<ContextClosedEvent> {

    @Resource
    private DubboHealthIndicator indicator;

    public ImmediateDownClosedEventListener(DubboHealthIndicator indicator) {
        this.indicator = indicator;
    }

    @Override
    public void onApplicationEvent(ContextClosedEvent event) {
        log.info("Context is closing, setting health status to DOWN...");
        indicator.shutdownSignalReceived();
    }
}

public class DubboHealthIndicator extends AbstractHealthIndicator {

    private volatile boolean isShutdownSignalReceived = false;

    private volatile boolean dubboStarted = false;

    @Override
    protected void doHealthCheck(Health.Builder builder) throws Exception {
        builder.up();

        //监听到 spring 的应用停机事件，即 kill 命令触发时。由于默认的 dubbo indicator优雅停机顺序为，
        // 先取消 dubbo 订阅再干掉 spring 容器，会导致 dubbo 已经不可用但 spring 内置的 tomcat 容器依然可用，且 health 端点返回结果为 UP，导致http 请求报错
        if (isShutdownSignalReceived) {
            builder.down().withDetail("reason", "Shutdown signal received");
        }
        // 监听 dubbo 真正启动了，可以正常调用了才挂 health。官方的实现为 starting 即返回 UP，但是实际上 dubbo 还没完全启动，会导致 http 请求阻塞
        if (!dubboStarted) {
            ModuleServiceRepository service = ApplicationModel.defaultModel().getDefaultModule().getServiceRepository();
            dubboStarted = (service.getReferredServices().isEmpty() && service.getExportedServices().isEmpty()) || DubboBootstrap.getInstance().isStarted();
        }
        if (!dubboStarted) {
            builder.down().withDetail("reason", "Dubbo is not started");
        }
    }

    public void shutdownSignalReceived() {
        this.isShutdownSignalReceived = true;
    }
}