Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better graceful shutdown #563

Open
liufuyang opened this issue Nov 16, 2023 · 0 comments
Open

Better graceful shutdown #563

liufuyang opened this issue Nov 16, 2023 · 0 comments

Comments

@liufuyang
Copy link
Contributor

It seems our framework cannot do a "proper" graceful shotdown due to we rely on the root context get canceled if SIGTERM is received, and we call those cancel functions AFTER the root context is already done.

F.g. a typical backend service is loaded up with such code:

func main() {
	var config config.Config
	if err := cloudrunner.Run(
		func(ctx context.Context) error {
			app, cleanup, err := server.Init(ctx, &config)
			if err != nil {
				return fmt.Errorf("init: %w", err)
			}
			defer cleanup()
			return cloudrunner.ListenGRPCHTTP(ctx, app.GRPCServer, app.HTTPServer)
		},
		cloudrunner.WithConfig("own", &config),
	); err != nil {
		log.Fatal(err)
	}
}

And in the server.Init(ctx, &config), I have some code init a db client, and it holds the root context, and I wire up the cleanup function like this:

                func() {
			select {
			case <-ctx.Done():
				fmt.Println("Root context is already done, cannot use client anymore.")
			default:
				fmt.Println("Root context is still live.")
			}
                        // In reality, I might have data in memory and I want to do a client.write(data) but in this case it is not possible anymore
			time.Sleep(2 * time.Second)
			client.Close()
		}

For such setup, you can see when the cleanup function is called, the root ctx is already done, making me cannot do some extra work (f.g. with the db client to do one more write) during the shutdown period.

... (logs omitted as those just showing service start normally)
2023-11-16T14:41:57.943+0100    debug   cloudmux/mux.go:63      serving gRPC
^C (ctrl-c to stop the service)
...
2023-11-16T14:42:02.487+0100    debug   cloudmux/mux.go:45      stopping cmux server
2023-11-16T14:42:02.488+0100    debug   cloudmux/mux.go:48      stopping HTTP server
2023-11-16T14:42:02.488+0100    debug   cloudmux/mux.go:67      stopped serving gRPC
2023-11-16T14:42:02.488+0100    debug   cloudmux/mux.go:76      stopped serving HTTP
2023-11-16T14:42:02.488+0100    debug   cloudmux/mux.go:56      stopping gRPC server
2023-11-16T14:42:02.488+0100    debug   cloudmux/mux.go:58      stopped both http and grpc server
...(waited for around 2 seconds)
Root context is already done, cannot use the client anymore.
2023-11-16T14:42:04.495+0100    info    cloudrunner-go/run.go:181       goodbye

I will try come up with a PR and see if I could fix this.

@liufuyang liufuyang changed the title Batter graceful shutdown Better graceful shutdown Nov 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant