Ensure GRPCServer.Stop() calls GRPCBroker.Close() #220
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Initializing a GRPCServer will do the following:
```go
// Register the broker service
brokerServer := newGRPCBrokerServer()
plugin.RegisterGRPCBrokerServer(s.server, brokerServer)
s.broker = newGRPCBroker(brokerServer, s.TLS)
go s.broker.Run()
```
The current Stop() method omits any handling of the underlying `s.broker`, which means the broker goroutine will leak:
```text
[Goroutine 53 in state select, with github.com/hashicorp/go-plugin.(*gRPCBrokerServer).Recv on top of the stack:
goroutine 53 [select]:
github.com/hashicorp/go-plugin.(*gRPCBrokerServer).Recv(0x29?)
/Users/bflad/go/pkg/mod/github.com/hashicorp/go-plugin@v1.4.4/grpc_broker.go:121 +0x58
github.com/hashicorp/go-plugin.(*GRPCBroker).Run(0x140003ca190)
/Users/bflad/go/pkg/mod/github.com/hashicorp/go-plugin@v1.4.4/grpc_broker.go:411 +0x40
created by github.com/hashicorp/go-plugin.(*GRPCServer).Init
/Users/bflad/go/pkg/mod/github.com/hashicorp/go-plugin@v1.4.4/grpc_server.go:85 +0x424
]
```
bflad
added a commit
to hashicorp/terraform-plugin-sdk
that referenced
this pull request
Nov 4, 2022
Reference: #1063 Reference: hashicorp/go-plugin#220 This includes the following small bug fixes: - Clarifies/fixes the trace logging during `terraform show` commands - Improves performance slightly for `terraform show` raw plan usage (terraform-exec already captures and returns stdout for raw plans) - Prevents a goroutine leak specific to the detached stop context in `schema.GRPCProvider` Previously, leak detection on testing would contain an entry per Terraform command such as: ``` [Goroutine 276 in state select, with github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.mergeStop on top of the stack: goroutine 276 [select]: github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.mergeStop({0x103a5f298?, 0x14000390c00?}, 0x1400033e5f0, 0x140004bb560) /Users/bflad/src/github.com/hashicorp/terraform-plugin-sdk/helper/schema/grpc_provider.go:46 +0x64 created by github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).StopContext /Users/bflad/src/github.com/hashicorp/terraform-plugin-sdk/helper/schema/grpc_provider.go:63 +0x178 ] ``` Leaked goroutines such as: ``` [Goroutine 53 in state select, with github.com/hashicorp/go-plugin.(*gRPCBrokerServer).Recv on top of the stack: goroutine 53 [select]: github.com/hashicorp/go-plugin.(*gRPCBrokerServer).Recv(0x29?) /Users/bflad/go/pkg/mod/github.com/hashicorp/go-plugin@v1.4.4/grpc_broker.go:121 +0x58 github.com/hashicorp/go-plugin.(*GRPCBroker).Run(0x140003ca190) /Users/bflad/go/pkg/mod/github.com/hashicorp/go-plugin@v1.4.4/grpc_broker.go:411 +0x40 created by github.com/hashicorp/go-plugin.(*GRPCServer).Init /Users/bflad/go/pkg/mod/github.com/hashicorp/go-plugin@v1.4.4/grpc_server.go:85 +0x424 ] ``` Will likely require a fix upstream in go-plugin.
SBGoods
approved these changes
Nov 4, 2022
bflad
added a commit
to hashicorp/terraform-plugin-sdk
that referenced
this pull request
Nov 7, 2022
Reference: #1063 Reference: hashicorp/go-plugin#220 This includes the following small bug fixes: - Clarifies/fixes the trace logging during `terraform show` commands - Improves performance slightly for `terraform show` raw plan usage (terraform-exec already captures and returns stdout for raw plans) - Prevents a goroutine leak specific to the detached stop context in `schema.GRPCProvider` Previously, leak detection on testing would contain an entry per Terraform command such as: ``` [Goroutine 276 in state select, with github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.mergeStop on top of the stack: goroutine 276 [select]: github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.mergeStop({0x103a5f298?, 0x14000390c00?}, 0x1400033e5f0, 0x140004bb560) /Users/bflad/src/github.com/hashicorp/terraform-plugin-sdk/helper/schema/grpc_provider.go:46 +0x64 created by github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).StopContext /Users/bflad/src/github.com/hashicorp/terraform-plugin-sdk/helper/schema/grpc_provider.go:63 +0x178 ] ``` Leaked goroutines such as: ``` [Goroutine 53 in state select, with github.com/hashicorp/go-plugin.(*gRPCBrokerServer).Recv on top of the stack: goroutine 53 [select]: github.com/hashicorp/go-plugin.(*gRPCBrokerServer).Recv(0x29?) /Users/bflad/go/pkg/mod/github.com/hashicorp/go-plugin@v1.4.4/grpc_broker.go:121 +0x58 github.com/hashicorp/go-plugin.(*GRPCBroker).Run(0x140003ca190) /Users/bflad/go/pkg/mod/github.com/hashicorp/go-plugin@v1.4.4/grpc_broker.go:411 +0x40 created by github.com/hashicorp/go-plugin.(*GRPCServer).Init /Users/bflad/go/pkg/mod/github.com/hashicorp/go-plugin@v1.4.4/grpc_server.go:85 +0x424 ] ``` Will likely require a fix upstream in go-plugin.
swenson
reviewed
Nov 8, 2022
| func (s *GRPCServer) Stop() { | ||
| if s.broker != nil { |
There was a problem hiding this comment.
I think you'll likely want to do this on GracefulStop() too?
There was a problem hiding this comment.
Good shout and looks like GracefulStop() blocks until operations are completed so should be safe to do afterwards. Updated in 3d6322f.
| func (s *GRPCServer) Stop() { | ||
| if s.broker != nil { | ||
| s.broker.Close() |
There was a problem hiding this comment.
I'd also recommend setting s.broker = nil here in case someone calls this function twice.
There was a problem hiding this comment.
Good call. Updated in 3d6322f.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Initializing a GRPCServer will do the following:
The current Stop() method omits any handling of the underlying
s.broker, which means the broker goroutine will leak:If there is another/better way to handle these, please reach out.