keepalive signal SIGSEGV: segmentation violation

[jiacai2050]

Please see the FAQ in our main README.md, then answer the questions below before
submitting your issue.

What version of gRPC are you using?

1.20.1

What version of Go are you using (go version)?

go version go1.13 linux/amd64

What operating system (Linux, Windows, …) and version?

Linux

What did you do?

I use those opts to build a grpc client

	opts := []grpc.DialOption{
		grpc.WithInsecure(),
		grpc.WithDefaultCallOptions(grpc.MaxCallRecvMsgSize(env.RPCMsgMaxSize * 1024 * 1024)),
		grpc.WithKeepaliveParams(keepalive.ClientParameters{
			Time:                10 * time.Second,
			Timeout:             5 * time.Second,
			PermitWithoutStream: true,
		}),
		grpc.WithBackoffMaxDelay(30 * time.Second),
	}

What did you expect to see?

No panic

What did you see instead?

In server's log, I find

unexpected fault address 0x0
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x88b3e4]

goroutine 612 [running]:
runtime.throw(0xbd4c1e, 0x5)
  /usr/local/go/src/runtime/panic.go:774 +0x72 fp=0xc0006d1630 sp=0xc0006d1600 pc=0x42f2e2
runtime.sigpanic()
  /usr/local/go/src/runtime/signal_unix.go:401 +0x3de fp=0xc0006d1660 sp=0xc0006d1630 pc=0x44491e
gitlab.com/xxx/yy/vendor/google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc000788540)
  /go/src/gitlab.com/xxx/yy/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1331 +0x94 fp=0xc0006d17d8 sp=0xc0006d1660 pc=0x88b3e4
runtime.goexit()
  /usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0006d17e0 sp=0xc0006d17d8 pc=0x45ec91
created by gitlab.com/xxx/yy/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client
  /go/src/gitlab.com/xxx/yy/vendor/google.golang.org/grpc/internal/transport/http2_client.go:281 +0x1695

the panic root from
https://github.com/grpc/grpc-go/blob/v1.20.1/internal/transport/http2_client.go#L1331

[easwars]

@jiacai2050
Thanks for your problem report. I see that you are on quite an old release. Is upgrading to a newer release an option for you? Although, I don't think I understand the root cause of the issue to guarantee that this wont happen on the latest release.

Could you give us some more details:

1. Does the panic happen immediately on the client side, or does it happen after an arbitrary amount of time?
2. How often do you see this panic?
3. Also you say In server's log, I find. Did you mean to say on the client's log?
4. What keepalive configuration do you have on your server, if any?

Thanks again.

[jiacai2050]

1. after an arbitrary amount of time
2. hard to tell. there is little change to see this error in our logs
3. Yep, It's in client's log
4. the server is built in rust, and have no keepalive-related configuration

[easwars]

It does seem quite strange that you could have a panic root at the line that you mentioned. It makes me think that this could be some strange memory corruption, that you are running into.

Could you please consider upgrading to a newer version of grpc-go. Apart from other general improvements in the codebase, we have some improvements in the keepalive code as well (#3102 and #2987)

[stale]

This issue is labeled as requiring an update from the reporter, and no update has been received after 7 days. If no update is provided in the next 7 days, this issue will be automatically closed.

[jiacai2050]

I will try upgrade to latest version 1.25.0, and see what happens.

[stale]

This issue is labeled as requiring an update from the reporter, and no update has been received after 7 days. If no update is provided in the next 7 days, this issue will be automatically closed.