Segfault in gpr_mpscq_pop_and_check_end

[michihenning]

I have a trivial client and server using streams. There are two streams between client and server, one for sending data to the server, and one for reading data from the server. The client sends 88-byte messages repeatedly. The server echoes those message back on the other stream.

All this works fine, for millions or billions of messages. Eventually, the client dumps core with the stack trace below. In that particular run, it fell over after 245 million successful messages. When the crash happens, all my threads are exactly where I expect them to be. The server does not ever encounter an error from gRPC when this happens, and none of the gRPC calls in the client return an error.

All this is with gRPC 1.18.0 on Ubuntu 16.04. Tool chain is gcc-7 (Ubuntu 7.4.0-1ubuntu116.04ppa1) 7.4.0.

I built gRPC from source, optimized build.

I've attached trivial source code for this. Please edit client and server to adjust the IP addresses as needed. I'm running client and server on machines with various CPUs (Core i-7, Xeon 8175M, and others) and the crash happens in virtual machines. Time to crash varies a lot, from minutes to days.

bug2.zip

244900000
245000000
245100000

Thread 8 "default-executo" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff2114700 (LWP 2320)]
0x00007ffff66525af in gpr_mpscq_pop_and_check_end(gpr_mpscq*, bool*) () from /home/mhenning/.local/lib/libgrpc.so.7
(gdb) bt
#0 0x00007ffff66525af in gpr_mpscq_pop_and_check_end(gpr_mpscq*, bool*) () from /home/mhenning/.local/lib/libgrpc.so.7
#1 0x00007ffff665265e in gpr_mpscq_pop(gpr_mpscq*) () from /home/mhenning/.local/lib/libgrpc.so.7
#2 0x00007ffff659aa55 in grpc_combiner_continue_exec_ctx() () from /home/mhenning/.local/lib/libgrpc.so.7
#3 0x00007ffff65a5e93 in grpc_core::ExecCtx::Flush() () from /home/mhenning/.local/lib/libgrpc.so.7
#4 0x00007ffff65a5f9f in GrpcExecutor::RunClosures(char const*, grpc_closure_list) ()
from /home/mhenning/.local/lib/libgrpc.so.7
#5 0x00007ffff65a621c in GrpcExecutor::ThreadMain(void*) () from /home/mhenning/.local/lib/libgrpc.so.7
#6 0x00007ffff6654b54 in ?? () from /home/mhenning/.local/lib/libgrpc.so.7
#7 0x00007ffff74a46ba in start_thread (arg=0x7ffff2114700) at pthread_create.c:333
#8 0x00007ffff69d141d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
(gdb)

[stale]

This issue/PR has been automatically marked as stale because it has not had any update (including commits, comments, labels, milestones, etc) for 180 days. It will be closed automatically if no further update occurs in 1 day. Thank you for your contributions!

[yashykt]

This should have been fixed by #19531