grpc client send stuck at epoll_pwait() forever

[Qian-Zhang7]

Grpc experters,

We have a grpc client (C++) that frequently stream data (multiple records for each stream) to the server side (server side is in a remote cluster implemented using google go). We see client stream often stuck at epoll_pwait() for long long time. Can anyone shed light on what it is waiting for? Is it client's fault or server's side fault?

Thank you very much!
Qian

----grpc client stack trace ---
Loaded symbols for /usr/lib64/libbz2.so.1
0x00007fb870200a3d in poll () from /usr/lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install augeas-libs-1.4.0-2.el7_4.2.x86_64 blosc-1.11.3-1.cisco.el7.centos.x86_64 bzip2-libs-1.0.6-13.el7.x86_64 elfutils-libelf-0.168-8.el7.x86_64 elfutils-libs-0.168-8.el7.x86_64 glibc-2.17-196.el7_4.2.x86_64 libattr-2.4.46-12.el7.x86_64 libcap-2.22-9.el7.x86_64 libgcc-4.8.5-16.el7_4.1.x86_64 libselinux-2.5-11.el7.x86_64 libstdc++-4.8.5-16.el7_4.1.x86_64 libxml2-2.9.1-6.el7_2.3.x86_64 libzstd-1.3.2-1.el7.x86_64 lz4-1.7.3-1.el7.x86_64 pcre-8.32-17.el7.x86_64 protobuf3-3.4.0-1.cisco.el7.centos.x86_64 qdbm-1.8.78-1.cisco.el7.centos.x86_64 snappy-1.1.0-3.el7.x86_64 systemd-libs-219-42.el7_4.4.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-17.el7.x86_64
(gdb) info thread
Id Target Id Frame
5 Thread 0x7fb86e578700 (LWP 15260) "cdb_export" 0x00007fb87020b587 in epoll_pwait () from /usr/lib64/libc.so.6
4 Thread 0x7fb86dd77700 (LWP 15261) "cdb_export" 0x00007fb870d01945 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
3 Thread 0x7fb86d576700 (LWP 15262) "cdb_export" 0x00007fb870d01cf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0
2 Thread 0x7fb867bd7700 (LWP 15263) "cdb_export" 0x00007fb870d01945 in pthread_cond_wait@@GLIBC_2.3.2 () from /usr/lib64/libpthread.so.0

• 1 Thread 0x7fb87223da80 (LWP 15259) "cdb_export" 0x00007fb870200a3d in poll () from /usr/lib64/libc.so.6
  (gdb) thread 5
  [Switching to thread 5 (Thread 0x7fb86e578700 (LWP 15260))]
  #0 0x00007fb87020b587 in epoll_pwait () from /usr/lib64/libc.so.6
  (gdb) bt
  #0 0x00007fb87020b587 in epoll_pwait () from /usr/lib64/libc.so.6
  Cleanup the win32 build a little. #1 0x00007fb86f725481 in ?? () from /usr/lib64/libgrpc.so.4
  Cleanup testing & polish infrastructure some. #2 0x00007fb86f741a3c in ?? () from /usr/lib64/libgrpc.so.4
  Fix compilation by checking return status. #3 0x00007fb86f74255b in grpc_completion_queue_pluck () from /usr/lib64/libgrpc.so.4
  Fix compile: check return value for error #4 0x00007fb871e129c8 in grpc::CoreCodegen::grpc_completion_queue_pluck(grpc_completion_queue*, void*, gpr_timespec, void*) () from /usr/lib64/libgrpc++.so.1
  Added libdl include to make the PHP library compile #5 0x000000000048f1a0 in Pluck (tag=0x7fb86e5775d0, this=0x7fb86809dca8) at /usr/include/grpc++/impl/codegen/completion_queue.h:224
  Fix and run C++ tests #6 ClientWriternam::PushResponse (response=0x7fb86e577730, context=0x7fb86e577790, method=..., channel=0x7fb8680024e0, this=0x7fb86809dc20)
  at /usr/include/grpc++/impl/codegen/sync_stream.h:282
  Add generated files to .gitignore #7 nam::NamData::Stub::PushRaw (this=, context=context@entry=0x7fb86e577790, response=response@entry=0x7fb86e577730) at gen-cpp/nam_data.grpc.pb.cc:34
  Tools to generate coverage reports #8 0x0000000000416bcd in Push (response=0x7fb86e577730, context=0x7fb86e577790, this=) at /home/cisco/nam/nam/protobuf-defs/gen-cpp/nam_data.grpc.pb.h:67
  Remove Go compiler from this repository #9 NamDataClient::stream_pages (this=0x7fb8680008c0, rec=rec@entry=0x7fb86800fc90, table_name=table_name@entry=0x70bdc8 <Schema+11656> "ARTCltSvr",
  num_records=num_records@entry=1004, record_size=record_size@entry=508, aggr_interval=aggr_interval@entry=25, exporter_addr=...,
  interval_index=interval_index@entry=45701) at nam_data_client.cc:209
  Remove libevent submodule #10 0x000000000040e4c8 in cdbe_grpc_send (index=index@entry=1, record_buff=0x7fb86800fc90, sch_p=sch_p@entry=0x70bdc8 <Schema+11656>, num_records=1004, record_size=508,
  aggr_interval=25, interval_index=interval_index@entry=45701) at cdb_export.cpp:389
  Compile fix for mac #11 0x000000000040e828 in cdbe_read_and_send (index=index@entry=1, read_info=read_info@entry=0x7fb86e577ac0, gen_info=gen_info@entry=0x7fb86e577ad0,
  file_id=file_id@entry=1) at cdb_export.cpp:489
  Fixes the shebang line #12 0x000000000040ec00 in cdbe_thread_run (ptr=) at cdb_export.cpp:673
  Remove go codegen from this repository #13 0x00007fb870cfde25 in start_thread () from /usr/lib64/libpthread.so.0
  clang-format codebase #14 0x00007fb87020b34d in clone () from /usr/lib64/libc.so.6
  (gdb) up

[kpayson64]

Can you show the output when it hangs with GRPC_VERBOSITY=debug GRPC_TRACE=all?

[kpayson64]

This issue has been dormant for months and was waiting for reporter feedback.