TMKMS stops signing

[mabalaru]

tmkms service stops signing and the validator is replying with:
E[2020-11-10|20:54:32.366] SignerListener: Ping timeout module=privval

There is no error msg on the tmkms side, only the validator is reporting this error. I need to restart both validator / tmkms in order to fix the issue.

This is happening on multiple networks (iris/terra/kava/gaiad) with version tmkms 0.8.0 and we have this issue starting with version 0.6.0.

Log from today from terra validator:

Nov 10 20:48:50 terrad[190405]: I[2020-11-10|20:48:50.547] Executed block                               module=state height=513449 validTxs=1 invalidTxs=0
Nov 10 20:48:50 terrad[190405]: I[2020-11-10|20:48:50.572] Committed state                              module=state height=513449 txs=1 appHash=A052FB0C265ABC247168289BE647747F848DA1D4DB8EBEDD6D7892505A7DB991
Nov 10 20:48:55 tmkms[190307]: #033[2mNov 10 20:48:55.350 #033[0m#033[32m INFO#033[0m tmkms::session: [columbus-4@tcp://127.0.0.1:26658] signed PreVote:79E334B43D at h/r/s 513450/0/1 (126 ms)
Nov 10 20:48:55 tmkms[190307]: #033[2mNov 10 20:48:55.689 #033[0m#033[32m INFO#033[0m tmkms::session: [columbus-4@tcp://127.0.0.1:26658] signed PreCommit:79E334B43D at h/r/s 513450/0/2 (125 ms)
Nov 10 20:48:56 terrad[190405]: I[2020-11-10|20:48:56.583] Executed block                               module=state height=513450 validTxs=0 invalidTxs=0
Nov 10 20:48:56 terrad[190405]: I[2020-11-10|20:48:56.669] Committed state                              module=state height=513450 txs=0 appHash=64419CB1866D5BE0296737F7784E7820100F4B76B9F86E92731832822852A95F
Nov 10 20:49:02 tmkms[190307]: #033[2mNov 10 20:49:02.100 #033[0m#033[32m INFO#033[0m tmkms::session: [columbus-4@tcp://127.0.0.1:26658] signed PreVote:CC5B90AC48 at h/r/s 513451/0/1 (126 ms)
Nov 10 20:49:08 terrad[190405]: E[2020-11-10|20:49:08.348] SignerListener: Ping timeout                 module=privval
Nov 10 20:49:11 terrad[190405]: E[2020-11-10|20:49:11.349] SignerListener: Ping timeout                 module=privval

Thank you

[zmanian]

Related change in Tendermint.
tendermint/tendermint#5638

[tony-iqlusion]

Possibly related previous issue: #37

@mabalaru can you try v0.9.0?

[tony-iqlusion]

Reopened #37 to track this