Resource temporarily unavailable (os error 11) during failovers

[matilote]

Whenever the validator needs to be failed over there is a need to change tmkms config and restart the service. When the service comes up it throws the errors below for ~1-5minutes. There are 5 protocols running using this tmkms instance.

Is there any workaround to mitigate the downtime during such maintenance?

This issue is likely related to #691 #323 and #73

Mar 07 08:44:08 host tmkms[1375109]: 2024-03-07T08:44:08.739880Z ERROR tmkms::client: [cosmoshub-4@tcp://X.X.X.X:26658] protocol error:
Mar 07 08:44:08 host  tmkms[1375109]:    0: io error
Mar 07 08:44:08 host  tmkms[1375109]:    1: Resource temporarily unavailable (os error 11)
Mar 07 08:44:08 host  tmkms[1375109]: Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Mar 07 08:44:08 host  tmkms[1375109]: Run with RUST_BACKTRACE=full to include source snippets.

[tony-iqlusion]

Never seen that particular error in our operations of TMKMS. It almost looks like you're hitting some sort of OS resource limit.

[tony-iqlusion]

Can you get the backtrace by adding RUST_BACKTRACE=1?

What operating system is this on? Is it possible you're exhausting available file descriptors or ephemeral TCP ports?

If you're on Linux, can you see if you can log the error from the OS using strace?