Persistent peer identity during restart make it unable to publish message with gossipsub #5097
Labels
Comments
|
This is very interesting. I don't have any resources to debug this but it is definitely something that shouldn't happen! :) |
|
From a little bit more tracing I found that logs for successful connection: After reboot: |
|
Does |
|
Both peers subscribe the topic on startup, and these are the logs of the peer that restarted. I'm just using the chat example, which does The node did not restart handles the subscription normally(note the time which corresponds to restarted connection): To ease understanding: in my setup, |
|
A little bit more observation:
But even after this, |
|
I noticed that the errored endpoint is the listen port of At this point, But a few seconds later, a After that So with a static listen port and address, the connection will eventually resume because we will receive a |
|
While tracing it further, my educated guess is that when a QUIC connection is not closed on I observed that |
|
Sounds almost related to #4917 (since the timout was reduced in #4965) but sounds much deeper. I noticed something similar while connected with just quic back then but only assume that it was due to the connection havent timed out on the other end. EDIT: I can reproduce this easily as well, which is an odd behaviour. |
|
To further isolate the issue, could somebody try reproducing this with just |
|
Have this ever been reproduced upstream yet? Interesting to know if this is something within libp2p-quic or quinn. |
|
I have had what I believe may be the same issue when connected a webrtc-peer (A) to a libp2p-rust relay node (B). In my case, once A restarts, it is unable to receive new gossipsub messages from B, even though B appears in A's list of connected peers. |
|
@julienmalard could you provide some logs? |
|
Here's an example log. I could also rerun with debug loggin enabled, but the log size becomes very large. |
|
I think the solution for now (at least for me and my use case) would be to use the following settings so that when PeerB disconnects, PeerA would handle the connection in a short time span so when PeerB comes back online and connects using the same identity (and even address), both peers will be able to send messages to each other. Not ideal, but is the workaround i think I may have to use as a default for now in my project (after first testing for any ill-effects of a short duration, which so far there dont seem to be any, but havent tried it on a larger scale yet). Long term, this should be looked into since PeerA does drop the old connection when PeerB does come online before the timeout kicks in, and even send a event to the new handle to join the mesh (after the old connection drops due to timeout), but PeerB refuse to receive any RPC messages from PeerA, although PeerB does receive packets from PeerA so that works, |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Summary
When using QUIC transport exclusively with a persistent identity, the restart of the client of one node would make that node unable to publish message and getting an
InsufficientPeerserror. However, bothswarmandgossipsubwould report that there is a connected peer.For example:
Both PeerA and PeerB has a static identity(and peer id derived from that identity) and connected to each other using QUIC transport exclusively, and gossipsub is able to send messages to each other.
Now PeerA restarts, connect to PeerB with the same identity, PeerA will be unable to publish any message and will always get
InsufficientPeers, but theswarm.connected_peers()andgossipsub.all_peers()are returning PeerB as connected peer. PeerB can publish message to PeerA.To reproduce this, we can simply modify the
chatexample:First we add
ed25519_dalekas a direct dependency, then add the following to thechatexample:Then change the
private_key.pemfor PeerA and PeerB, we can use openssl to generate the pem:Make sure PeerA and PeerB is using different private key, and restart one of the client, you will see
Publish error: InsufficientPeerson the restarted node.Expected behavior
The peer should be able to publish message even after restart
Actual behavior
The peer restarted will report
InsufficientPeers, the peer not restarted is able to publish message to the peer restarted.Relevant log output
Enter messages via STDIN and they will be sent to connected peers using Gossipsub Local node is listening on /ip4/127.0.0.1/udp/62392/quic-v1 Local node is listening on /ip4/192.168.125.145/udp/62392/quic-v1 mDNS discovered a new peer: 12D3KooWC1tB5J6W7NWwzXVFHZBggNLU1T5etuTtmpnf9WBQaJCm mDNS discovered a new peer: 12D3KooWC1tB5J6W7NWwzXVFHZBggNLU1T5etuTtmpnf9WBQaJCm test Publish error: InsufficientPeersPossible Solution
Current workarounds including:
swarm::ConnectionError::IOevent, then start the node. Starting the node after remote closed the connection will not triggerInsufficientPeersVersion
I tried "0.53.2" as well as running the chat example directly on
masterwith3e57cf494792031459d7f831408af24dbedc1a0b.Would you like to work on fixing this bug ?
Yes
The text was updated successfully, but these errors were encountered: