Description
Denial of Service 1
Tendermint 0.33.2 and earlier does not limit the number of P2P connection requests. For each p2p connection, Tendermint allocates XXX bytes. Even though this memory is garbage collected once the connection is terminated (due to duplicate IP or reaching a maximum number of inbound peers), temporary memory spikes can lead to OOM (Out-Of-Memory) exceptions.
Tendermint 0.33.3 (and 0.32.10) limits the total number of P2P incoming connection requests to to p2p.max_num_inbound_peers + len(p2p.unconditional_peer_ids).
Notes:
- Tendermint does not rate limit P2P connection requests per IP (an attacker can saturate all the inbound slots);
- Tendermint does not rate limit HTTP(S) requests. If you expose any RPC endpoints to the public, please make sure to put in place some protection (https://www.nginx.com/blog/rate-limiting-nginx/). We may implement this in the future (#1696).
Denial of Service 2
Tendermint 0.33.2 and earlier does not reclaim activeID of a peer after it's removed in Mempool reactor. This does not happen all the time. It only happens when a connection fails (for any reason) before the Peer is created and added to all reactors. RemovePeer is therefore called before AddPeer, which leads to always growing memory (activeIDs map). The activeIDs map has a maximum size of 65535 and the node will panic if this map reaches the maximum. An attacker can create a lot of connection attempts (exploiting Denial of Service 1), which ultimately will lead to the node panicking.
Tendermint 0.33.3 (and 0.32.10) claims activeID for a peer in InitPeer, which is executed before MConnection is started.
Notes:
InitPeer function was added to all reactors to combat a similar issue - #3338;- Denial of Service 2 is independent of Denial of Service 1 and can be executed without it.
Impact
- All full nodes (except for validators who are behind closed networks)
- Node's memory usage increases, then it panics either in the mempool or due to OOM.
Patches
- v0.33.3
- v0.32.10
- v0.31.12
Workarounds
No workarounds.
References
For more information
If you have any questions or comments about this advisory:
More information can be found here.
Credits
- fudongbai for discovering and reporting Denial of Service 2
- Ethan Buchman (@ebuchman) for writing a test case for Denial of Service 2 and Tess Rinearson (@tessr) for fixing it
- Anton Kaliaev (@melekes) for fixing Denial of Service 1
Description
Denial of Service 1
Tendermint 0.33.2 and earlier does not limit the number of P2P connection requests. For each p2p connection, Tendermint allocates XXX bytes. Even though this memory is garbage collected once the connection is terminated (due to duplicate IP or reaching a maximum number of inbound peers), temporary memory spikes can lead to OOM (Out-Of-Memory) exceptions.
Tendermint 0.33.3 (and 0.32.10) limits the total number of P2P incoming connection requests to to
p2p.max_num_inbound_peers + len(p2p.unconditional_peer_ids).Notes:
Denial of Service 2
Tendermint 0.33.2 and earlier does not reclaim
activeIDof a peer after it's removed inMempoolreactor. This does not happen all the time. It only happens when a connection fails (for any reason) before thePeeris created and added to all reactors.RemovePeeris therefore called beforeAddPeer, which leads to always growing memory (activeIDsmap). TheactiveIDsmap has a maximum size of 65535 and the node will panic if this map reaches the maximum. An attacker can create a lot of connection attempts (exploiting Denial of Service 1), which ultimately will lead to the node panicking.Tendermint 0.33.3 (and 0.32.10) claims
activeIDfor a peer inInitPeer, which is executed beforeMConnectionis started.Notes:
InitPeerfunction was added to all reactors to combat a similar issue - #3338;Impact
Patches
Workarounds
No workarounds.
References
For more information
If you have any questions or comments about this advisory:
More information can be found here.
Credits