Protect vectorized payloads from wasteful and unneeded deserializations

[AndreaLanfranchi]

Abstract

Several message types carry vectorized payloads. There are limits to the number of items such payloads can hold (e.g. an inv message can't carry more than 50k items).

Issue

The check for max items threshold exceeded happens after the deserialization has been performed and therefore the code discards a content which had an additional cost to be produced. See here for example.
As a result it can happen a malicious peer can send several inv messages with way more items than max allowed per message before being disconnected for excessive penalization which leads to wasteful usage of resources and possibly increases the surface of attack for DoS.

Proposed solution

Early perform a lightweight check on behalf of raw datastream contents before starting the deserialization (and creation of vector) and return immediately in case of validation failure. This is a revised version of the partially implemented solution I pointed out on Horizen's Zend
The overall implementation adds negligible overhead to the processing of legit messages.
The checking function also verifies (optionally) whether the vector of items is actually not empty : this is to avoid a flood of messages (e.g. inv) with empty payloads that has the only goal to waste cycles for processing totally un-useful messages without any penalization.

Thank you for your attention.

Addendum

Questions for the devs:

• I believe the behavioral precondition for messages getblocks and getheaders should be standardized. The latter discards the message if IBD is true while getblocks doesn't. Which of the two is more appropriate ? Or is it correct as is ?
• If I'm not mistaken bitcoin protocol states for getheaders and getblocks the max number of block locator hashes must not exceed 2'000. Unless I miss something there's not such a check in the deserialization of the payload and as a result the code could end up deserializing up to 65.5K hashes in a single message. Is it worth checking early the size of the vector ?