Re-Apply "Refactor connection.rs to make fail_with errors impossible" (double revert)

[yaahc]

TODO

• work out why the refactor sends more errors to the syncer
• merge these changes
• follow-up by making this code a lot easier to maintain - split functions, map complex matches to custom enums

Motivation

We want to refactor connection.rs to make fail_with errors impossible.

Solution

Reverts #1803

The reverts will continue until moral (sync speed) improves

[dconnolly]

Pending merge conflict fix

[teor2345]

This PR causes multiple errors during message processing, which slows down sync a lot.

We need to work out what's going wrong with the Connection state machine, and fix it.

I have some ideas for a design that would make this code a lot easier to maintain. @yaahc let's chat when you're ready to work on this?

[teor2345]

I also wonder if we should implement the sync/connection tests in #1592 before merging this PR.

[yaahc]

So status report from where I left this. The issue with this PR currently is that it causes the sync component to restart syncing in response to mixed inventory errors. The previous version seems to somehow not error out in a way that restarts syncing when it encounters these same errors. The next steps are to figure out how the current version handles mixed inventory errors, and see how that differs from the refactor.

[teor2345]

It's also worth noting that these errors have an outsized impact because:

• they seem to trigger connection failure bugs like Zebra gradually loses peer connections, until it has none left #1905
• they restart the sync, causing a delay of at least a minute per error, and even more due to pipeline error persistence Verification pipeline errors can persist through multiple sync restarts #1590
• Zebra should just split small mixed inv responses (merging those responses is a zcashd bug) Process inv with mixed item types #1768

It's weird that the current connection code appears to be dropping some of these errors - or at least responding to them differently.

But regardless of the fragility of this code and its context, we still need to make sure the refactor performs as well as the previous code.

[teor2345]

We've decided to close this PR for now, see ticket #1610 for next steps.