Prevent Concurrent Writes

[steve-gray]

When writing via multiple goroutines, it's possible to end up causing the grpc-go streams to deadlock/block if you don't guard against this by putting a mutex outside calls to SendMsg. This is fine, until you want to use a compressor/decompressor - and suddenly these operations now mean you are blocking sends while compression happens - and there's no way to mitigate this externally, short of writing your own compression system on top of your gRPC base protocol.

Fixes #2355

[thelinuxfoundation]

Thank you for your pull request. Before we can look at your contribution, we need to ensure all contributors are covered by a Contributor License Agreement.

After the following items are addressed, please respond with a new comment here, and the automated system will re-verify.

• User @steve-gray isn't covered by a CLA. They will need to complete the form at https://identity.linuxfoundation.org/projects/cncf

Regards,
CLA GitHub bot

[steve-gray]

@thelinuxfoundation CLA signed.

[dfawley]

@steve-gray thanks for the contribution. Unfortunately, we do not want to enable concurrent SendMsg calls at this time. There are a few reasons I can provide for this:

1. Because ordering often does matter for streams, it could be confusing for users if we allowed this.
2. Streams not attempting this type of operation become penalized by the cost of taking this mutex.
3. Similarly, in a heavily loaded system, attempting to send in parallel will not end up leading to any performance gains; the CPU and network are already fully saturated, and the cost of synchronization becomes a negative.
4. If we were to allow this, the issue of fairness arises pretty quickly thereafter. sync.Mutexes have no concept of fairness, so it's possible that even if the user implements their own message reordering on the receiving end, one message could be postponed until many subsequent messages (hundreds/thousands/more) have been sent. This could result in problems for users not expecting that possibility - e.g. if they are buffering a message until all prior messages are received.

If this is a performance bottleneck for your use case, I would recommend something like what @snowzach described in #1879. Thanks again for the PR.