Backport of #5967 - Allow PersistentShardCoordinator to tolerate duplicate ShardHomeAllocated messages

[Arkatufus]

Backport of #5967

[ismaelhamed]

A couple questions: 1) Is this safe all the times?; 2) Will there be a way to disable this new behavior?

[Aaronontheweb]

1. Not sure - the alternative is defaulting the current behavior, which is that the ShardCoordinator blows up and the cluster needs to be restarted and all of the ShardCoordinator data deleted. Given that, I think this probably is safer than the current defaults. The worst case scenario I can imagine here is that the Shard was actually allocated to two different nodes, but that would if that were the case then the sharding system would already be in very bad shape (i.e. violating its consistency rules) and this would allow the newest home for the shard to supersede the old one, which would stop receiving message traffic. In the logs where I saw this occuring it was duplicate records for the same Shard in the same ShardRegion IActorRef - so this change would just make the recovery idempotent.

2. We could add it, but I don't think it should be necessary.

[Aaronontheweb]

@ismaelhamed I think I can make this change safer by narrowing the conditions during which we de-duplicate ShardHomeAllocated messages - which is we don't throw when the exact data is already inside the PersistentShardCoordinator.State since that's a no-op. That should open this system up to less unpredictable behavior since the shard locations are identical.

[Arkatufus]

Could be superseeded by #5976