fix: synchronize the reserve sampling process

[aloknerurkar]

Checklist

• I have read the coding guide.
• My change requires a documentation update, and I have done it.
• I have added tests to cover my changes.
• I have filled out the description and linked the related issues.

Description

In the localstore we usually guard parallel operations using the batchMu mutex. This is a huge bottleneck, but one that cannot be avoided at the moment. Due to the nature of the current localstore code, we need to have this exclusive lock to protect any changes to the underlying DB.

For the sampling, we never acquired this lock. But we should as there could be changes to the pullIndex during iteration which should not happen. Also, currently, we do not acquire this lock when we evict batches from the batchstore. Both these cases are now fixed.

With this change, if there is a sampling process ongoing and reserve evictions happen, we will cancel the sampler. This will cause the player to not participate in the lottery game for that round. This is considered better as if the player does participate and end up submitting a wrong sample, he might get frozen. Also, having the lock for the sampling process seems to be an overkill and would slow down the Put path causing bottlenecks.

Along with this, I have added metrics for the BatchEviction process which were missing.

Fixes #3611 hopefully.

---

This change is 

[janos]

The approach looks pretty good with not so drastic changes. It should be good to go after linter is satisfied.