Use a worker-specific random source to remove lock contention.

[matthewdale]

What does this PR do ?

Use a separate random number source per worker. Currently, the worker function shouldSample() calls rand.Float64(), which uses a lock to serialize all calls to the package-global pseudorandom number generator, creating lock contention between all instances of worker.

Changes:

• Create and seed a separate pseudorandom number generator for every worker to remove the lock contention Use of rand.Float64() reduces parallelism #80
• Add a test for the shouldSample() function to assert that it returns true the expected percentage of the time
• Add a benchmark for the shouldSample() function to measure improvement from removing lock contention

Benchmarks against master

name            old time/op  new time/op  delta
ShouldSample     444ns ± 0%   190ns ± 1%  -57.26%  (p=0.000 n=8+9)
ShouldSample-2   876ns ± 2%    94ns ± 0%  -89.27%  (p=0.000 n=10+9)
ShouldSample-4  1.17µs ± 1%  0.09µs ± 2%  -92.59%  (p=0.000 n=8+8)
ShouldSample-8  1.25µs ± 3%  0.09µs ± 4%  -92.77%  (p=0.000 n=9+10)

[hush-hush]

Thanks for the PR, this looks great. I added a few nit-pick but otherwise this looks ready to go.

[hush-hush]

Thanks for the PR !

[rafaeljusto]

@matthewdale @hush-hush Shouldn't we have a local lock here? From the docs:

The default Source is safe for concurrent use by multiple goroutines, but Sources created by NewSource are not.

I think this is causing race conditions.

[matthewdale]

You're correct. When MutexMode is enabled, the call to worker.processMetric() is made from the same goroutine as the call to the top-level metric emit function (e.g. Count). I was previously under the impression that all calls to worker.processMetric() would be made from a single goroutine per worker, but that's not the case.

I'm working on a fix for the data race bug and will submit it when it's ready.