Clarify Nanny environment variable precedence

[fjetter]

Using keyword arguments for the nanny environ would overwrite the entire dask.config. I don't think this is intended and this fixes it by creating a soft copy.

I added a test and documentation about the precedence of configuration options

[quasiben]

Thanks @fjetter for the updates and doc changes as well

[jrbourbeau]

Thanks for the added test and documentation @fjetter. Both look totally reasonable to me.

Using keyword arguments for the nanny environ would overwrite the entire dask.config

I'm sure I'm just missing something, but I don't see how this was happening before. Could you elaborate a bit on how the soft copy helps avoid this?

[fjetter]

@jrbourbeau This test asserts the "wrong" behaviour and was green on main before this merge.

# git checkout 0528d8dc56dd42ef0049ee8293d240acdf909bd7

@gen_cluster(nthreads=[], client=True)
async def test_environment_variable_dont_overwrite(c, s):

    with dask.config.set({"distributed.nanny.environ": {"FOO": "456"}}):

        a = Nanny(s.address, loop=s.loop, memory_limit=0, env={"FOO": "123"})
        b = Nanny(s.address, loop=s.loop, memory_limit=0)

        await asyncio.gather(a, b)
        results = await c.run(lambda: os.environ["FOO"])
        assert results == {
            a.worker_address: "123",
            b.worker_address: "123",  # OOPS
            # b.worker_address: "456", # This should be correct
        }

        # The env kwarg should *never* overwrite the dask config
        assert dask.config.get("distributed.nanny.environ") == {"FOO": "123"}  # Ouch
        await asyncio.gather(a.close(), b.close())

The soft copy helps because the value in distributed.nanny.environ is actually just a dict and is passed on by reference, i.e. the following lines would modify the dict in dask.config

distributed/distributed/nanny.py

Lines 191 to 193 in 2298861

	self.env[k] = os.environ[k]
	if env:
	self.env.update(env)

by assigning a soft copy of the dict to self.env we're protected

[jrbourbeau]

Ah, I see. I missed the self.env.update(env) code path. Thanks for the clarification @fjetter!