Is Dask distributing files to nodes?

[odo2063]

Hi Everybeeing!

Could it be that Dask distributes datafiles to the nodes?

imagine something like:
client.submit(doSomething, someFilename)
with
def doSomething(someFilename): import json data = json.load(someFilename)

Since I run a cluster file system i see low transferrates but the machine with daskscheduler sends things over the network with it's max rate. If dask distributes these data files how can I switch it off?

[mrocklin]

Dask serializes arguments using pickle (or variants thereof). If you send a string like "myfilename.txt' then it's just serializing that string and sending it. Dask does not know if a string that you provide as an argument is a filename or not. It makes no special consideration for strings that look like filenames.

…

On Thu, Nov 3, 2022 at 6:48 AM odo2063 ***@***.***> wrote: Hi Everybeeing! Could it be that Dask distributes datafiles to the nodes? imagine something like: client.submit(doSomething, someFilename) with def doSomething(someFilename): import json data = json.load(someFilename) Since I run a cluster file system i see low transferrates but the machine with daskscheduler sends things over the network with it's max rate. If dask distributes these data files how can I switch it off? — Reply to this email directly, view it on GitHub <#9618>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTGPCRJHZZWIAE5IXFDWGORA5ANCNFSM6AAAAAARWBRUAQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[odo2063]

Thank You very much...It was a configuration issue on my side with the clusterFS, now everything works as expected.