New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Imported object's method cannot be run properly when submited to dask client #4233
Comments
Thanks for raising an issue @jiangtianli91. Here's a minimal reproducer: In [1]: !cat test_module.py
class Foo:
bar = []
def append(self, value):
self.bar.append(value)
def show(self):
return self.bar
In [2]: from test_module import Foo
In [3]: foo = Foo()
In [4]: foo.append(124)
In [5]: foo.show()
Out[5]: [124]
In [6]: from distributed import Client
In [7]: client = Client()
In [8]: client.submit(foo.show).result()
Out[8]: [] My guess is this has to do with how cloudpickle handles sending class-level attributes across processes. For example, if in the above snippet we use |
@jrbourbeau Thank you for your explanation! I found the same problem in ray. I wonder is there a way to make accessing class-level attributed possible in dask? |
As this appears to be an issue with |
@jiangtianli91 this is a known limitation in In the meantime there is a temporary workaround where you can set the class Foo:
__module__ = "main"
bar = []
def append(self, value):
self.bar.append(value) but be aware this may also introduce side effects |
Closing as this is more of a |
What happened:
When I import an class and submit its method to client, it does not run properly. It runs fine if the class is defined in the same script.
I also checked cloudpickle. Cloudpickle can successfully pickle the object.
Minimal Complete Verifiable Example:
In this repo
Anything else we need to know?:
Environment:
dask 2.30.0
python 3.8.5
The text was updated successfully, but these errors were encountered: