Class attribute in ray actor cannot be accessed/copied

[jiangtianli91]

What is the problem?

Ray actor fails to access class attributes.

Ray version and other system information (Python version, TensorFlow version, OS):
ray 1.0.0
Python 3.7.7

Reproduction (REQUIRED)

This code is modified from dask/distributed#4233 (comment)

from module import Foo
import ray

@ray.remote
class ray_Foo:
    def __init__(self, Foo):
        self.Foo = Foo

    def show(self):
        return self.Foo.bar


ray.init()
foo = Foo()
foo.append(123)
print(foo.show())

ray_foo = ray_Foo.remote(foo)
print(ray.get(ray_foo.show.remote()))

$ cat module.py
class Foo:
    bar = []

    def append(self, value):
        self.bar.append(value)

    def show(self):
        return self.bar

$ python ray_issue_new.py
[123]
[]

If we cannot run your script, we cannot fix your issue.

• I have verified my script runs in a clean environment and reproduces the issue.
• I have verified the issue also occurs with the latest wheels.

[rkooo567]

This is an interesting issue. It seems like it only happens when we send serialized objects to other processes.

from module import Foo
import ray

@ray.remote
class ray_Foo:
    def __init__(self, Foo):
        self.Foo = Foo

    def show(self):
        return self.Foo.bar

@ray.remote
def f(foo):
    return foo.bar

ray.init()
foo = Foo()
foo.append(123)
print(foo.show())
p = ray.put(foo)
print(ray.get(p).bar)
ray_foo = ray_Foo.remote(foo)
print(ray.get(f.remote(p)))
print(ray.get(ray_foo.show.remote()))

As you can see

p = ray.put(foo)
print(ray.get(p).bar)

works, but it doesn't work only when objects are sent to other processes.

[rkooo567]

cc @robertnishihara Have you seen similar issues before?

[rkooo567]

Also @richardliaw

[wuisawesome]

Since this seems to be a pickling issue, I wonder if we should first try to reproduce the issue with just cloudpickle and forward this to their team.

[rkooo567]

@wuisawesome That sounds like a good idea. @jiangtianli91 Does this issue block you, or is it something that you can get around? (a.k.a, is it possible to not use class attributes? I assume this can be a blocker if this happens from other libraries that you depend on)

[jiangtianli91]

@wuisawesome That sounds like a good idea. @jiangtianli91 Does this issue block you, or is it something that you can get around? (a.k.a, is it possible to not use class attributes? I assume this can be a blocker if this happens from other libraries that you depend on)

So far we have hit this issue because our own libraries used class attributes, so we could move those although it does disrupt the organization a bit.

[rkooo567]

Gotcha. Btw, @suquark Have you seen any similar issue before?

[jenny-hm-lee]

I have this issue.

[jenny-hm-lee]

Have this issues been resolved in latest version of Ray?

[jiangtianli91]

@rkooo567 @wuisawesome Hello, it seems that cloudpickle has merged a pull request (here) that can solve this issue. I wonder whether it can be implemented here?

[mbwmbw1337]

We're seeing the same issue. What is the best workaround or have you been converting all class attributes to instances?

[mbwmbw1337]

Have this issues been resolved in latest version of Ray?

Not resolved as far as I am able to tell.

[jiangtianli91]

@rkooo567 Hello, I wonder is there any plan to fix this bug?

[rkooo567]

Hmm, if this is correct #11959 (comment), using the latest cloudpickle should fix the issue? Have we merged the upstream cloudpickle? cc @suquark

[filip-halt]

I am also having this issue, sending an object to an actor causes it to lose its attributes.

[clarkzinzow]

@suquark Do you know if that upstream fix was merged?

[qmihau]

Hi, I'm having that issue in ray 1.13.0 with cloudpickle 2.1.0. Is there any chance it will be resolved fairly soon?

[rkooo567]

Let us revisit this soon. Will bump up the priority.

[qmihau]

Thanks! Happy to help test or if you'll need any extra information.

[qmihau]

@rkooo567 Do you have a feel for when that might get into the work pipeline? Definitely don't want to be pushy, but just trying to plan our work accordingly :)

[rkooo567]

I think it is hard to guarantee the specific timeline for the fix. We are planning to take a look, but there are plenty of other issues, and this is currently not the highest priority.

[thomasbbrunner]

Can confirm that this is still an issue in Ray 2.6.3.