Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failing to deserialize user function #8454

Closed
rjzamora opened this issue Jan 11, 2024 · 3 comments · Fixed by #8456 · May be fixed by #8455
Closed

Failing to deserialize user function #8454

rjzamora opened this issue Jan 11, 2024 · 3 comments · Fixed by #8456 · May be fixed by #8455
Assignees
@fjetter
Labels
bug Something is broken regression

Comments

@rjzamora
Copy link
Member

rjzamora commented Jan 11, 2024 •
edited

Ran into this error in RAPIDS CI, but no GPU is needed to reproduce:

import dask.dataframe as dd
from dask.distributed import LocalCluster, Client, wait

if __name__ == "__main__":
    with LocalCluster() as cluster:
        client = Client(cluster)
        ddf = dd.from_dict({"a": range(10)}, 1)
        def func(df):
            return (df + 5) * 3 - 11
        wait(ddf.map_partitions(func).persist())
...
  File "/datasets/rzamora/miniconda3/envs/cudf_2402_dev/lib/python3.10/site-packages/distributed/protocol/core.py", line 172, in _decode_default
    return pickle.loads(sub_header["pickled-obj"], buffers=sub_frames)
  File "/datasets/rzamora/miniconda3/envs/cudf_2402_dev/lib/python3.10/site-packages/distributed/protocol/pickle.py", line 99, in loads
    return pickle.loads(x, buffers=buffers)
AttributeError: Can't get attribute 'func' on <module '__mp_main__' from '/home/nfs/rzamora/workspace/cudf-24.02/bug.py'>

I suspect that this was cause by #8447, but haven't confirmed this yet.
@rjzamora rjzamora added the bug Something is broken label Jan 11, 2024
@hendrikmakait
Copy link
Member

It looks like #8443 is to blame.

@rjzamora
Copy link
Member Author

It looks like #8443 is to blame.

Cool, was just about to say #8447 is not the problem.

@hendrikmakait
Copy link
Member

Feel free to dig deeper into this, otherwise I'll have a go at it tomorrow to get this fixed before the release.

@hendrikmakait hendrikmakait mentioned this issue Jan 11, 2024
Closed
4 tasks
@rjzamora rjzamora mentioned this issue Jan 11, 2024
Merged
3 tasks
@jakirkham jakirkham mentioned this issue Jan 11, 2024
Merged
vyasr pushed a commit to rapidsai/rapids-dask-dependency that referenced this issue Jan 11, 2024
@jakirkham
Pin Distributed to last working version (#17)
4a2dd6f 
To workaround issue ( dask/distributed#8454 ),
pin Distributed to a version before PR (
dask/distributed#8443 ) was included
This was referenced Jan 12, 2024
Draft
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fjetter fjetter

Labels
bug Something is broken regression
Projects
None yet
Milestone
No milestone
3 participants
@hendrikmakait @fjetter @rjzamora