Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Register package mismatches for serialization with cloudpickle 2.0.0 #5306

Open
jacobtomlinson opened this issue Sep 10, 2021 · 1 comment
Open

Comments

@jacobtomlinson
Copy link
Member

As of cloudpickle 2.0.0 (cloudpipe/cloudpickle#417) you can now manually register external dependencies to be pickled.

Currently, if we see a version mismatch between packages on the client and workers we show a warning to the user.

I wonder with this new feature whether we could instead register those packages for serialization so that the client-side packages are pickled and sent to the workers.

Thoughts?

@jrbourbeau
Copy link
Member

Using cloudpickles new pickle by value feature is definitely intriguing and opens up some new doors for distributed. I get the sense it's worth experimenting in this space, but I do wonder what sorts of edge cases will pop up (e.g. today we cache tasks on the worker to avoid repeated expensive (de)serialization, how will that interact with pickling by value?)

FWIW there's an initial POC related to pickling by value over in #4954

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants