You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As of cloudpickle 2.0.0 (cloudpipe/cloudpickle#417) you can now manually register external dependencies to be pickled.
Currently, if we see a version mismatch between packages on the client and workers we show a warning to the user.
I wonder with this new feature whether we could instead register those packages for serialization so that the client-side packages are pickled and sent to the workers.
Thoughts?
The text was updated successfully, but these errors were encountered:
Using cloudpickles new pickle by value feature is definitely intriguing and opens up some new doors for distributed. I get the sense it's worth experimenting in this space, but I do wonder what sorts of edge cases will pop up (e.g. today we cache tasks on the worker to avoid repeated expensive (de)serialization, how will that interact with pickling by value?)
FWIW there's an initial POC related to pickling by value over in #4954
As of cloudpickle 2.0.0 (cloudpipe/cloudpickle#417) you can now manually register external dependencies to be pickled.
Currently, if we see a version mismatch between packages on the client and workers we show a warning to the user.
I wonder with this new feature whether we could instead register those packages for serialization so that the client-side packages are pickled and sent to the workers.
Thoughts?
The text was updated successfully, but these errors were encountered: