Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature request: support for pickling/unpickling interactively defined Cython functions and classes #502

Open
ogrisel opened this issue Feb 16, 2023 · 1 comment

Comments

@ogrisel
Copy link
Contributor

ogrisel commented Feb 16, 2023

It would be nice to be able to run interactively defined Cython functions in parallel on dask/ray/spark clusters or joblib/loky process pools.

Such interactively defined functions can typically be created using the %%cython magic in a Jupyter session or by a pyximport call in a Python script (e.g. from the __main__ module).

In both cases, it quite easy to interactively inspect the callable from Python to identify that their definition come from such an interactively defined Cython module.

A first step would be to add a new assume_shared_dynamic_cython_module_folder=False constructor parameter to the cloudpickle.Pickler class, in which case we could not need to ship the backing .so file over the wire (nor rebuild it from a source code sent over the wire).

This assumption would hold for the single host parallelization calls (e.g. joblib/loky or single host ray/dask setups) or for cluster computing setups with shared home folders between the driver node and the worker nodes (e.g. NFS home folder in jupyter hub + HPC cluster).

@ogrisel
Copy link
Contributor Author

ogrisel commented Feb 16, 2023

Somewhat related to #439 for numba.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant