Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update the cloudpickling, possibly put into dill #2

Open
rgbkrk opened this issue Apr 13, 2015 · 3 comments
Open

Update the cloudpickling, possibly put into dill #2

rgbkrk opened this issue Apr 13, 2015 · 3 comments

Comments

@rgbkrk
Copy link
Member

rgbkrk commented Apr 13, 2015

Since cloud is no longer maintained and folks want the cloudpickle, we should extract it out into dill or a separate library.

Example usage: ipython/ipython#5718

@ygravrand
Copy link

I tested multyvac with our new "cloudpickle" module instead of the one in "util" and it fails. Currently looking into it but there are lots of differences between the two.

Traceback (most recent call last):
  File "simpletest.py", line 13, in <module>
    job_id = multyvac.submit(add, 2, 3)
  File "$$edited$$/multyvac-fork/multyvac/job.py", line 391, in submit
    v = self.multyvac.volume.get(vol_name)
  File "$$edited$$/multyvac-fork/multyvac/volume.py", line 211, in get
    vs = self.list(name)
  File "$$edited$$/multyvac-fork/multyvac/volume.py", line 252, in list
    params=params)
  File "$$edited$$/multyvac-fork/multyvac/multyvac.py", line 233, in _ask
    files=files)
  File "$$edited$$/multyvac-fork/multyvac/multyvac.py", line 283, in _ask_helper
    hint=r.text)
multyvac.multyvac.RequestError: Could not parse body (Code: None Hint: 404 page not found)

@ygravrand
Copy link

Update: server side, we try to import "cloudpickle.cloudpickle"

multyvac.job.JobError: Traceback (most recent call last):
  File "/home/rho/anaconda/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/home/rho/anaconda/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/rho/anaconda/lib/python2.7/site-packages/multyvacinit/pybootstrap.py", line 10, in <module>
    (f, args, kwargs) = pickle.load(sys.stdin)
  File "/home/rho/anaconda/lib/python2.7/pickle.py", line 1378, in load
    return Unpickler(file).load()
  File "/home/rho/anaconda/lib/python2.7/pickle.py", line 858, in load
    dispatch[key](self)
  File "/home/rho/anaconda/lib/python2.7/pickle.py", line 1090, in load_global
    klass = self.find_class(module, name)
  File "/home/rho/anaconda/lib/python2.7/pickle.py", line 1124, in find_class
    __import__(module)
ImportError: No module named cloudpickle.cloudpickle

References to this module come from "_fill_function" and "_make_skel_func".
See client output when the first "print" in "save_function" function is uncommented:

which gives __main__ <function add at 0x108c9aa28> add
which gives cloudpickle.cloudpickle <function _fill_function at 0x108e2c6e0> _fill_function
which gives cloudpickle.cloudpickle <function _make_skel_func at 0x108e2c848> _make_skel_func

Now we would want to see why the actual pickling fails.

Edit: could the root cause be #5 ?

@ygravrand
Copy link

Even with the right 'cloudpickle.cloudpickle' import in the docker container, cloudpickle (git version) having been installed in the docker container, we still mysteriously fail client side with the error "multyvac.multyvac.RequestError: Could not parse body." No exception server side.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants