types.CodeType arguments differ from CPython

[lesteve]

import types

types.CodeType(
    0,
    0,
    0,
    0,
    0,
    19,
    b"\x06\x004\x00K",
    (None, "test.<locals>.func"),
    (),
    (),
    "/tmp/test.py",
    "func",
    17,
    b"\x02\x00\x00\x01",
    (),
    (),
)

With nogil:

TypeError: an integer is required (got type bytes)

The same snippet works fine with CPython 3.9.

Context: I was trying to run the joblib test suite and one of the error happens when pickling a local function with cloudpickle:

import cloudpickle
import pickle

import types


def test():
    def func():
        pass

    pickled = cloudpickle.dumps(func)
    unpickled = pickle.loads(pickled)

test()

py-tb
Traceback (most recent call last):
  File "/tmp/test.py", line 14, in <module>
    test()
  File "/tmp/test.py", line 12, in test
    unpickled = pickle.loads(pickled)
TypeError: an integer is required (got type bytes)

Side-comment: CPython error are a bit nicer when wrong arguments are passed to types.CodeType for example something like this:

TypeError: code() argument 7 must be bytes, not int

Having similar more informative error messages in nogil would be very useful!

[colesbury]

Thanks for the bug report. I'll look into this.

[colesbury]

I've made some changes to nogil Python better support cloudpickle and submitted a PR to cloudpickle:

cloudpipe/cloudpickle#470

You can use it immediately by running:

pip install cloudpickle==2.1.0dev0

The above will get the patched version of cloudpickle from the alternative Python package index.

I don't plan to change the signature of types.CodeType. That API isn't stable and changes in nearly every Python version.

I agree about the error messages, but I don't plan to change them either. "nogil" Python is using "argument clinic" for the constructor -- that's also what 3.10+ is using -- 3.9 used PyArg_ParseTuple. The error messages for argument clinic weren't as good in 3.9, but were improved in 3.10. I don't think it's worth back porting those changes.

[ogrisel]

Thanks @colesbury for the fix. I have integrated it upstream and it's now part of cloudpickle 2.1.0 so you should not need to maintain a forked version of cloudpickle with nogil specific fixes.

We can even work on adding a CI entry on cloudpickle to run its tests against the nogil branch if needed.

[colesbury]

@ogrisel - awesome! pip install cloudpickle picks up the latest version (2.1.0) from upstream. It works with the latest docker image and building from source (the pyenv install is a bit out of date).