python 3.11.0b1 changes codeobject args

[mmckerns]

Python 3.11.0a7 (main, Apr  9 2022, 09:43:27) [Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> c = (lambda x:x).__code__
>>> c.co_
c.co_argcount        c.co_firstlineno     c.co_names          
c.co_cellvars        c.co_flags           c.co_nlocals        
c.co_code            c.co_freevars        c.co_positions(     
c.co_columntable     c.co_kwonlyargcount  c.co_posonlyargcount
c.co_consts          c.co_lines(          c.co_qualname       
c.co_endlinetable    c.co_linetable       c.co_stacksize      
c.co_exceptiontable  c.co_lnotab          c.co_varnames       
c.co_filename        c.co_name           

in 3.11.0b1: c.co_endlinetable is renamed c.co_linetable, and c.co_columntable seems to be removed. Potentially more. This will impact the already condition-heavy _create_code. Potentially should ignore support for 3.11.0a1 - 3.11.0a7... however, it's implicitly in released dill-0.3.5.1.

[mmckerns]

I'd generally ignore alpha release behavior if it changes in the beta, and 3.11 is not yet explicitly supported by dill. However, dill-0.3.5.1 was just released with the alpha behavior. So... should the alpha behavior be supported going forward (for people who created 3.11.0a* pickles and want to restore them)? Should it be supported only if it's easy? I can't imagine there are a lot of people that should expect their not-explicitly 3.11.0a*-series pickles should be supported going forward. Probably best to deal with this, and maybe drop 2.7 support, and then do a quick new release. @matsjoyce, @anivegesana: Thoughts?

[anivegesana]

Roping @leogama into this too since he was looking into the code object and the confusing nature of its save/create code. Unfortunately, this means that code.replace is not as stable as we once thought, which sucks. An alternative is to introduce a namedtuple which we can change based on Python version. I need to look into it, but that is a very strange change. It is possible that the semantics of those attributes changed. I know CPython has been trying to more closely integrate the C and Python stacks to boost performance, and it could be a consequence of that.

[anivegesana]

But yes, I agree that alpha builds don't need support, only the releases and nightly.

[mmckerns]

Looks like they decided in 3.11.0a* to rename co_linetable to co_endlinetable... then reverted it in 3.11.0b1. It looks like the other changes, like co_columntable, are to try to deal with the better error markup in 3.11... and although I didn't track down why exactly the new attributes went away.

Here's some data to consider for whether the 3.11 alpha behavior needs to be supported:
https://pypistats.org/packages/dill

3.11 has at most ~50 dill users per day, and is less than 10 starting a month ago. With the release of 0.3.5.1 that may have a significant increase, because to use dill with 3.11 previously, you'd have to use GitHub master.

I'm thinking that the best course of action is to convert as soon as possible to the beta behavior, drop the alpha behavior, and release again soon afterward (e.g. after also dropping 2.7).

[leogama]

Roping @leogama into this too since he was looking into the code object and the confusing nature of its save/create code. Unfortunately, this means that code.replace is not as stable as we once thought, which sucks.

This is bad, but not super bad. The only use for code.replace() I have in sight is for bootstrapping _create_code(). For this task, it's not necessary to pass all the arguments to replace. Mainly these new ones, which seem to be related to exceptions and error messages(?), can be ignored. The bootstrapped "create_code" just need to be called once to construct the actual one.

I did a quick test in Python 3.11a defining a function f that do some stuff and raises an exception at the end, and an empty function g (def g(): pass). Then defined a function h using FunctionType(g.__code__.replace(**some_f_code_atts), globals()) (just some code attributes common from Py3.8 to Py3.11) and called it. Worked fine, only the error message came indicating error in "line -1". At most you wouldn't be able to run it in a debugger...

[anivegesana]

Trust me. You definitely want to make sure that every case is accounted for and that nothing unpredicted (like the -1 line number thing) happens, or people will notice and it will come back to bite (#478). I am actually planning on reading over the Python data model to have a mental model of what is in the Python interpreter. Maybe the Code objects section of it can shed some light on how to fix the -1 issue you are having.

https://docs.python.org/3/reference/datamodel.html

[mmckerns]

Specifically, the change is from this (in 3.11.0a7):

class code(object)
 |  code(argcount, posonlyargcount, kwonlyargcount, nlocals, stacksize, flags, codestring, constants, names, varnames, filename, name, qualname, firstlineno, linetable, endlinetable, columntable, exceptiontable, freevars=(), cellvars=(), /)

to this (in 3.11.0b1):

class code(object)
 |  code(argcount, posonlyargcount, kwonlyargcount, nlocals, stacksize, flags, codestring, constants, names, varnames, filename, name, qualname, firstlineno, linetable, exceptiontable, freevars=(), cellvars=(), /)

So... it appears that two arguments have been dropped, and not one being renamed.

Python 3.11.0b1 (main, May 10 2022, 09:36:04) [Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> obj = (lambda x:x).__code__
>>> obj.co_
obj.co_argcount        obj.co_flags           obj.co_names          
obj.co_cellvars        obj.co_freevars        obj.co_nlocals        
obj.co_code            obj.co_kwonlyargcount  obj.co_positions(     
obj.co_consts          obj.co_lines(          obj.co_posonlyargcount
obj.co_exceptiontable  obj.co_linetable       obj.co_qualname       
obj.co_filename        obj.co_lnotab          obj.co_stacksize      
obj.co_firstlineno     obj.co_name            obj.co_varnames

[leogama]

Note that co_positions is a new method: https://docs.python.org/3.11/reference/datamodel.html#codeobject.co_positions

Edit: and co_lines seems to be an old, undocumented one.

[anivegesana]

I think we desperately need to should change over to using dictionaries to store code attributes. Or otherwise, we cannot tell what attributes disappeared.

[mmckerns]

that's probably a good idea...

[anivegesana]

I guess I was wrong about not being able to tell what was present in the code object, but using the actual attribute names would increase readability.

[mmckerns]

again... you are probably right. I, however, was just finishing up the testing for it as is

[anivegesana]

I agree that it is the best idea for now. Sometimes it is best to follow the adage "if it ain't broke, don't fix it."

[mmckerns]

Oh, BTW, when fixing this, I also fixed a bug (it's in 0.3.5.1) that will show up in the wild from people converting pickled objects between 3.11.0a* and earlier versions of code objects that don't have co_exceptiontable. Rare case, but still there. I'd like to get this fixed code released soon, as I assume people will start trying to use dill for 3.11 using 0.3.5.1. So, there might be a case for 0.3.5.2, coming in June (and getting dill back on the regular six month schedule). I would not expect 3.11.0 to change too much between b1 and final (2022-10-03). The change from 3.11.0a7 to 3.11.0b1 broke multiprocess (due to a breaking change in at least subprocess and pickle), so I'm sure there are other breaking changes, as b1 for 3.11 is the feature freeze.