GIL Refs API Removal - Tracking Issue

[davidhewitt]

The main steps remaining to remove GIL Refs:

For 0.22

In 0.22 the gil-refs feature will continue to be available, but with two major differences from 0.21. The deprecation warnings will be unconditional and the APIs will be gated behind the feature.

• Upgrade all of the deprecated APIs to be gated behind the gil-refs feature:

  #[cfg_attr(
       not(feature = "gil-refs"),
       deprecated(
           since = "0.21.0",
           note = "use `Py::from_owned_ptr(py, ptr)` or `Bound::from_owned_ptr(py, ptr)` instead"
       )
   )]

  To

  #[cfg(feature = "gil-refs")]
  #[deprecated(
       since = "0.21.0",
       note = "use `Py::from_owned_ptr(py, ptr)` or `Bound::from_owned_ptr(py, ptr)` instead"
   )]

  • buffer.rs feature gate deprecated APIs for PyBuffer #4144
  • conversion.rs feature gate deprecated APIs for Python #4173
  • conversions/num_complex.rs feature gate deprecated APIs for PyFloat and PyComplex #4145
  • err/mod.rs feature gate deprecated APIs for PyErr and exceptions #4136
  • exceptions.rs feature gate deprecated APIs for PyErr and exceptions #4136
  • gil.rs
  • impl_/deprecations.rs feature gate APIs using into_gil_ref (Part 2) #4166
  • instance.rs feature gate deprecated APIs for Py #4142
  • marker.rs feature gate APIs using into_gil_ref (Part 1) #4160 feature gate APIs using into_gil_ref (Part 2) #4166 feature gate deprecated more APIs for Py #4169 feature gate as/into_gil_ref APIs (Part 3) #4172 feature gate deprecated APIs for Python #4173
  • marshal.rs feature gate deprecated APIs for marshal #4149
  • pycell.rs split more impl blocks (preparation of PyCell feature gate) #4175 feature gate PyCell #4177
  • type_object.rs feature gate deprecated APIs for PyType, PyTypeInfo and PySuper #4134
  • types/any.rs (Tests) port PyAny tests to Bound API #4140
  • types/boolobject.rs feature gate deprecated APIs for PyBool #4159
  • types/bytearray.rs feature gate deprecated APIs for PyBytes and PyPyByteArray #4131
  • types/bytes.rs feature gate deprecated APIs for PyBytes and PyPyByteArray #4131
  • types/capsule.rs feature gate deprecated APIs for PyCapsule #4112
  • types/complex.rs feature gate deprecated APIs for PyFloat and PyComplex #4145
  • types/datetime.rs feature gate deprecated APIs for datetime types #4111
  • types/dict.rs feature gate deprecated APIs for PyDict #4108
  • types/ellipsis.rs feature gate deprecated APIs for PyEllipsis, PyNone and PyNotImplemented #4132
  • types/float.rs feature gate deprecated APIs for PyFloat and PyComplex #4145
  • types/frozenset.rs feature gate deprecated APIs for PyFrozenSet #4118
  • types/function.rs feature gate deprecated APIs for PyCFunction #4154
  • types/iterators.rs feature gate deprecated APIs for PyIterator #4119
  • types/list.rs feature gate deprecated APIs for PyList #4127
  • types/memoryview.rs feature gate deprecated APIs for PyMemoryView #4152
  • types/module.rs feature gate deprecated APIs for PyModule #4151
  • types/none.rs feature gate deprecated APIs for PyEllipsis, PyNone and PyNotImplemented #4132
  • types/notimplemented.rs feature gate deprecated APIs for PyEllipsis, PyNone and PyNotImplemented #4132
  • types/pysuper.rs feature gate deprecated APIs for PyType, PyTypeInfo and PySuper #4134
  • types/sequence.rs (Tests) port PySequence tests to Bound API #4139
  • types/set.rs feature gate deprecated APIs for PySet #4096
  • types/slice.rs feature gate deprecated APIs for PySlice #4141
  • types/string.rs feature gate deprecated APIs for PyString #4101
  • types/tuple.rs feature gate deprecated APIs for PyTuple #4107
  • types/typeobject.rs feature gate deprecated APIs for PyType, PyTypeInfo and PySuper #4134

• Remove default implementation of FromPyObject::extract_bound (to force users to implement it). feature gate as/into_gil_ref APIs (Part 3) #4172

• To get 0.21 out the door I left several test suites with #[allow(deprecated)] or #[cfg_attr(not(feature = "gil-refs"), allow(deprecated)]. We'll need to update these. I like to keep coverage high but I think for sake of practicality I'm ok if we migrate all the tests to the bound API and trust the GIL Refs API is correct by virtue of being implemented on top of the Bound API.

• Update documentation on Python type

For 0.23

• Mass deletion of deprecated APIs🚀
• For _bound variants like PyTuple::new_bound:
  • Add plain variants like PyTuple::new again.
  • Add deprecation markers on PyTuple::new_bound to nudge users to the plain versions again.

---

Optional polishing tasks from #3684

• Split PyModuleMethods to move functionality related to creating / building new modules into a PyModuleBuilder? implement PyModuleMethods #3703 (comment)
• Generic IntoPy<Py<T>> for Bound<'_, T>? implement PyTupleMethods #3681 (comment)
• Should Borrowed<'py, 'py, T>::into_gil_ref become public API as well?
• FromPyObject could gain a lifetime 'a to its argument &'a Bound<'py>, allowing:
  • downcasting the Bound to allow impl FromPyObject for &Bound
  • impl FromPyObject<'a, '_> for &'a str without the pool; as the lifetime of the string borrow depends on the lifetime of the input, not the GIL lifetime
  • would allow PyRef and PyRefMut to store &'a Bound<'py, T> (or borrowed) and avoid reference count overhead
• Update py.None(), py.Ellipsis() and py.NotImplemented() to return Borrowed<'py, 'py, PyNone> (etc.)
• It would be nice to accept #[pyo3(from_py_with = "PyAnyMethods::len")] (and similar)

[alex]

Question: What do we need to do with PyClass, which currently has the following definition: pub trait PyClass: PyTypeInfo<AsRefTarget = crate::PyCell<Self>> + PyClassImpl { ... }.

Except PyCell is deprecated!

[davidhewitt]

I think AsRefTarget can also be conditional as that's a GIL Ref concept which needs to be removed.

It might mean it's necessary to have an extra level of indirection e.g. pub trait PyClass: PyClassConstraints where PyClassConstraints is conditional.

[alex]

Pain point here: https://github.com/PyO3/pyo3/blob/main/pyo3-macros-backend/src/pyclass.rs#L1281-L1285

This unconditionally generates emits this, and currently gil-refs is not wired into pyo3-macros-backend.

[Icxolu]

Another pain point is as_gil_ref and especially into_gil_ref which are used a lot in the gil-ref inherent methods now. Maybe we need to split all these impl blocks and feature gate all these inherent methods, except for the new _bound constructors.