gh-103134: Update multiprocessing.managers.ListProxy and multiprocessing.managers.DictProxy

[invisibleroads]

from multiprocessing import Manager
with Manager() as manager:
    xs = manager.list()
    xs.clear()

For now, we can use the workaround del xs[:]

• Issue: Update multiprocessing.managers.ListProxy and multiprocessing.managers.DictProxy #103134

[bedevere-bot]

Most changes to Python require a NEWS entry.

Please add it using the blurb_it web app or the blurb command-line tool.

[cpython-cla-bot]

All commit authors signed the Contributor License Agreement.

[terryjreedy]

Are there any tests for BaseList Proxy or derivatives that are not automatically expanded by expanding this list? And which should be manually augmented?

[terryjreedy]

Given the definition of method __iadd__ in the definition of ListProxy below, it would seem that it should be in the list above. On the other hand, adding xs += [2] to your example works as is. Maybe '_imul' does not need to be listed. Does it hurt to list 'iadd'?

[invisibleroads]

It looks like ListProxy.__iadd__ uses BaseListProxy.extend and ListProxy.__imul__ calls BaseListProxy.__imul__. I'm not sure of the reasoning behind why ListProxy defines __iadd__ and __imul__ separately instead of including them directly in BaseListProxy. Testing xs += [2] and xs *= 2 works when tested manually.

I'm afraid of replacing BaseListProxy with ListProxy for fear of breaking something. Looking at git blame, it looks like this part of the code was last changed 15 years ago? @benjaminp would it be safe to replace BaseListProxy with ListProxy and just include __iadd__ and __imul__?

[terryjreedy]

As mentioned on issue, DictProxy and maybe others need updating.

[invisibleroads]

As mentioned on issue, DictProxy and maybe others need updating.

Added a few more bits to DictProxy to support the cases you mentioned, including fromkeys, d | {}, {} | d, reversed(d).

After your comments, I'm tempted to replace BaseListProxy with ListProxy but would like to hear back from @benjaminp if @benjaminp remembers why BaseListProxy was defined separately.

[invisibleroads]

As an update, I did try removing BaseListProxy and just adding __iadd__ and __imul__ directly to ListProxy and it didn't work. I'm not sure why. Here is the code that I used to test this functionality manually.

from multiprocessing import Manager, Process

def f(x):
    x += [1]
    x *= 2
    print(x)

def g(d):
    d |= {'b': 2}
    print(d)

with Manager() as manager:

    x = manager.list()
    x.append(0)
    p = Process(target=f, args=(x,))
    p.start()
    p.join()
    q = Process(target=print, args=(x,))
    q.start()
    q.join()

    d = manager.dict()
    d['a'] = 1
    p = Process(target=g, args=(d,))
    p.start()
    p.join()
    q = Process(target=print, args=(d,))
    q.start()
    q.join()

RESULTS

ACTUAL after replacing BaseListProxy with ListProxy (the __imul__ doesn't seem to work).
ACTUAL when DictProxy is missing __ior__

[0, 1, 0, 1]
[0, 1]
{'a': 1, 'b': 2}
{'a': 1}

EXPECTED and ACTUAL with current commits for this pull request -- works with this pull request

[0, 1, 0, 1]
[0, 1, 0, 1]
{'a': 1, 'b': 2}
{'a': 1, 'b': 2}

The pull request passes the above manual test.

[invisibleroads]

We could write a test that uses a process to modify a manager.list or manager.list, then check that the list or dict has actually been modified.

I found the multiprocessing tests here
https://github.com/python/cpython/blob/main/Lib/test/_test_multiprocessing.py#L5703

will try to add tests over the weekend

[invisibleroads]

This pull request is ready for review. Thanks to @arhadthedev for accepting this issue, @terryjreedy for the suggestions and @sunmy2019 for the very important hints.

[invisibleroads]

@Yhg1s
Is there any chance we could get these multiprocessing manager updates into Python 3.12?
The code changes are relatively straightforward and add missing functionality to multiprocessing.managers.ListProxy and multiprocessing.managers.DictProxy.
Thanks for managing these releases.

[invisibleroads]

This minor enhancement backports recently added list and dictionary functionality into multiprocessing.managers.ListProxy and multiprocessing.managers.DictProxy so that scripts that use multiprocessing.Manager can use the same list and dictionary functionality. Would it be possible for someone to review these proposed enhancements?

@kumaraditya303
@serhiy-storchaka
@pablogsal
@iritkatriel
@gvanrossum

[serhiy-storchaka]

Please test the type and the value of results.

[invisibleroads]

It's been almost a year, but I finally got around to making the changes requested by @serhiy-storchaka.

Would it be possible for someone to review these changes?
@pablogsal
@iritkatriel
@gvanrossum
@kumaraditya303

[invisibleroads]

Thanks to @encukou for taking the time to review this during the PyCon 2024 sprints!!