Semaphore is not implemented.

[ghost]

Relates to #388

Operating system: Android 8.0.0 (OnePlus 3T, Termux)
Python version: 3.6.6 3.6.5
Black version: 18.6b4
Does also happen on master: yes

It has come to my attention that there is no semaphore synchronization primitive on Android. This means that black will not work on this platform whatsoever, which is disappointing, especially as I find this to be a fairly useful tool that I use regularly. I cannot get the traceback off of my phone while at work unfortunately, but I believe this issue has been put in before. It occurs on line 341 of black.py when initialising a ProcessPool. In master, this is at https://github.com/ambv/black/blob/master/black.py#L343

I have tested this in the Python console with both the multiprocessing pool and the concurrent.futures process pool. Both complain about said missing synchronization primitives.

Python issue #3770 discusses this issue, which appears to not just occur on Android, but also on the BSDs. This would mean most likely (haven't tested as I don't use BSD) that both Android and BSD cannot support Black.

Whilst this is not ideal, the following could be a work around for ensuring that this package would actually still work on these platforms. Again, I haven't tested, but both TPE and PPE in concurrent.futures share the same interface.

import concurrent.futures
...
def generate_pool(*args, **kwargs) -> concurrent.futures.Executor:
    try:
        return concurrent.futures.ProcessPoolExecutor(*args, **kwargs)
    except ImportError:
        return concurrent.futures.ThreadPoolExecutor(*args, **kwargs)
...
# 344
executor = generate_pool(max_workers=os.cpu_count())

Alternatively, could we just run the tasks in series if this issue is detected, rather than using a pool to perform multiple things at once? As much as it is an inconvenience for things to run slowly or inefficiently, I would much prefer that and be able to use this tool at least somewhat on this platform.

Would this be worth considering?

Cheers 😄

[JelleZijlstra]

I think it makes sense to fall back to processing files serially when there is no working process pool implementation. I'd prefer not to use a thread pool, because it is unlikely to provide a speedup over serial execution.

[ghost]

My logic behind a TPE was that it requires less modification to work. Also it will likely still provide some speedup in environments that dont have the constraint of a GIL like CPython does. But yeah, I agree either way. As long as it works at least somewhat; anything is better than nothing ;) Get Outlook for Android<https://aka.ms/ghei36>

…

________________________________ From: Jelle Zijlstra <notifications@github.com> Sent: Friday, September 21, 2018 4:31:39 PM To: ambv/black Cc: Ashley; Author Subject: Re: [ambv/black] Semaphore is not implemented. (#524) I think it makes sense to fall back to processing files serially when there is no working process pool implementation. I'd prefer not to use a thread pool, because it is unlikely to provide a speedup over serial execution. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#524 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ApCo0A6ZqfTYcaqgcxN2t2HtbXCLYhj-ks5udQZagaJpZM4WzmLw>.

[ambv]

Thanks for your report! This is a big surprise to me that there are attempts to run Black on Android :-)

Also it will likely still provide some speedup in environments that dont have the constraint of a GIL like CPython does.

There are currently no alternative runtimes implementing Python 3.6+ which don't have a GIL.

I think we could probably easily provide a SerialExecutor implementation which, while not very fast, would solve the compatibility problem. I would want to avoid TPE because Black was not designed with careful thread-safety in mind. It should be fine as we isolate the state to function arguments but we might be missing something.

By the way, do you have a way to test if your TPE suggestion makes Black run?

[ghost]

I know for sure that it is the process pool that causes the issue, from projects of my own. If I remember tomorrow then I will have a look at putting a PR together that tests it out though, and put that in if it solves the issue? Get Outlook for Android<https://aka.ms/ghei36>

…

________________________________ From: Łukasz Langa <notifications@github.com> Sent: Tuesday, September 25, 2018 7:51:34 PM To: ambv/black Cc: Ashley; Author Subject: Re: [ambv/black] Semaphore is not implemented. (#524) Thanks for your report! This is a big surprise to me that there are attempts to run Black on Android :-) Also it will likely still provide some speedup in environments that dont have the constraint of a GIL like CPython does. There are currently no alternative runtimes implementing Python 3.6+ which don't have a GIL. I think we could probably easily provide a SerialExecutor implementation which, while not very fast, would solve the compatibility problem. I would want to avoid TPE because Black was not designed with careful thread-safety in mind. It should be fine as we isolate the state to function arguments but we might be missing something. By the way, do you have a way to test if your TPE suggestion makes Black run? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#524 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ApCo0Eu0eNxqhYYuK5v51A6RAI9utYnpks5uens2gaJpZM4WzmLw>.

[ghost]

See #533

Fixes the issue for me :)

[aureliojargas]

More than a year after the original report, I have tried in a newer environment:

• Android 10 (app Termux)
• Device: Google Pixel 2 XL
• Python 3.7.4
• Black 19.3b

I confirm that the issue still persists.

I confirm that @ghost patch still works! 🎉

[waynew]

Same here - but just a regular Google Pixel.

Some of us are crazy about writing code in strange places where our laptops just don't go 😄

[MatthewScholefield]

Btw @JelleZijlstra we can close this ticket now; idk when but the fix for this has been merged into master a while ago (and also of course recently in #2631 a fix for the fix was merged).