Add BSD support to CPU count check

[outpaddling]

--- joblib/externals/loky/backend/context.py.orig       2023-12-29 13:45:11 UTC
+++ joblib/externals/loky/backend/context.py
@@ -274,6 +274,15 @@ def _count_physical_cores():
             )
             cpu_info = cpu_info.stdout
             cpu_count_physical = int(cpu_info)
+        # Maybe also openbsd, dragonfly, etc?
+        elif sys.platform.startswith('freebsd') or sys.platform.startswith('netbsd'):
+            cpu_info = subprocess.run(
+                "sysctl -n hw.ncpu".split(),
+                capture_output=True,
+                text=True,
+            )
+            cpu_info = cpu_info.stdout
+            cpu_count_physical = int(cpu_info)
         else:
             raise NotImplementedError(f"unsupported platform: {sys.platform}")

[outpaddling]

Or maybe use os.sysconf(NPROCESSORS_CONF), as suggested by another pkgsrc developer.

[GaelVaroquaux]

Thank you for bringing this up. It would indeed be very useful to have better support of BSD systems. I am afraid that none of the core devs of joblib run BSD systems.

The os.sysconf feels like a better route to me, as a call to subprocess.run will have a lot of overhead. I do wonder how it will behave with regards to physical versus logical cores for processors with hyperthreading. I also wonder if it will handle well the case of programs running in jails, to avoid oversubscription in these settings. Under Linux, we have found that these details mattered to have efficient computing.

What would be really great is if you could make the appropriate changes on a branch, check that they work fine, and do a pull request. As we do not run BSD systems, we cannot test the changes, and I would not feel comfortable doing them blindly.

Thanks!!

[outpaddling]

I can test on FreeBSD, macOS, and NetBSD. I'll leave the Linux case alone.

Running "getconf NPROCESSORS_CONF", which I believe is the shell equivalent so os.sysconf() (both are just interfaces to sysconf(3)), does indeed return 4 on my 2-core i5 running FreeBSD and my 2-core i5 MacBook.

Maybe the python sysctl package would be a better route for BSDs (including macOS). We should be able to get the same information as subprocess.run("sysctl -n hw.physicalcpu".split(), capture_output=True, text=True) without spawning a subprocess. I'll explore this and get back to you.

[GaelVaroquaux]

Maybe the python sysctl package would be a better route for BSDs (including macOS). We should be able to get the same information as subprocess.run("sysctl -n hw.physicalcpu".split(), capture_output=True, text=True) without spawning a subprocess. I'll explore this and get back to you.

Thanks a lot for looking this up. You are right that macOS is a place to experiment. I don't have one around but when I'm in the office I can ask colleagues. Not spawning a subprocess will probably be a big gain. And thanks for the pull request!

[outpaddling]

I think we'll have to stick with subprocesses. Apparently there is no portable sysctl interface for python. The sysctl on PYPI is FreeBSD-specific, developed by the TrueNAS project. I found other projects for Linux and NetBSD with completely different APIs. I'll do a pull request with code for FreeBSD and NetBSD, adapted from the existing Darwin case.