Autodetect number of jobs with --jobs auto #605
Conversation
| @@ -9,6 +10,20 @@ | |||
|
|
|||
| _ARG_PREFIX = '--' | |||
|
|
|||
| def _auto_detect_cpus(): | |||
| try: | |||
| return len(os.sched_getaffinity(0)) | |||
There was a problem hiding this comment.
This return the number of CPUs the current process is restricted to on Linux.
There was a problem hiding this comment.
What's wrong with os.cpu_count()?
There was a problem hiding this comment.
On Linux len(os.sched_getaffinity(0)) returns the number of CPUs available to the current process, where as os.cpu_count() will return the total number of CPUs on the host system which might be more than the current process can use.
There was a problem hiding this comment.
answering my question - it's the number of available cores (not clear whether this includes hyperthreading cores).
More details: https://stackoverflow.com/questions/1006289/how-to-find-out-the-number-of-cpus-using-python/14840102
There was a problem hiding this comment.
not clear whether this includes hyperthreading cores
As far as I know this includes hyperthreading cores. On my dual-core macbook --jobs auto will result in 4 parallel jobs.
Fixed in 4e48291 |
This adds the possibility to autodetect the number of jobs based on the number of available CPU cores on the host system using
pytype --jobs auto. This allows for a usage similar topytest-xdistwhich can be useful when usingpytypein scripts that might run on different systems.