Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Autodetect number of jobs with --jobs auto #605

Closed
wants to merge 2 commits into from
Closed

Autodetect number of jobs with --jobs auto #605

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
28fe284
Autodetect number of jobs with --jobs auto
lgeiger Jun 20, 2020
4e48291
Style fixes
lgeiger Jun 22, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
3 changes: 2 additions & 1 deletion pytype/tools/analyze_project/config.py
View file
Open in desktop
Expand Up @@ -48,7 +48,8 @@
'Keep going past errors to analyze as many files as possible.'),
'jobs': Item(
1, '4', None,
'Run N jobs in parallel.'),
"Run N jobs in parallel. When 'auto' is used, this will be equivalent "
'to the number of CPUs on the host system.'),
'output': Item(
'.pytype', '.pytype', None, 'All pytype output goes here.'),
'pythonpath': Item(
Expand Down
18 changes: 17 additions & 1 deletion pytype/tools/analyze_project/parse_args.py
View file
Open in desktop
@@ -1,6 +1,7 @@
"""Argument parsing for analyze_project."""

import argparse
import os

from pytype import config as pytype_config
from pytype.tools import arg_parser
Expand All @@ -9,6 +10,20 @@

_ARG_PREFIX = '--'

def _auto_detect_cpus():
lgeiger marked this conversation as resolved.
Show resolved Hide resolved
try:
return len(os.sched_getaffinity(0))
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This return the number of CPUs the current process is restricted to on Linux.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's wrong with os.cpu_count()?

Copy link
Contributor Author

@lgeiger lgeiger Jun 20, 2020 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On Linux len(os.sched_getaffinity(0)) returns the number of CPUs available to the current process, where as os.cpu_count() will return the total number of CPUs on the host system which might be more than the current process can use.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

answering my question - it's the number of available cores (not clear whether this includes hyperthreading cores).
More details: https://stackoverflow.com/questions/1006289/how-to-find-out-the-number-of-cpus-using-python/14840102

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not clear whether this includes hyperthreading cores

As far as I know this includes hyperthreading cores. On my dual-core macbook --jobs auto will result in 4 parallel jobs.

except AttributeError:
return os.cpu_count()


def parse_jobs(s):
"""Parse the --jobs option"""
lgeiger marked this conversation as resolved.
Show resolved Hide resolved
if s == 'auto':
n = _auto_detect_cpus()
return n if n else 1
elif s is not None:
return int(s)
lgeiger marked this conversation as resolved.
Show resolved Hide resolved

class Parser(arg_parser.Parser):
"""Subclasses Parser to add a config file processor."""
Expand Down Expand Up @@ -93,7 +108,8 @@ def make_parser():
(('-x', '--exclude'), {'nargs': '*', 'action': 'flatten'}),
(('inputs',), {'metavar': 'input', 'nargs': '*', 'action': 'flatten'}),
(('-k', '--keep-going'), {'action': 'store_true', 'type': None}),
(('-j', '--jobs'), {'action': 'store', 'type': int, 'metavar': 'N'}),
(('-j', '--jobs'), {'action': 'store', 'type': parse_jobs,
'metavar': 'N'}),
(('-P', '--pythonpath'),),
(('-V', '--python-version'),)
]:
Expand Down