Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deduplicate module file paths to prevent redundant scans. #7747

Merged
merged 20 commits into from Nov 18, 2022
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
9e345ba
Deduplicate module file paths to prevent redundant scans.
emcd Nov 11, 2022
cb1eaab
Merge branch 'PyCQA:main' into main
emcd Nov 12, 2022
0538ffd
Use dict for 'expand_modules' result rather than list.
emcd Nov 12, 2022
4bcbcc4
Update towncrier description for patch.
emcd Nov 12, 2022
387c240
Fix deduplication to account for CLI args marker. Add unit test for d…
emcd Nov 12, 2022
5a6d70b
Fix corner case with CLI arg flag handling during deduplication. Also…
emcd Nov 12, 2022
887eafb
Merge branch 'PyCQA:main' into main
emcd Nov 12, 2022
e2e326d
Factor out package module list for 'expand_modules' unit tests.
emcd Nov 13, 2022
08b9e1d
Add 'deduplication' to custom Pyenchant dict.
emcd Nov 13, 2022
2165ddf
Merge branch 'PyCQA:main' into main
emcd Nov 15, 2022
c59248b
Appease Mypy. Improve accuracy of comment.
emcd Nov 15, 2022
d0f66ef
Merge branch 'PyCQA:main' into main
emcd Nov 17, 2022
3d147e3
Integration Test: Pylint argument deduplication with highly constrain…
emcd Nov 17, 2022
dffd61d
Appease Pyenchant.
emcd Nov 17, 2022
e45a731
Use 'too-many-branches' functional test file as target for deduplicat…
emcd Nov 18, 2022
d641d69
Merge branch 'PyCQA:main' into main
emcd Nov 18, 2022
82948a0
Appease Pyenchant. Again.
emcd Nov 18, 2022
7a8fe98
Update tests/test_pylint_runners.py
Pierre-Sassoulas Nov 18, 2022
ae05856
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 18, 2022
c67f204
Update tests/test_pylint_runners.py
Pierre-Sassoulas Nov 18, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
5 changes: 5 additions & 0 deletions doc/whatsnew/fragments/6242.bugfix
@@ -0,0 +1,5 @@
Maintain and reference a set of visited files while expanding the module files
from list of module directories, package names, and standalone files to
prevent duplicate scans.

Closes #6242
emcd marked this conversation as resolved.
Show resolved Hide resolved
6 changes: 5 additions & 1 deletion pylint/lint/pylinter.py
Expand Up @@ -14,7 +14,7 @@
import traceback
import warnings
from collections import defaultdict
from collections.abc import Callable, Iterator, Sequence
from collections.abc import Callable, Iterator, MutableSet, Sequence
from io import TextIOWrapper
from pathlib import Path
from re import Pattern
Expand Down Expand Up @@ -878,8 +878,12 @@ def _iterate_file_descrs(

The returned generator yield one item for each Python module that should be linted.
"""
seen_filepaths: MutableSet[str] = set() # For deduplication of paths.
for descr in self._expand_files(files_or_modules):
name, filepath, is_arg = descr["name"], descr["path"], descr["isarg"]
if filepath in seen_filepaths:
emcd marked this conversation as resolved.
Show resolved Hide resolved
continue
seen_filepaths.add(filepath)
if self.should_analyze_file(name, filepath, is_argument=is_arg):
yield FileItem(name, filepath, descr["basename"])

Expand Down