Azure import try/except over doubles pylint memory usage #9442
Comments
Jamie-
added
the
Needs triage 📥
jacobtylerwalls
added
performance
Needs astroid update
|
Thanks for the report! Sounds reasonable to expect I'm showing this is solved with two tiny patches to pylint and astroid. If you're up for it, would you be willing to confirm and open the PRs? We'd need just a little test and some documentation spruce-ups for the ignored-modules option. diff --git a/pylint/lint/pylinter.py b/pylint/lint/pylinter.py
index 30250154e..473cd0900 100644
--- a/pylint/lint/pylinter.py
+++ b/pylint/lint/pylinter.py
@@ -1073,6 +1073,7 @@ class PyLinter(
MANAGER.always_load_extensions = self.config.unsafe_load_any_extension
MANAGER.max_inferable_values = self.config.limit_inference_results
MANAGER.extension_package_whitelist.update(self.config.extension_pkg_allow_list)
+ MANAGER.module_denylist.extend(self.config.ignored_modules)
if self.config.extension_pkg_whitelist:
MANAGER.extension_package_whitelist.update(
self.config.extension_pkg_whitelistdiff --git a/astroid/manager.py b/astroid/manager.py
index c499fe55..386a3838 100644
--- a/astroid/manager.py
+++ b/astroid/manager.py
@@ -59,6 +59,7 @@ class AstroidManager:
"optimize_ast": False,
"max_inferable_values": 100,
"extension_package_whitelist": set(),
+ "module_denylist": [],
"_transform": TransformVisitor(),
}
@@ -70,6 +71,7 @@ class AstroidManager:
self.extension_package_whitelist = AstroidManager.brain[
"extension_package_whitelist"
]
+ self.module_denylist = AstroidManager.brain["module_denylist"]
self._transform = AstroidManager.brain["_transform"]
@property
@@ -200,6 +203,8 @@ class AstroidManager:
# importing a module with the same name as the file that is importing
# we want to fallback on the import system to make sure we get the correct
# module.
+ if modname in self.module_denylist:
+ raise AstroidImportError("Skipping")
if modname in self.astroid_cache and use_cache:
return self.astroid_cache[modname]
if modname == "__main__": |
|
Aha ok. I incorrectly assumed ignored-modules did that already. From what I'm seeing, those changes make no effect to the speed or memory usage when adding Any pointers? |
|
Make sure you're using the full module name, e.g. |
|
Yep, that helps! Seeing a considerable improvement now. Thanks for that! |
|
Wonderful. Feel free to make improvements (e.g. for consistency, that collection should probably be a set...) |
Bug description
We run pylint on a reasonably large codebase, ~900 .py files, ~350k lines.
Recently we've needed to support two similar versions of the Azure SDK and as such made the change below in a single file.
from
to
This change has increased pylint's run time but more importantly, over doubled resident memory usage! I've tried various combinations of disables and module ignores but cannot get anywhere close to previous figures without reverting the code change. Including attempting to ignore the azure module altogether with
--ignored-modules=azureNumbers below are taken from
/usr/bin/time -v python3 -m pylint package1 package2 -f colorized -r n -j 1Old code:
New code:
New code with
--ignored-modules=azure:To make this more generic, I'm running without a pylintrc file using the latest version from PyPI. This has the side effect of producing masses of warning/error output as usually we have a fair number of disables, however the issue described is still present in this state.
Configuration
No response
Command used
Pylint output
n/a
Expected behavior
Original run times and memory usage
Pylint version
OS / Environment
Ubuntu 22.04
Additional dependencies
The text was updated successfully, but these errors were encountered: