New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Create a framework of functional tests for configuration files #5287
Merged
Pierre-Sassoulas
merged 23 commits into
main
from
add-functional-tests-for-configuration
Nov 12, 2021
Merged
Changes from all commits
Commits
Show all changes
23 commits
Select commit
Hold shift + click to select a range
6626c23
Create a framework of functional tests for configuration files
Pierre-Sassoulas 88d08e2
Better handling of removal in functional conf tests
Pierre-Sassoulas a450796
Simplify a list creation in OptionParser()
Pierre-Sassoulas a26e356
Easier understanding of what happen for the expected result
Pierre-Sassoulas a8e61d3
Apply suggestions from code review
Pierre-Sassoulas 476ce6b
Update pylint/testutils/configuration_test.py
Pierre-Sassoulas 2146e3d
Move test_can_read_toml_rich_types to functional framework
Pierre-Sassoulas 0676bfb
Factorize duplicated code from test_config using the new testutil
Pierre-Sassoulas e9c89f4
More precise typing for Configuration type when Any is used
Pierre-Sassoulas 38b7c41
Create a constant for accepted configuration extension
Pierre-Sassoulas 58a1d66
Simplification of for loop for functional conf test following review
Pierre-Sassoulas 2a98bb0
Better explanation on why we use logging following review
Pierre-Sassoulas 774c373
Add a docstring to get_expected_or_default following review
Pierre-Sassoulas 47dae6a
Simplification of for loop on dict by using items()
Pierre-Sassoulas 388a2db
Rename variable that was still named for toml only
Pierre-Sassoulas b1f3b41
Add a regression test for #4746
Pierre-Sassoulas 59e119d
Apply suggestions from code review
Pierre-Sassoulas a08e38e
Add proper typing in configuration test following review
Pierre-Sassoulas 7fac09b
Remove option that are not required for the test itself
Pierre-Sassoulas 4a50fa5
Proper import for python import of CaptureFixture
Pierre-Sassoulas dddaf0d
Use the existing function for relative path instead of a custom one
Pierre-Sassoulas 10135bb
Simplification in OptionManagerMixin following review
Pierre-Sassoulas a4a5a3a
Update pylint/config/option_manager_mixin.py
Pierre-Sassoulas File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,110 @@ | ||
# Licensed under the GPL: https://www.gnu.org/licenses/old-licenses/gpl-2.0.html | ||
# For details: https://github.com/PyCQA/pylint/blob/main/LICENSE | ||
|
||
"""Utility functions for configuration testing.""" | ||
import copy | ||
import json | ||
import logging | ||
import unittest | ||
from pathlib import Path | ||
from typing import Any, Dict, Tuple, Union | ||
from unittest.mock import Mock | ||
|
||
from pylint.lint import Run | ||
|
||
USER_SPECIFIC_PATH = Path(__file__).parent.parent.parent | ||
# We use Any in this typing because the configuration contains real objects and constants | ||
# that could be a lot of things. | ||
ConfigurationValue = Any | ||
PylintConfiguration = Dict[str, ConfigurationValue] | ||
|
||
|
||
def get_expected_or_default( | ||
tested_configuration_file: str, suffix: str, default: ConfigurationValue | ||
) -> str: | ||
"""Return the expected value from the file if it exists, or the given default.""" | ||
|
||
def get_path_according_to_suffix() -> Path: | ||
path = Path(tested_configuration_file) | ||
return path.parent / f"{path.stem}.{suffix}" | ||
|
||
expected = default | ||
expected_result_path = get_path_according_to_suffix() | ||
if expected_result_path.exists(): | ||
with open(expected_result_path, encoding="utf8") as f: | ||
expected = f.read() | ||
# logging is helpful to realize your file is not taken into | ||
# account after a misspell of the file name. The output of the | ||
# program is checked during the test so printing messes with the result. | ||
logging.info("%s exists.", expected_result_path) | ||
DanielNoord marked this conversation as resolved.
Show resolved
Hide resolved
|
||
else: | ||
logging.info("%s not found, using '%s'.", expected_result_path, default) | ||
return expected | ||
|
||
|
||
EXPECTED_CONF_APPEND_KEY = "functional_append" | ||
EXPECTED_CONF_REMOVE_KEY = "functional_remove" | ||
|
||
|
||
def get_expected_configuration( | ||
configuration_path: str, default_configuration: PylintConfiguration | ||
) -> PylintConfiguration: | ||
"""Get the expected parsed configuration of a configuration functional test""" | ||
DanielNoord marked this conversation as resolved.
Show resolved
Hide resolved
|
||
result = copy.deepcopy(default_configuration) | ||
config_as_json = get_expected_or_default( | ||
configuration_path, suffix="result.json", default="{}" | ||
) | ||
to_override = json.loads(config_as_json) | ||
for key, value in to_override.items(): | ||
if key == EXPECTED_CONF_APPEND_KEY: | ||
for fkey, fvalue in value.items(): | ||
result[fkey] += fvalue | ||
elif key == EXPECTED_CONF_REMOVE_KEY: | ||
for fkey, fvalue in value.items(): | ||
new_value = [] | ||
for old_value in result[fkey]: | ||
if old_value not in fvalue: | ||
new_value.append(old_value) | ||
result[fkey] = new_value | ||
else: | ||
result[key] = value | ||
return result | ||
|
||
|
||
def get_expected_output(configuration_path: str) -> Tuple[int, str]: | ||
"""Get the expected output of a functional test.""" | ||
output = get_expected_or_default(configuration_path, suffix="out", default="") | ||
if output: | ||
# logging is helpful to see what the expected exit code is and why. | ||
# The output of the program is checked during the test so printing | ||
# messes with the result. | ||
logging.info( | ||
DanielNoord marked this conversation as resolved.
Show resolved
Hide resolved
|
||
"Output exists for %s so the expected exit code is 2", configuration_path | ||
) | ||
exit_code = 2 | ||
else: | ||
logging.info(".out file does not exists, so the expected exit code is 0") | ||
exit_code = 0 | ||
return exit_code, output.format( | ||
abspath=configuration_path, | ||
relpath=Path(configuration_path).relative_to(USER_SPECIFIC_PATH), | ||
) | ||
|
||
|
||
def run_using_a_configuration_file( | ||
configuration_path: Union[Path, str], file_to_lint: str = __file__ | ||
) -> Tuple[Mock, Mock, Run]: | ||
"""Simulate a run with a configuration without really launching the checks.""" | ||
configuration_path = str(configuration_path) | ||
args = ["--rcfile", configuration_path, file_to_lint] | ||
# We do not capture the `SystemExit` as then the `runner` variable | ||
# would not be accessible outside the `with` block. | ||
with unittest.mock.patch("sys.exit") as mocked_exit: | ||
# Do not actually run checks, that could be slow. We don't mock | ||
# `Pylinter.check`: it calls `Pylinter.initialize` which is | ||
# needed to properly set up messages inclusion/exclusion | ||
# in `_msg_states`, used by `is_message_enabled`. | ||
check = "pylint.lint.pylinter.check_parallel" | ||
with unittest.mock.patch(check) as mocked_check_parallel: | ||
runner = Run(args) | ||
return mocked_exit, mocked_check_parallel, runner | ||
DanielNoord marked this conversation as resolved.
Show resolved
Hide resolved
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,10 @@ | ||
from pathlib import Path | ||
|
||
import pytest | ||
|
||
HERE = Path(__file__).parent | ||
|
||
|
||
@pytest.fixture() | ||
def file_to_lint_path() -> str: | ||
return str(HERE / "file_to_lint.py") |
5 changes: 5 additions & 0 deletions
5
tests/config/functional/ini/pylintrc_with_message_control.ini
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
# Check that we can read the "regular" INI .pylintrc file | ||
[messages control] | ||
disable = logging-not-lazy,logging-format-interpolation | ||
jobs = 10 | ||
reports = yes |
7 changes: 7 additions & 0 deletions
7
tests/config/functional/ini/pylintrc_with_message_control.result.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
{ | ||
"functional_append": { | ||
"disable": [["logging-not-lazy"], ["logging-format-interpolation"]] | ||
}, | ||
"jobs": 10, | ||
"reports": true | ||
} |
5 changes: 5 additions & 0 deletions
5
tests/config/functional/setup_cfg/setup_cfg_with_message_control.cfg
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
# setup.cfg is an INI file where section names are prefixed with "pylint." | ||
[pylint.messages control] | ||
disable = logging-not-lazy,logging-format-interpolation | ||
jobs = 10 | ||
reports = yes |
7 changes: 7 additions & 0 deletions
7
tests/config/functional/setup_cfg/setup_cfg_with_message_control.result.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
{ | ||
"functional_append": { | ||
"disable": [["logging-not-lazy"], ["logging-format-interpolation"]] | ||
}, | ||
"jobs": 10, | ||
"reports": true | ||
} |
2 changes: 2 additions & 0 deletions
2
tests/config/functional/toml/issue_4746/loaded_plugin_does_not_exists.out
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
************* Module {abspath} | ||
{relpath}:1:0: E0013: Plugin 'pylint_websockets' is impossible to load, is it installed ? ('No module named 'pylint_websockets'') (bad-plugin-value) |
3 changes: 3 additions & 0 deletions
3
tests/config/functional/toml/issue_4746/loaded_plugin_does_not_exists.result.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
{ | ||
"load_plugins": ["pylint_websockets"] | ||
} |
3 changes: 3 additions & 0 deletions
3
tests/config/functional/toml/issue_4746/loaded_plugin_does_not_exists.toml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
# The pylint_websockets plugin does not exist and therefore this toml is invalid | ||
[tool.pylint.MASTER] | ||
load-plugins = 'pylint_websockets' |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
{ | ||
"functional_append": { | ||
"disable": [["logging-not-lazy"], ["logging-format-interpolation"]] | ||
}, | ||
"jobs": 10, | ||
"reports": true | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
# Check that we can read a TOML file where lists, integers and | ||
# booleans are expressed as such (and not as strings), using TOML | ||
# type system. | ||
|
||
[tool.pylint."messages control"] | ||
disable = [ | ||
"logging-not-lazy", | ||
"logging-format-interpolation", | ||
] | ||
jobs = 10 | ||
reports = true |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,9 @@ | ||
{ | ||
"functional_append": { | ||
"disable": [["logging-not-lazy"], ["logging-format-interpolation"]], | ||
"enable": [["suppressed-message"], ["locally-disabled"]] | ||
}, | ||
"functional_remove": { | ||
"disable": [["suppressed-message"], ["locally-disabled"]] | ||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
# Check that we can add or remove value in list | ||
# (This is mostly a check for the functional test themselves) | ||
[tool.pylint."messages control"] | ||
disable = "logging-not-lazy,logging-format-interpolation" | ||
enable = "locally-disabled,suppressed-message" |
7 changes: 7 additions & 0 deletions
7
tests/config/functional/toml/toml_with_message_control.result.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
{ | ||
"functional_append": { | ||
"disable": [["logging-not-lazy"], ["logging-format-interpolation"]] | ||
}, | ||
"jobs": 10, | ||
"reports": true | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
# Check that we can read a TOML file where lists and integers are | ||
# expressed as strings. | ||
|
||
[tool.pylint."messages control"] | ||
disable = "logging-not-lazy,logging-format-interpolation" | ||
jobs = "10" | ||
reports = "yes" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Pierre-Sassoulas Why is this necessary? Does this add anything other than allowing the test framework to function?
Ran into this while working on
argparse
.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it's setting the file that the
add_message
will display if we add a message later. I.E.[relpath}:1:0: E0013:
later.