Update reporters to (allow) use of `end_line` and `end_column` #5372

DanielNoord · 2021-11-22T16:40:49Z

Add yourself to CONTRIBUTORS if you are a new contributor.
Add a ChangeLog entry describing what your PR does.
If it's a new feature, or an important bug fix, add a What's New entry in
doc/whatsnew/<current release.rst>.
Write a good description on what the PR does.

Type of Changes

	Type
✓	✨ New feature
✓	📜 Docs

Description

Blocked by #5343
Ref #5336

This updates the reporters of pylint to start using end_line and end_column. It is the first PR as mentioned in #5343 (comment)

…porter

pylint/reporters/text.py

coveralls · 2021-11-22T22:12:17Z

Pull Request Test Coverage Report for Build 1499421022

17 of 17 (100.0%) changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.007%) to 93.483%

Totals
Change from base Build 1499266391:	0.007%
Covered Lines:	13972
Relevant Lines:	14946

💛 - Coveralls

ChangeLog

doc/user_guide/output.rst

doc/whatsnew/2.12.rst

pylint/reporters/text.py

cdce8p · 2021-11-22T22:06:15Z

pylint/reporters/text.py

+        for key, value in self_dict.items():
+            # pylint: disable=fixme
+            # TODO: Add column to list of attributes to be printed as an empty string
+            if value is None and key in PRINT_AS_EMPTY_STRING:
+                self_dict[key] = ""


Is a for - loop really necessary here? Maybe iterate only over PRINT_AS_EMPTY_STRING and do

self_dict[key] = self_dict[key] or ""

Or even better, what about this?

Initialize end_lineno and end_col_offset with an empty string instead "". You still would be able to differentiate between a null value and a number. I.e. int | Literal[""] as type.

Or even better, what about this?

Initialize end_lineno and end_col_offset with an empty string instead "". You still would be able to differentiate between a null value and a number. I.e. int | Literal[""] as type.

I don't really like that as then we we're changing the type of end_lineno and end_col_offset while still "internal" to fit a specific type of reporter. For example, for the JSONReporter it makes no sense to have it be "". I'm not sure if other developer have written other Reporter but I think keeping both attributes None until a specific reporter starts handling them is the better option here.
I'll add your first suggestion!

cdce8p · 2021-11-22T22:12:25Z

pylint/reporters/text.py

+            # TODO: Add column to list of attributes to be printed as an empty string
+            if value is None and key in PRINT_AS_EMPTY_STRING:
+                self_dict[key] = ""
+        self.writeln(self._template.format(**self_dict))


self.writeln(msg.format(self._template)) has been quite elegant, but there is one drawback. Calling it with an unknown key raises a KeyError. Ideally we could check a template and replace invalid values with "" beforehand. I'm currently having the issue to implement some kind of version check in vscode-python as calling pylint with the new msg-template would raise this error.

Traceback (most recent call last): File "/.../venv-310/bin/pylint", line 33, in <module> sys.exit(load_entry_point('pylint', 'console_scripts', 'pylint')()) File "/.../pylint/__init__.py", line 24, in run_pylint PylintRun(sys.argv[1:]) File "/.../pylint/lint/run.py", line 398, in __init__ linter.check(args) File "/.../pylint/lint/pylinter.py", line 992, in check self._check_files( File "/.../pylint/lint/pylinter.py", line 1041, in _check_files self.add_message(symbol, args=msg) File "/.../pylint/lint/pylinter.py", line 1513, in add_message self._add_one_message( File "/.../pylint/lint/pylinter.py", line 1471, in _add_one_message self.reporter.handle_message( File "/.../pylint/reporters/text.py", line 202, in handle_message self.write_message(msg) File "/.../pylint/reporters/text.py", line 192, in write_message self.writeln(msg.format(self._template)) File "/.../pylint/message/message.py", line 94, in format return template.format(**self._asdict()) KeyError: 'msg1'

Shouldn't that just raise a new warning? Something like config-parse-error as introduced in #5365. We could emit it on an except for the KeyError.

That doesn't really solve the issue. At the core of it its backwards incompatibility.

We should be able to use a new template, i.e. with end_lineno, in old pylint versions and still get a usable result. The way it's now, I need to make sure only to add end_lineno if we are using the new pylint version.

If at some point we add another key, this will only repeat. Every time incompatible with old versions.

Came up with something. A little hacky, but let me know what you think. I added if not self._checked_template: because I thought the additional if statement would be better than doing the regex statement each line.

pylint/reporters/text.py

Co-authored-by: Marc Mueller <30130371+cdce8p@users.noreply.github.com>

pylint/reporters/text.py

cdce8p · 2021-11-22T23:03:03Z

pylint/reporters/text.py

+            # TODO: Add column to list of attributes to be printed as an empty string
+            if value is None and key in PRINT_AS_EMPTY_STRING:
+                self_dict[key] = ""
+        self.writeln(self._template.format(**self_dict))


That doesn't really solve the issue. At the core of it its backwards incompatibility.

We should be able to use a new template, i.e. with end_lineno, in old pylint versions and still get a usable result. The way it's now, I need to make sure only to add end_lineno if we are using the new pylint version.

If at some point we add another key, this will only repeat. Every time incompatible with old versions.

cdce8p · 2021-11-22T23:56:00Z

pylint/reporters/text.py

+        # Check to see if all parameters in the template are attributes of the Message
+        if not self._checked_template:
+            template = self._template
+            parameters = re.findall(r"\{(.+?)\}", template)
+            for parameter in parameters:
+                if parameter not in self_dict:
+                    template = template.replace(f"{{{parameter}}}", "")
+            self._template = template
+            self._checked_template = True


Can this be moved to on_set_current_module or even __init__? Don't now if the config settings are defined at the point already.

Moved to on_set_current_module

cdce8p · 2021-11-22T23:57:30Z

pylint/reporters/text.py

+        # Check to see if all parameters in the template are attributes of the Message
+        if not self._checked_template:
+            template = self._template
+            parameters = re.findall(r"\{(.+?)\}", template)


The regex doesn't quiet work with formatting strings. Maybe something like this?

{([^:]+?)(?::([^:]+))?}

for

"{line:03d}"

Used a new pattern. I think my new one works as well although the replacing is quite ugly.

cdce8p · 2021-11-22T23:58:27Z

pylint/reporters/text.py

+            parameters = re.findall(r"\{(.+?)\}", template)
+            for parameter in parameters:
+                if parameter not in self_dict:
+                    template = template.replace(f"{{{parameter}}}", "")


Is it save to overwrite self._template directly?

It should be in its new place

cdce8p · 2021-11-23T00:38:37Z

pylint/reporters/text.py

+        # Check to see if all parameters in the template are attributes of the Message
+        template = self._template
+        parameters = re.findall(r"\{(.+?)(:.*)?\}", template)
+        for parameter in parameters:
+            if parameter[0] not in Message._fields:
+                template = re.sub(r"\{" + parameter[0] + r"(:.*?)?\}", "", template)
+        self._template = template


Just an idea, currently this is down for every module. Can we store the original template and the new one. Then for every subsequent call, compare the "new" self._template with the stored template and if they match use the fixed one.

It might also be a good idea to at least emit a short warning: Hey, we don't know this option are you sure?. Otherwise we fail silently.

Done and done!

tests/unittest_reporting.py

Co-authored-by: Pierre Sassoulas <pierre.sassoulas@gmail.com>

for more information, see https://pre-commit.ci

pylint/reporters/text.py

Pierre-Sassoulas · 2021-11-23T19:22:34Z

doc/whatsnew/2.12.rst

+  end of a node to the output of Pylint. If these numbers are unknown, they are represented
+  by an empty string.
+
+* Add ``end_line`` and ``end_column`` fields to the output of the ``JSONReporter``.


It look like a breaking change that should wait for #4741. We have a 3.0 alpha branch, might be the time to dust it off ?

Reverted! See #5380

Did not see change to JsonReporter

… into column-reporter

Pierre-Sassoulas · 2021-11-24T12:56:07Z

@DanielNoord I fixed the merged conflict but is this ready to review ? I'm a little lost about the order to follow between #5349 #5372 and #5336

DanielNoord · 2021-11-24T12:59:21Z

@DanielNoord I fixed the merged conflict but is this ready to review ? I'm a little lost about the order to follow between #5349 #5372 and #5336

I think you mean #5349, #5372 and #5376.

They are already for review and can be merged separately. They don't depend on each other!

Pierre-Sassoulas · 2021-11-24T13:12:18Z

#5376 was not in the 2.12.0 milestone I added it.

Pierre-Sassoulas

LGTM ! I only have a small question.

Pierre-Sassoulas · 2021-11-24T13:15:27Z

doc/whatsnew/2.12.rst

+* Add ability to add ``end_line`` and ``end_column`` to the ``--msg-template`` option.
+  With the standard ``TextReporter`` this will add the line and column number of the
+  end of a node to the output of Pylint. If these numbers are unknown, they are represented
+  by an empty string.


I think we could also change the default for msg-template in 3.0. Do we want the default to have end line and end column ? This seems like something that is useful in an IDE but not a lot for a text output read by humans.

@cdce8p Argued that we might leave the default output as is because of the potential to break many tools. I think this is a discussion which (if we want to have it anyway) we could leave until we are closer to the release of 3.0. With this merge everybody that wants to can use this new feature!

Update reporters to (allow) use end_line and end_column

f395226

DanielNoord added Enhancement ✨ Improvement to a component Blocked 🚧 Blocked by a particular issue Documentation 📗 labels Nov 22, 2021

DanielNoord added this to the 2.12.0 milestone Nov 22, 2021

DanielNoord mentioned this pull request Nov 22, 2021

Add end_lineno and end_col_offset to MessageLocationTuple #5343

Merged

2 tasks

Merge branch 'main' of https://github.com/PyCQA/pylint into column-re…

09bd841

…porter

DanielNoord commented Nov 22, 2021

View reviewed changes

pylint/reporters/text.py Outdated Show resolved Hide resolved

DanielNoord mentioned this pull request Nov 22, 2021

Include end column / range in message output #5336

Closed

6 tasks

Update tests and spelling

837b65f

DanielNoord marked this pull request as ready for review November 22, 2021 22:11

DanielNoord requested review from cdce8p and Pierre-Sassoulas November 22, 2021 22:11

cdce8p requested changes Nov 22, 2021

View reviewed changes

DanielNoord commented Nov 22, 2021

View reviewed changes

pylint/reporters/text.py Outdated Show resolved Hide resolved

DanielNoord and others added 2 commits November 22, 2021 23:19

Apply suggestions from code review

9792d04

Co-authored-by: Marc Mueller <30130371+cdce8p@users.noreply.github.com>

Code review

45e7bd0

cdce8p reviewed Nov 22, 2021

View reviewed changes

DanielNoord added 2 commits November 23, 2021 00:17

Code review

d980fb8

Code review

1ed7e17

cdce8p reviewed Nov 22, 2021

View reviewed changes

DanielNoord added 2 commits November 23, 2021 00:58

Fix mistake with regex

35fd523

Move the overwrite

45e0033

DanielNoord removed the Blocked 🚧 Blocked by a particular issue label Nov 23, 2021

cdce8p reviewed Nov 23, 2021

View reviewed changes

DanielNoord added 2 commits November 23, 2021 09:05

Code review

0a6e6d6

Add warning

3c3f141

DanielNoord requested a review from cdce8p November 23, 2021 08:17

Pierre-Sassoulas previously approved these changes Nov 23, 2021

View reviewed changes

tests/unittest_reporting.py Outdated Show resolved Hide resolved

tests/unittest_reporting.py Outdated Show resolved Hide resolved

DanielNoord and others added 2 commits November 23, 2021 09:58

Apply suggestions from code review

b24dd1c

Co-authored-by: Pierre Sassoulas <pierre.sassoulas@gmail.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

6ea66c6

for more information, see https://pre-commit.ci

DanielNoord commented Nov 23, 2021

View reviewed changes

pylint/reporters/text.py Outdated Show resolved Hide resolved

Update pylint/reporters/text.py

a1bdf7c

Pierre-Sassoulas reviewed Nov 23, 2021

View reviewed changes

DanielNoord and others added 3 commits November 24, 2021 08:33

Revert changes to JSONReporter

6e91225

Merge branch 'column-reporter' of https://github.com/DanielNoord/pylint…

e280f8c

… into column-reporter

Merge branch 'main' into column-reporter

da4932d

Pierre-Sassoulas approved these changes Nov 24, 2021

View reviewed changes

Pierre-Sassoulas merged commit 7c3533c into pylint-dev:main Nov 24, 2021

DanielNoord deleted the column-reporter branch November 24, 2021 14:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update reporters to (allow) use of `end_line` and `end_column` #5372

Update reporters to (allow) use of `end_line` and `end_column` #5372

DanielNoord commented Nov 22, 2021

coveralls commented Nov 22, 2021 •

edited

cdce8p Nov 22, 2021

cdce8p Nov 22, 2021

DanielNoord Nov 22, 2021

cdce8p Nov 22, 2021

DanielNoord Nov 22, 2021

cdce8p Nov 22, 2021

DanielNoord Nov 22, 2021

cdce8p Nov 22, 2021

cdce8p Nov 22, 2021

DanielNoord Nov 23, 2021

cdce8p Nov 22, 2021

DanielNoord Nov 23, 2021

cdce8p Nov 22, 2021

DanielNoord Nov 23, 2021

cdce8p Nov 23, 2021

cdce8p Nov 23, 2021

DanielNoord Nov 23, 2021

Pierre-Sassoulas Nov 23, 2021

DanielNoord Nov 24, 2021

Pierre-Sassoulas commented Nov 24, 2021 •

edited

DanielNoord commented Nov 24, 2021

Pierre-Sassoulas commented Nov 24, 2021

Pierre-Sassoulas left a comment

Pierre-Sassoulas Nov 24, 2021

DanielNoord Nov 24, 2021

Update reporters to (allow) use of end_line and end_column #5372

Update reporters to (allow) use of end_line and end_column #5372

Conversation

DanielNoord commented Nov 22, 2021

Type of Changes

Description

coveralls commented Nov 22, 2021 • edited

Pull Request Test Coverage Report for Build 1499421022

💛 - Coveralls

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Pierre-Sassoulas commented Nov 24, 2021 • edited

DanielNoord commented Nov 24, 2021

Pierre-Sassoulas commented Nov 24, 2021

Pierre-Sassoulas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Update reporters to (allow) use of `end_line` and `end_column` #5372

Update reporters to (allow) use of `end_line` and `end_column` #5372

coveralls commented Nov 22, 2021 •

edited

Pierre-Sassoulas commented Nov 24, 2021 •

edited