Empirical study of linters' rules documentation

[jrfaller]

Hi all, and thanks for the amazing work on PHP-CS-Fixer! We are a team of software engineering researchers and we have worked on finding out how to best document rules from tools such as PHP-CS-FIXER (that was included in our population of linters :-D). Our results are available on our replication website : https://icpc2024-asats.github.io/ (including full paper and data). We hope that you'll find this useful (and don't hesitate to provide us with any feedback)!

[Wirone]

Hi @jrfaller, thanks for the information! Took a look at survey's results and first of all I am very happy that most of the developers see the value in linters 😁. When it comes to Fixer, it can be considered as a linter when using --dry-run option or when check command is used - otherwise it has 100% coverage of found-fixed, which is a fundamental difference between our tool and other ASATs.

FYI: I believe it would be better to point to official documentation for rules instead of Github's listing.

When it comes to nomenclature, I believe data is missing there:

• When Not To Use It: I believe all risky rules have such information, regular rules just don't need it as they're considered safe (so the only scenario for not using them is personal preference).
• Further information: I believe we try to provide any valuable information in the description. But most often rules don't require external resources to describe what and why they do.
• Compatibility: each Fixer rule has the same compatibility, which is defined on tool's level (currently PHP 7.4-8.3), it is just omitted if the runtime is not supported (e.g. attribute_empty_parentheses obviously won't do anything on PHP 7.4, as attributes were added in 8.0)
• Error output: error output depends on how Fixer is executed. With --diff flag it will report diff that is going to be applied, with --verbose it will show rules' names that are going to be applied. Maybe I don't get the purpose of this particular column in the table, but in general there's nothing like "rule error output", there's only top-level tool's error output.
• IDE Fix: I don't understand why it's empty, but in general it's possible to use Fixer as IDE's formatter, at least in PHPStorm. So definitely it's possible to fix code using IDE's controls.
• Since: I believe it does not matter that much which version introduced the rule, as they are improved and enhanced later anyway. Maintaining changelog for each rule would not be that easy.
• Usage example: I believe code examples in each fixer clearly show what each rule does. When it comes to actual usage of the tool, it's documented separately (also here for config file), and it wouldn't make sense to do it per rule
• Related rules: I don't see much value in it, honestly. Rules are standalone, but are grouped by "category" (e.g. array notation).
• Rule definition: I believe each rule has a description and there's no need for describing algorithms etc. Rule just does X if Y is found, that's included in the description and shown by code examples.
• Severity: as far as I see, you considered rules' riskiness as severity which IMHO is wrong. Our rules don't have severity, only priorities. All found violations have the same severity and are fixed in priorities' order (because one rule can be based on the code produced by the other rule). Riskiness is just an indicator that applying such a rule may result in the errors in the application, because Fixer does not have whole codebase's context when it applies fixes - it just do "dumb" changes in a single file, without ability to know if the change produces working code (Fixer only checks syntax correctness after the fix is applied). So when it comes to rules, it's not possible to assign severity, as it's everything opinionated - people are free to use, disable or reconfigure any of the rules.

I don't understand the table under Taxonomy, so can't really comment about it. Looking at "Taxonomy Validation" I think we just should include "Why" to our rules, but I am not sure if it's possible. Really often it's just an opinionated logic and the "why" can be subjective.

Anyway, interesting study, congrats 👏.

[jrfaller]

Hi @Wirone and thanks a lot for your detailed feedback that is very helpful in understanding the specificities of fixer! We will fix the links to refer to the website.

Thanks for the detailed explanation of how the nomenclature applies to fixer. For IDE fix, it is empty for fixer because there is no dedicated documentation about this at the rule level (for some other tools, there is specific documentation at the rule level for this). For error output, some tools document at the rule level which kind of warning will be displayed in the console. Since fixer is a 100% autofix tool, it does not make sense in your case that's why it's not present in your documentation. You're right that we conflated riskiness and severity, therefore riskiness should be a new column in our nomenclature for the next iteration of this work 😄.

Developers we surveyed were very explicit about the importance for them to understand why they should be applying or not a rule. We completely agree that some rules, especially rules for code formatting or readability, are really subjective and I think that it's the case for most if not all rules for fixer (other tools have rules that are less subjective like no-var in javascript), so I guess they boil down to the same overarching rationale: we think that PHP code is easier to read when etc... Therefore it would perhaps be redundant to explain that for each rule of fixer 😆.

Cheers!