New: multiple processors support (fixes #11035, fixes #11725) (#11552)

* New: multiple processors support (fixes #11035) * improve coverage * trivial fix * improve coverage * improve document Co-Authored-By: Ilya Volodin <ivolodin@gmail.com> * improve document Co-Authored-By: Ilya Volodin <ivolodin@gmail.com> * improve document Co-Authored-By: Ilya Volodin <ivolodin@gmail.com> * improve document Co-Authored-By: Ilya Volodin <ivolodin@gmail.com> * add note that `filterCodeBlock` option overrides the default behavior * improve document about code block name * extRegExp → extensionRegExp * processor → processorName * share unIndent function
eslint · May 25, 2019 · b5fa149 · b5fa149
1 parent 2d32a9e
commit b5fa149
Show file tree

Hide file tree

Showing 18 changed files with 787 additions and 130 deletions.
diff --git a/conf/config-schema.js b/conf/config-schema.js
@@ -17,6 +17,7 @@ const baseConfigProperties = {
     parser: { type: ["string", "null"] },
     parserOptions: { type: "object" },
     plugins: { type: "array" },
+    processor: { type: "string" },
     rules: { type: "object" },
     settings: { type: "object" },
 

diff --git a/docs/developer-guide/nodejs-api.md b/docs/developer-guide/nodejs-api.md
@@ -96,8 +96,10 @@ The most important method on `Linter` is `verify()`, which initiates linting of
     * **Note**: If you want to lint text and have your configuration be read and processed, use CLIEngine's [`executeOnFiles`](#cliengineexecuteonfiles) or [`executeOnText`](#cliengineexecuteontext) instead.
 * `options` - (optional) Additional options for this run.
     * `filename` - (optional) the filename to associate with the source code.
-    * `preprocess` - (optional) A function that accepts a string containing source text, and returns an array of strings containing blocks of code to lint. Also see: [Processors in Plugins](/docs/developer-guide/working-with-plugins.md#processors-in-plugins)
-    * `postprocess` - (optional) A function that accepts an array of problem lists (one list of problems for each block of code from `preprocess`), and returns a one-dimensional array of problems containing problems for the original, unprocessed text. Also see: [Processors in Plugins](/docs/developer-guide/working-with-plugins.md#processors-in-plugins)
+    * `preprocess` - (optional) A function that [Processors in Plugins](/docs/developer-guide/working-with-plugins.md#processors-in-plugins) documentation describes as the `preprocess` method.
+    * `postprocess` - (optional) A function that [Processors in Plugins](/docs/developer-guide/working-with-plugins.md#processors-in-plugins) documentation describes as the `postprocess` method.
+    * `filterCodeBlock` - (optional) A function that decides which code blocks the linter should adopt. The function receives two arguments. The first argument is the virtual filename of a code block. The second argument is the text of the code block. If the function returned `true` then the linter adopts the code block. If the function was omitted, the linter adopts only `*.js` code blocks. If you provided a `filterCodeBlock` function, it overrides this default behavior, so the linter doesn't adopt `*.js` code blocks automatically.
+    * `disableFixes` - (optional) when set to `true`, the linter doesn't make the `fix` property of the lint result.
     * `allowInlineConfig` - (optional) set to `false` to disable inline comments from changing ESLint rules.
     * `reportUnusedDisableDirectives` - (optional) when set to `true`, adds reported errors for unused `eslint-disable` directives when no problems would be reported in the disabled area anyway.
 

diff --git a/docs/developer-guide/working-with-plugins.md b/docs/developer-guide/working-with-plugins.md
@@ -54,15 +54,16 @@ You can also create plugins that would tell ESLint how to process files other th
 ```js
 module.exports = {
     processors: {
-
-        // assign to the file extension you want (.js, .jsx, .html, etc.)
-        ".ext": {
+        "processor-name": {
             // takes text of the file and filename
             preprocess: function(text, filename) {
                 // here, you can strip out any non-JS content
                 // and split into multiple strings to lint
 
-                return [string];  // return an array of strings to lint
+                return [ // return an array of code blocks to lint
+                    { text: code1, filename: "0.js" },
+                    { text: code2, filename: "1.js" },
+                ];
             },
 
             // takes a Message[][] and filename
@@ -72,7 +73,7 @@ module.exports = {
                 // to the text that was returned in array from preprocess() method
 
                 // you need to return a one-dimensional array of the messages you want to keep
-                return messages[0];
+                return [].concat(...messages);
             },
 
             supportsAutofix: true // (optional, defaults to false)
@@ -81,9 +82,13 @@ module.exports = {
 };
 ```
 
-The `preprocess` method takes the file contents and filename as arguments, and returns an array of strings to lint. The strings will be linted separately but still be registered to the filename. It's up to the plugin to decide if it needs to return just one part, or multiple pieces. For example in the case of processing `.html` files, you might want to return just one item in the array by combining all scripts, but for `.md` file where each JavaScript block might be independent, you can return multiple items.
+**The `preprocess` method** takes the file contents and filename as arguments, and returns an array of code blocks to lint. The code blocks will be linted separately but still be registered to the filename.
+
+A code block has two properties `text` and `filename`; the `text` property is the content of the block and the `filename` property is the name of the block. Name of the block can be anything, but should include the file extension, that would tell the linter how to process the current block. The linter will check [`--ext` CLI option](../user-guide/command-line-interface.md#--ext) to see if the current block should be linted, and resolve `overrides` configs to check how to process the current block.
+
+It's up to the plugin to decide if it needs to return just one part, or multiple pieces. For example in the case of processing `.html` files, you might want to return just one item in the array by combining all scripts, but for `.md` file where each JavaScript block might be independent, you can return multiple items.
 
-The `postprocess` method takes a two-dimensional array of arrays of lint messages and the filename. Each item in the input array corresponds to the part that was returned from the `preprocess` method. The `postprocess` method must adjust the locations of all errors to correspond to locations in the original, unprocessed code, and aggregate them into a single flat array and return it.
+**The `postprocess` method** takes a two-dimensional array of arrays of lint messages and the filename. Each item in the input array corresponds to the part that was returned from the `preprocess` method. The `postprocess` method must adjust the locations of all errors to correspond to locations in the original, unprocessed code, and aggregate them into a single flat array and return it.
 
 Reported problems have the following location information:
 
@@ -117,6 +122,41 @@ By default, ESLint will not perform autofixes when a processor is used, even whe
 You can have both rules and processors in a single plugin. You can also have multiple processors in one plugin.
 To support multiple extensions, add each one to the `processors` element and point them to the same object.
 
+#### Specifying Processor in Config Files
+
+To use a processor, add its ID to a `processor` section in the config file. Processor ID is a concatenated string of plugin name and processor name with a slash as a separator. This can also be added to a `overrides` section of the config, to specify which processors should handle which files.
+
+For example:
+
+```yml
+plugins:
+  - a-plugin
+overrides:
+  - files: "*.md"
+    processor: a-plugin/markdown
+```
+
+See [Specifying Processor](../user-guide/configuring.md#specifying-processor) for details.
+
+#### File Extension-named Processor
+
+If a processor name starts with `.`, ESLint handles the processor as a **file extension-named processor** especially and applies the processor to the kind of files automatically. People don't need to specify the file extension-named processors in their config files.
+
+For example:
+
+```js
+module.exports = {
+    processors: {
+        // This processor will be applied to `*.md` files automatically.
+        // Also, people can use this processor as "plugin-id/.md" explicitly.
+        ".md": {
+            preprocess(text, filename) { /* ... */ },
+            postprocess(messageLists, filename) { /* ... */ }
+        }
+    }
+}
+```
+
 ### Configs in Plugins
 
 You can bundle configurations inside a plugin by specifying them under the `configs` key. This can be useful when you want to provide not just code style, but also some custom rules to support it. Multiple configurations are supported per plugin. Note that it is not possible to specify a default configuration for a given plugin and that users must specify in their configuration file when they want to use one.

diff --git a/docs/user-guide/configuring.md b/docs/user-guide/configuring.md
@@ -80,6 +80,55 @@ The following parsers are compatible with ESLint:
 
 Note when using a custom parser, the `parserOptions` configuration property is still required for ESLint to work properly with features not in ECMAScript 5 by default. Parsers are all passed `parserOptions` and may or may not use them to determine which features to enable.
 
+## Specifying Processor
+
+Plugins may provide processors. Processors can extract JavaScript code from another kind of files, then lets ESLint lint the JavaScript code. Or processors can convert JavaScript code in preprocessing for some purpose.
+
+To specify processors in a configuration file, use the `processor` key with the concatenated string of a plugin name and a processor name by a slash. For example, the following enables the processor `a-processor` that the plugin `a-plugin` provided:
+
+```json
+{
+    "plugins": ["a-plugin"],
+    "processor": "a-plugin/a-processor"
+}
+```
+
+To specify processors for a specific kind of files, use the combination of the `overrides` key and the `processor` key. For example, the following uses the processor `a-plugin/markdown` for `*.md` files.
+
+```json
+{
+    "plugins": ["a-plugin"],
+    "overrides": [
+        {
+            "files": ["*.md"],
+            "processor": "a-plugin/markdown"
+        }
+    ]
+}
+```
+
+Processors may make named code blocks such as `0.js` and `1.js`. ESLint handles such a named code block as a child file of the original file. You can specify additional configurations for named code blocks in the `overrides` section of the config. For example, the following disables `strict` rule for the named code blocks which end with `.js` in markdown files.
+
+```json
+{
+    "plugins": ["a-plugin"],
+    "overrides": [
+        {
+            "files": ["*.md"],
+            "processor": "a-plugin/markdown"
+        },
+        {
+            "files": ["**/*.md/*.js"],
+            "rules": {
+                "strict": "off"
+            }
+        }
+    ]
+}
+```
+
+ESLint checks the file extension of named code blocks then ignores those if [`--ext` CLI option](../user-guide/command-line-interface.md#--ext) didn't include the file extension. Be sure to specify the `--ext` option if you wanted to lint named code blocks other than `*.js`.
+
 ## Specifying Environments
 
 An environment defines global variables that are predefined. The available environments are:

diff --git a/lib/cli-engine/cli-engine.js b/lib/cli-engine/cli-engine.js
@@ -202,6 +202,7 @@ function calculateStatsPerRun(results) {
  * @param {boolean} config.fix If `true` then it does fix.
  * @param {boolean} config.allowInlineConfig If `true` then it uses directive comments.
  * @param {boolean} config.reportUnusedDisableDirectives If `true` then it reports unused `eslint-disable` comments.
+ * @param {RegExp} config.extensionRegExp The `RegExp` object that tests if a file path has the allowed file extensions.
  * @param {Linter} config.linter The linter instance to verify.
  * @returns {LintResult} The result of linting.
  * @private
@@ -214,6 +215,7 @@ function verifyText({
     fix,
     allowInlineConfig,
     reportUnusedDisableDirectives,
+    extensionRegExp,
     linter
 }) {
     const filePath = providedFilePath || "<text>";
@@ -233,7 +235,18 @@ function verifyText({
             allowInlineConfig,
             filename: filePathToVerify,
             fix,
-            reportUnusedDisableDirectives
+            reportUnusedDisableDirectives,
+
+            /**
+             * Check if the linter should adopt a given code block or not.
+             * Currently, the linter adopts code blocks if the name matches `--ext` option.
+             * In the future, `overrides` in the configuration would affect the adoption (https://github.com/eslint/rfcs/pull/20).
+             * @param {string} blockFilename The virtual filename of a code block.
+             * @returns {boolean} `true` if the linter should adopt the code block.
+             */
+            filterCodeBlock(blockFilename) {
+                return extensionRegExp.test(blockFilename);
+            }
         }
     );
 
@@ -773,6 +786,7 @@ class CLIEngine {
                 fix,
                 allowInlineConfig,
                 reportUnusedDisableDirectives,
+                extensionRegExp: fileEnumerator.extensionRegExp,
                 linter
             });
 
@@ -817,6 +831,7 @@ class CLIEngine {
     executeOnText(text, filename, warnIgnored) {
         const {
             configArrayFactory,
+            fileEnumerator,
             ignoredPaths,
             lastConfigArrays,
             linter,
@@ -860,6 +875,7 @@ class CLIEngine {
                 fix,
                 allowInlineConfig,
                 reportUnusedDisableDirectives,
+                extensionRegExp: fileEnumerator.extensionRegExp,
                 linter
             }));
         }

diff --git a/lib/cli-engine/config-array-factory.js b/lib/cli-engine/config-array-factory.js
@@ -528,7 +528,7 @@ class ConfigArrayFactory {
             parser: parserName,
             parserOptions,
             plugins: pluginList,
-            processor, // processor is only for file extension processors.
+            processor,
             root,
             rules,
             settings,

diff --git a/lib/cli-engine/file-enumerator.js b/lib/cli-engine/file-enumerator.js
@@ -89,7 +89,7 @@ const IGNORED = 2;
  * @typedef {Object} FileEnumeratorInternalSlots
  * @property {CascadingConfigArrayFactory} configArrayFactory The factory for config arrays.
  * @property {string} cwd The base directory to start lookup.
- * @property {RegExp} extRegExp The RegExp to test if a string ends with specific file extensions.
+ * @property {RegExp} extensionRegExp The RegExp to test if a string ends with specific file extensions.
  * @property {boolean} globInputPaths Set to false to skip glob resolution of input file paths to lint (default: true). If false, each input file paths is assumed to be a non-glob path to an existing file.
  * @property {boolean} ignoreFlag The flag to check ignored files.
  * @property {IgnoredPaths} ignoredPathsWithDotfiles The ignored paths but don't include dot files.
@@ -196,7 +196,7 @@ class FileEnumerator {
         internalSlotsMap.set(this, {
             configArrayFactory,
             cwd,
-            extRegExp: new RegExp(
+            extensionRegExp: new RegExp(
                 `.\\.(?:${extensions
                     .map(ext => escapeRegExp(
                         ext.startsWith(".")
@@ -217,6 +217,14 @@ class FileEnumerator {
         });
     }
 
+    /**
+     * The `RegExp` object that tests if a file path has the allowed file extensions.
+     * @type {RegExp}
+     */
+    get extensionRegExp() {
+        return internalSlotsMap.get(this).extensionRegExp;
+    }
+
     /**
      * Iterate files which are matched by given glob patterns.
      * @param {string|string[]} patternOrPatterns The glob patterns to iterate files.
@@ -384,7 +392,7 @@ class FileEnumerator {
             return;
         }
         debug(`Enter the directory: ${directoryPath}`);
-        const { configArrayFactory, extRegExp } = internalSlotsMap.get(this);
+        const { configArrayFactory, extensionRegExp } = internalSlotsMap.get(this);
 
         /** @type {ConfigArray|null} */
         let config = null;
@@ -407,7 +415,7 @@ class FileEnumerator {
                     ? options.selector.match(filePath)
 
                     // Started with a directory path; choose by file extensions.
-                    : extRegExp.test(filePath);
+                    : extensionRegExp.test(filePath);
 
                 if (matched) {
                     debug(`Yield: ${filename}${ignored ? " but ignored" : ""}`);