Rule options from implicitly included plugins are not validated

[jarrodldavis]

Tell us about your environment

• ESLint Version: v5.15.3
• Node Version: v11.12.0
• npm Version: 6.9.0

What parser (default, Babel-ESLint, etc.) are you using? default (Espree)

Please show your full configuration:

Configuration

{
    "root": true,
    "env": {
        "commonjs": true,
        "es6": true,
        "node": true
    },
    "extends": ["eslint:recommended", "plugin:node/recommended"],
    "globals": {
        "Atomics": "readonly",
        "SharedArrayBuffer": "readonly"
    },
    "parserOptions": {
        "ecmaVersion": 2018
    },
    "rules": {
        "node/exports-style": ["error", "invalid-option"]
    }
}

What did you do? Please include the actual source code causing the issue, as well as the command that you used to run ESLint.

const none = require("./doesnt-exist");

$ ./node_modules/.bin/eslint .

What did you expect to happen?

I expected ESLint to report a validation error like so:

$ ./node_modules/.bin/eslint .
Error: /Users/jarrodldavis/source/repos/repro-eslint-validate-issue/explicit-plugin/.eslintrc.json:
	Configuration for rule "node/exports-style" is invalid:
	Value "invalid-option" should be equal to one of the allowed values.

    at validateRuleOptions (/Users/jarrodldavis/source/repos/repro-eslint-validate-issue/explicit-plugin/node_modules/eslint/lib/config/config-validator.js:133:19)
    at Object.keys.forEach.id (/Users/jarrodldavis/source/repos/repro-eslint-validate-issue/explicit-plugin/node_modules/eslint/lib/config/config-validator.js:176:9)
    at Array.forEach (<anonymous>)
    at validateRules (/Users/jarrodldavis/source/repos/repro-eslint-validate-issue/explicit-plugin/node_modules/eslint/lib/config/config-validator.js:175:30)
    at Object.validate (/Users/jarrodldavis/source/repos/repro-eslint-validate-issue/explicit-plugin/node_modules/eslint/lib/config/config-validator.js:262:5)
    at loadFromDisk (/Users/jarrodldavis/source/repos/repro-eslint-validate-issue/explicit-plugin/node_modules/eslint/lib/config/config-file.js:544:19)
    at Object.load (/Users/jarrodldavis/source/repos/repro-eslint-validate-issue/explicit-plugin/node_modules/eslint/lib/config/config-file.js:587:20)
    at Config.getLocalConfigHierarchy (/Users/jarrodldavis/source/repos/repro-eslint-validate-issue/explicit-plugin/node_modules/eslint/lib/config.js:240:44)
    at Config.getConfigHierarchy (/Users/jarrodldavis/source/repos/repro-eslint-validate-issue/explicit-plugin/node_modules/eslint/lib/config.js:192:43)
    at Config.getConfigVector (/Users/jarrodldavis/source/repos/repro-eslint-validate-issue/explicit-plugin/node_modules/eslint/lib/config.js:299:21)

What actually happened? Please include the actual, raw output from ESLint.

The validation error is ignored and ESLint continues as if the plugin option is valid

$ ./node_modules/.bin/eslint .

/Users/jarrodldavis/source/repos/repro-eslint-validate-issue/implicit-plugin/index.js
  1:7   error  'none' is assigned a value but never used  no-unused-vars
  1:22  error  "./doesnt-exist" is not found              node/no-missing-require

✖ 2 problems (2 errors, 0 warnings)

---

I've created a reproduction repository that demonstrates the difference in plugin rule validation behavior when implicitly vs explicitly including a plugin.

I believe I have narrowed it down to this code:

eslint/lib/config/config-file.js

Lines 541 to 552 in a6168f8

	const ruleMap = configContext.linterContext.getRules();
	// validate the configuration before continuing
	validator.validate(config, ruleMap.get.bind(ruleMap), configContext.linterContext.environments, resolvedPath.configFullName);
	/*
	* If an `extends` property is defined, it represents a configuration file to use as
	* a "parent". Load the referenced file and merge the configuration recursively.
	*/
	if (config.extends) {
	config = applyExtends(config, configContext, resolvedPath.filePath, dirname);
	}

eslint/lib/config/config-validator.js

Lines 170 to 173 in a6168f8

	function validateRules(rulesConfig, ruleMapper, source = null) {
	if (!rulesConfig) {
	return;
	}

It validates the local config (including rule options) before applying config extensions, so the plugin and its rules aren't loaded into the linter until after the rules enabled by the local config are validated. Since the rules aren't present at the time of validation, validation is silently skipped.

[not-an-aardvark]

Thanks for the report. This seems like a serious issue, I'm surprised no one had noticed this before now.

I think this is probably already fixed by #11546 (cc @mysticatea).

[jarrodldavis]

Thanks for the quick response!

I pulled down the issue11510 branch from #11546 to npm link into my reproduction repository and can confirm that the validation works as expected in the implicit-plugin directory.

[mysticatea]

I confirmed that this reproduced.

And yes, #11546 fixed this bug because it has separated config schema validation and rule option validation. In #11546, ESLint validates config schema when each config file was read and validates rule options and environments when the configuration was determined for each source file (i.e. after all cascading config files, shareable configs, plugins, and parsers are loaded).

[fvhockney]

I've pulled this into a vue project where I have been having problems with eslint and their plugin for months. I can confirm that this fixes the error I was having. Is this going to be merged in before version 6.0.0?

[mysticatea]

We have published the final release of ESLint 5.x in the last week. We have a plan to release the first 6.0.0 pre-release in the next week and #11546 will be included in that.

Unfortunately, we have not backported bug fixes to old major versions because backporting makes complex our release process and we don't have enough human resource for that. And, #11546 changes our codebase about configuration from almost scratch and that fixes some bugs which are considered breaking changes, so it's hard to backport.

At a glance, the current --fix-type implementation doesn't support plugin rules because it caches rules before finding any configs... :(

[fvhockney]

Great. #11546 seems to fix the issue for --fix-type as well. I'll keep my eyes peeled for the releases.