Validate against invalid rule schema defaults in RuleTester

[not-an-aardvark]

The version of ESLint you are using.

v5.15.0

The problem you want to solve.

As of v5.14.0, we allow rule schemas to include the default keyword to specify default options. However, it's easy for a rule author to inadvertently use the default keyword in a place where the keyword is ignored, resulting in a buggy rule. (Also see: #11427)

Your take on the correct solution to problem.

When we validate rule schemas in RuleTester, we should use the new strictDefaults option in Ajv to raise an error if a rule schema has an invalid default keyword.

This is a breaking change for users who use RuleTester, because RuleTester would sometimes start reporting additional validation errors. However, rules themselves would still work as before, even if they contain invalid defaults. Additionally, the impact is expected to be very low since we only added support for the default keyword recently.

Are you willing to submit a pull request to implement this change?

Yes

[aladdin-add]

And, we can enable validateSchema: true in rule tester.

eslint/lib/util/ajv.js

Lines 18 to 25 in b00a5e9

	const ajv = new Ajv({
	meta: false,
	useDefaults: true,
	validateSchema: false,
	missingRefs: "ignore",
	verbose: true,
	schemaId: "auto"
	});

[not-an-aardvark]

This proposal was accepted in yesterday's TSC meeting.