Error while linting large application

[mcfdez]

Current Behavior

Running npx nx lint appName starts the linting process. After a long time, it fails without yielding any information.

It is worth saying that if we reduce the scope of the files to lint, it works correctly, it only fails when the amount of files to lint is high (about 3000).

This is the Angular configuration of the project:

{
  "name": "frontend",
  "$schema": "../../node_modules/nx/schemas/project-schema.json",
  "projectType": "application",
  "prefix": "frontend",
  "sourceRoot": "code/angular/src",
  "tags": [],
  "targets": {
    "build-client": {
      "executor": "@angular-devkit/build-angular:browser",
      "outputs": ["{options.outputPath}"],
      "options": {
        "outputPath": "code/dist/frontend/client",
        "index": "code/angular/src/index.html",
        "main": "code/angular/src/main.ts",
        "polyfills": [
          "zone.js",
          "code/angular/src/polyfills.ts"
        ],
        "tsConfig": "code/angular/tsconfig.app.json",
        "assets": ["code/angular/src/favicon.ico", "code/angular/src/assets"],
        "styles": [
          {
            "input": "code/angular/src/assets/styles/styles.scss",
            "bundleName": "styles"
          },
          {
            "input": "code/angular/src/assets/styles/checkout.scss",
            "bundleName": "checkout"
          }
        ],
        "scripts": [],
        "baseHref": "static/frontend/",
        "preserveSymlinks": true,
        "budgets": [
          {
            "type": "anyComponentStyle",
            "maximumWarning": "15kb",
            "maximumError": "100kb"
          }
        ],
        "allowedCommonJsDependencies": [
          "dayjs",
          "dayjs/plugin/customParseFormat",
          "dayjs/plugin/isBetween",
          "dayjs/plugin/isoWeek",
          "dayjs/plugin/isSameOrAfter",
          "dayjs/plugin/isSameOrBefore",
          "fast-deep-equal",
          "gsap/dist/Draggable",
          "gsap/dist/gsap",
          "gsap/dist/ScrollTrigger",
          "hls.js",
          "moment-timezone",
          "moment",
          "naver-id-login",
          "qrcode",
          "sha.js",
          "tcomb-validation",
          "ts-md5/dist/md5",
          "uuidv4"
        ]
      },
      "configurations": {
        "pro": {
          "vendorChunk": true,
          "namedChunks": false,
          "optimization": true,
          "sourceMap": true,
          "extractLicenses": true,
          "buildOptimizer": true,
          "fileReplacements": [
            {
              "replace": "code/angular/src/environments/environment.ts",
              "with": "code/angular/src/environments/environment.pro.ts"
            }
          ],
          "budgets": [
            {
              "type": "initial",
              "maximumWarning": "2mb",
              "maximumError": "5mb"
            },
            {
              "type": "anyComponentStyle",
              "maximumWarning": "6kb",
              "maximumError": "20kb"
            }
          ]
        },
        "proLocal": {
          "vendorChunk": true,
          "namedChunks": true,
          "optimization": false,
          "sourceMap": true,
          "extractLicenses": false,
          "buildOptimizer": true,
          "fileReplacements": [
            {
              "replace": "code/angular/src/environments/environment.ts",
              "with": "code/angular/src/environments/environment.pro.local.ts"
            }
          ]
        },
        "pre": {
          "vendorChunk": true,
          "namedChunks": false,
          "optimization": true,
          "sourceMap": true,
          "extractLicenses": true,
          "buildOptimizer": true,
          "fileReplacements": [
            {
              "replace": "code/angular/src/environments/environment.ts",
              "with": "code/angular/src/environments/environment.pre.ts"
            }
          ],
          "budgets": [
            {
              "type": "initial",
              "maximumWarning": "2mb",
              "maximumError": "100mb"
            }
          ]
        },
        "prepruLocal": {
          "vendorChunk": true,
          "namedChunks": true,
          "optimization": false,
          "sourceMap": true,
          "extractLicenses": false,
          "buildOptimizer": true,
          "fileReplacements": [
            {
              "replace": "code/angular/src/environments/environment.ts",
              "with": "code/angular/src/environments/environment.pre.local.ts"
            }
          ]
        }
      },
      "defaultConfiguration": "pro"
    },
    "build-server": {
      "executor": "@angular-devkit/build-angular:server",
      "outputs": ["{options.outputPath}"],
      "options": {
        "outputPath": "code/dist/frontend/server",
        "main": "code/angular/src/main.server.ts",
        "tsConfig": "code/angular/tsconfig.server.json"
      },
      "configurations": {
        "pro": {
          "vendorChunk": true,
          "namedChunks": false,
          "optimization": true,
          "sourceMap": true,
          "extractLicenses": true,
          "buildOptimizer": true,
          "outputHashing": "media",
          "fileReplacements": [
            {
              "replace": "code/angular/src/environments/environment.ts",
              "with": "code/angular/src/environments/environment.pro.ts"
            }
          ]
        },
        "proLocal": {
          "vendorChunk": true,
          "namedChunks": true,
          "optimization": false,
          "sourceMap": true,
          "extractLicenses": false,
          "buildOptimizer": true,
          "outputHashing": "media",
          "fileReplacements": [
            {
              "replace": "code/angular/src/environments/environment.ts",
              "with": "code/angular/src/environments/environment.pro.local.ts"
            }
          ]
        },
        "pre": {
          "vendorChunk": true,
          "namedChunks": true,
          "optimization": true,
          "sourceMap": true,
          "extractLicenses": false,
          "buildOptimizer": false,
          "fileReplacements": [
            {
              "replace": "code/angular/src/environments/environment.ts",
              "with": "code/angular/src/environments/environment.pre.ts"
            }
          ]
        },
        "prepruLocal": {
          "vendorChunk": true,
          "namedChunks": true,
          "optimization": false,
          "sourceMap": true,
          "extractLicenses": false,
          "buildOptimizer": true,
          "outputHashing": "media",
          "fileReplacements": [
            {
              "replace": "code/angular/src/environments/environment.ts",
              "with": "code/angular/src/environments/environment.pre.local.ts"
            }
          ]
        }
      },
      "defaultConfiguration": "pro"
    },
    "extract-i18n": {
      "executor": "@angular-devkit/build-angular:extract-i18n",
      "options": {
        "browserTarget": "frontend:build"
      }
    },
    "lint": {
      "executor": "@nx/linter:eslint",
      "outputs": ["{options.outputFile}"],
      "options": {
        "lintFilePatterns": ["code/angular/**/*.ts", "code/angular/**/*.html"],
        "exclude": ["**/node_modules/**"]
      }
    },
    "test": {
      "executor": "@nx/jest:jest",
      "outputs": ["{workspaceRoot}/coverage/{projectRoot}"],
      "options": {
        "jestConfig": "code/angular/jest.config.ts",
        "passWithNoTests": true
      },
      "configurations": {
        "ci": {
          "ci": true,
          "codeCoverage": true
        }
      }
    }
  }
}

And this one, the ESLint configuration of the application

module.exports = {
  extends: [ "../../.eslintrc.js"],
  ignorePatterns: ["!**/*"],
  overrides: [
    // Typescript
    {
      files: ["*.ts"],
      parser: "@typescript-eslint/parser",
      parserOptions: {
        tsconfigRootDir: __dirname + '/../..',
        project: ["tsconfig.base.json"]
      },
      extends: [
        // "plugin:@nx/angular",
        // "plugin:@angular-eslint/template/process-inline-templates",
        "../tools/linters/frontend/plugins/base",
        "../tools/linters/frontend/plugins/prefer-arrow",
        "../tools/linters/frontend/plugins/regex",
        "../tools/linters/frontend/plugins/typescript",
        "../tools/linters/frontend/plugins/angular",
        "../tools/linters/frontend/plugins/import",
        "../tools/linters/frontend/plugins/jsdoc",
        "../tools/linters/frontend/plugins/rxjs",
        "../tools/linters/frontend/plugins/ngrx",
        "../tools/linters/frontend/plugins/deprecations",
        "../tools/linters/frontend/plugins/security",
        "../tools/linters/frontend/plugins/sonar"
      ]
    },
    // Templates
    {
      files: [ "*.html"],
      parser: "@angular-eslint/template-parser",
      extends: [
        "plugin:@nx/angular-template",
        "plugin:prettier/recommended",
        "../tools/linters/frontend/plugins/templates",
        "../tools/linters/frontend/plugins/html"
      ]
    },
    // Mocks and strubs
    {
      files: ["*.mock*.ts", "*.stub*.ts"],
      rules: {
        "@angular-eslint/component-class-suffix": "off",
        "@typescript-eslint/no-empty-function": "off"
      }
    },
    // Tests
    {
      files: [
        "*.spec.ts",
        "*.test.module.ts"
      ],
      extends: [
        "../tools/linters/frontend/plugins/jest"
      ]
    }
  ]

This is the output of the command

> nx run frontend:lint

Linting "frontend"...

 >  NX   Ran target lint for project frontend (4m)

    ✖    1/1 failed
    ✔    0/1 succeeded [0 read from cache]
}

Expected Behavior

Finish the lining process correctly.

Nx Report

>  NX   Report complete - copy this into the issue template

   Node   : 18.12.1
   OS     : linux-x64
   npm    : 8.19.2

   nx                 : 16.7.3
   @nx/js             : 16.7.3
   @nx/jest           : 16.7.3
   @nx/linter         : 16.7.3
   @nx/workspace      : 16.7.3
   @nx/angular        : 16.7.3
   @nx/cypress        : 16.7.3
   @nx/devkit         : 16.7.3
   @nx/eslint-plugin  : 16.7.3
   @nrwl/tao          : 16.7.3
   @nx/webpack        : 16.7.3
   typescript         : 5.1.6
   ---------------------------------------
   Community plugins:
   @ngrx/effects         : 16.0.0
   @ngrx/router-store    : 16.0.0
   @ngrx/schematics      : 16.0.0
   @ngrx/store           : 16.0.0
   @ngrx/store-devtools  : 16.0.0
   @nguniversal/builders : 16.0.0
   ng-mocks              : 14.10.1

Failure Logs

> nx run frontend:lint

Linting "frontend"...

 >  NX   Ran target lint for project frontend (4m)

    ✖    1/1 failed
    ✔    0/1 succeeded [0 read from cache]
}

Operating System

• macOS
• Linux
• Windows
• Other (Please specify)

Additional Information

No response

[mcfdez]

Increasing Node's memory to 16GB completes the linting task, but it takes a long time, about an hour! If I run the eslint command manually, it takes much less time and consumes less memory resources.

[meeroslav]

This is an interesting problem, @mcfdez.

For us to investigate this, we need more information:

• Can you run nx lint ... with --verbose? This would hopefully return some logs on why the linting is failing
• Can you provide access to the repo where this can be reproduced?
• Why does your lint target config have "exclude": ["**/node_modules/**"]? Are you installing dependencies for that application separately from the root?
• How does your root .eslintrc.js look like? (please note that nx is only tested on JSON-based and flat config ESLint configurations)

[mcfdez]

Hello,

Running the npx nx lint frontend --verbose command, this also takes a very considerable time, even freezing without producing any output in which to see if there is a problem.

Unfortunately, we can't share the repository as I don't have permissions for it, but I can say that the Frontend application is an Angular 16 application with a few thousand files and in which, we have configured dozens of eslint plugins and hundreds of rules.

As for the configuration of **node_modules** in the Angular project.json, the truth is that we have only put it as a precaution, but we do not really know if it has any effect.

About the reason why we have decided to convert the ESLint configuration files from JSON to JS is because we need to use the __dirname variable in some cases.

Attached is the configuration of the base eslintrc.js from the repository.

module.exports = {
    root: true,
    ignorePatterns: ["**/*"],
    plugins: ["@nx"],
    overrides: [
      {
        files: ["*.ts", "*.tsx", "*.js", "*.jsx"],
        rules: {
          "@nx/enforce-module-boundaries": [
            "error",
            {
              "enforceBuildableLibDependency": true,
              "allow": [],
              "depConstraints": [
                {
                  "sourceTag": "*",
                  "onlyDependOnLibsWithTags": ["*"]
                }
              ]
            }
          ]
        }
      },
      /* {
        "files": ["*.ts", "*.tsx"],
        "extends": ["plugin:@nx/typescript"],
        "rules": {}
      }, */
      {
        files: ["*.js", "*.jsx"],
        extends: ["plugin:@nx/javascript"],
        rules: {}
      },
      {
        files: ["*.spec.ts", "*.spec.tsx", "*.spec.js", "*.spec.jsx", "*.test.module.ts"],
        env: {
          jest: true
        },
        rules: {}
      }
    ]
  }
  

Thanks

[meeroslav]

If you would run lint with eslint ... would that make any difference?

This should cover all the files

/code/angular> DEBUG=eslint:cli npx eslint **/*.ts **/*.html **/*.json

If that works, next you can run it with TIMING to see the report of the slowest rules:

/code/angular> TIMING=10 DEBUG=eslint:cli npx eslint **/*.ts **/*.html **/*.json

[mcfdez]

Hello,

If I run ESLint without NX, with the command "npx eslint ..." it takes much less time to linte the application and in case of errors, it displays them smoothly on screen.

The problem has only appeared running npx nx lint frontend with a Node memory lower than 16GB.

[meeroslav]

Can you share what the report of the TIMING=10 DEBUG=eslint:cli ... command was?

I'm interested to know which rules eat up most of your memory. You are likely using some expensive rules in those overrides (tools/linter/frontend/plugins/*), which might be skipped over when running with plain eslint or our linter does not know how to optimize them.

Can you also share what is in those override files?

On a side note, this:

      parserOptions: {
        tsconfigRootDir: __dirname + '/../..',
        project: ["tsconfig.base.json"]
      },

can be rewritten without __dirname so there would be no reason for using JS over simpler JSON:

      parserOptions: {
        project: ["../../tsconfig.base.json"]
      },

It's advised, though, not to use root config directly but rather point to the application's tsconfig, which then extends the root one. This allows you to optimize the config without modifying the root one.

[meeroslav]

Since you can't share access to the repo, I would be open for a short debugging/pairing session to investigate why the runs are slow. Let me know if this would be an option.

[mcfdez]

Hello

We have changed the .eslintrc.js files to .json again and correctly referenced the tsconfig.json of the applications instead of the tsconfig.base.json as you suggested.

The linter works correctly, but only with a large memory consumption (between 16 and 24GB). With lower memory, the task terminates with no result.

We have run the eslint command directly (without NX), and that way, as we have said before, it runs faster and if a failure occurs, we can see it.

We attach the execution of the output of the command with the TIMING parameter (we have put 100 but we could put a higher number since we have many rules).

TIMING=100 DEBUG=eslint:cli npx eslint --ext .ts code/apps/frontend -c code/apps/frontend/.eslintrc.json

import/namespace                                     | 60807.910 |    32.1%
deprecation/deprecation                              | 20561.542 |    10.8%
@typescript-eslint/no-confusing-void-expression      | 16651.817 |     8.8%
@typescript-eslint/indent                            | 12336.288 |     6.5%
no-secrets/no-secrets                                | 10333.178 |     5.5%
@typescript-eslint/no-extra-parens                   |  3736.607 |     2.0%
unused-imports/no-unused-imports                     |  3276.791 |     1.7%
import/no-relative-packages                          |  3021.768 |     1.6%
@nx/enforce-module-boundaries                        |  2719.596 |     1.4%
camelcase                                            |  2406.203 |     1.3%
comma-style                                          |  2381.304 |     1.3%
import/order                                         |  2362.514 |     1.2%
import/no-nodejs-modules                             |  1755.444 |     0.9%
@typescript-eslint/no-unnecessary-qualifier          |  1671.856 |     0.9%
import/extensions                                    |  1556.623 |     0.8%
no-trailing-spaces                                   |  1539.216 |     0.8%
@typescript-eslint/keyword-spacing                   |  1526.414 |     0.8%
import/no-deprecated                                 |  1486.687 |     0.8%
pii/no-phone-number                                  |  1386.998 |     0.7%
rxjs/no-exposed-subjects                             |  1332.056 |     0.7%
function-paren-newline                               |  1256.428 |     0.7%
@typescript-eslint/unbound-method                    |  1103.584 |     0.6%
import/named                                         |  1047.354 |     0.6%
@typescript-eslint/comma-dangle                      |  1043.883 |     0.6%
@typescript-eslint/func-call-spacing                 |  1017.889 |     0.5%
@typescript-eslint/object-curly-spacing              |   900.227 |     0.5%
max-len                                              |   854.543 |     0.5%
no-whitespace-before-property                        |   849.677 |     0.4%
object-curly-newline                                 |   804.792 |     0.4%
@typescript-eslint/no-misused-promises               |   783.984 |     0.4%
import/no-import-module-exports                      |   777.807 |     0.4%
semi-spacing                                         |   710.981 |     0.4%
@typescript-eslint/prefer-readonly                   |   706.160 |     0.4%
@typescript-eslint/no-loss-of-precision              |   702.885 |     0.4%
@typescript-eslint/space-infix-ops                   |   621.861 |     0.3%
jsdoc/no-types                                       |   606.490 |     0.3%
object-property-newline                              |   601.073 |     0.3%
@typescript-eslint/comma-spacing                     |   551.548 |     0.3%
pii/no-ip                                            |   551.436 |     0.3%
optimize-regex/optimize-regex                        |   549.543 |     0.3%
@typescript-eslint/await-thenable                    |   539.721 |     0.3%
rxjs/no-ignored-notifier                             |   501.726 |     0.3%
pii/no-email                                         |   488.706 |     0.3%
key-spacing                                          |   450.246 |     0.2%
semi-style                                           |   407.481 |     0.2%
@typescript-eslint/no-restricted-imports             |   386.168 |     0.2%
@typescript-eslint/brace-style                       |   371.313 |     0.2%
no-tabs                                              |   366.540 |     0.2%
no-useless-return                                    |   363.367 |     0.2%
@typescript-eslint/quotes                            |   337.391 |     0.2%
space-in-parens                                      |   336.548 |     0.2%
@typescript-eslint/semi                              |   331.386 |     0.2%
import/no-unresolved                                 |   330.661 |     0.2%
computed-property-spacing                            |   328.585 |     0.2%
no-octal-escape                                      |   326.558 |     0.2%
@typescript-eslint/no-unnecessary-type-assertion     |   321.526 |     0.2%
import/no-commonjs                                   |   320.610 |     0.2%
no-spaced-func                                       |   319.330 |     0.2%
no-useless-escape                                    |   293.974 |     0.2%
quote-props                                          |   282.854 |     0.1%
no-multi-str                                         |   270.114 |     0.1%
no-script-url                                        |   269.451 |     0.1%
import/newline-after-import                          |   268.450 |     0.1%
no-multi-spaces                                      |   261.409 |     0.1%
@typescript-eslint/no-invalid-this                   |   254.128 |     0.1%
import/no-self-import                                |   251.522 |     0.1%
@typescript-eslint/dot-notation                      |   244.461 |     0.1%
rxjs/no-unbound-methods                              |   235.725 |     0.1%
padded-blocks                                        |   220.907 |     0.1%
spaced-comment                                       |   208.496 |     0.1%
operator-linebreak                                   |   207.698 |     0.1%
no-global-assign                                     |   203.384 |     0.1%
block-spacing                                        |   202.468 |     0.1%
one-var                                              |   201.066 |     0.1%
no-alert                                             |   196.871 |     0.1%
function-call-argument-newline                       |   196.224 |     0.1%
import/export                                        |   192.381 |     0.1%
@typescript-eslint/no-empty-function                 |   181.529 |     0.1%
@typescript-eslint/type-annotation-spacing           |   176.730 |     0.1%
import/no-useless-path-segments                      |   175.307 |     0.1%
@typescript-eslint/no-use-before-define              |   173.505 |     0.1%
dot-location                                         |   166.969 |     0.1%
no-octal                                             |   166.140 |     0.1%
import/no-named-as-default-member                    |   162.051 |     0.1%
array-callback-return                                |   161.541 |     0.1%
@typescript-eslint/no-implied-eval                   |   159.253 |     0.1%
@typescript-eslint/adjacent-overload-signatures      |   155.489 |     0.1%
@typescript-eslint/no-unused-expressions             |   155.030 |     0.1%
space-before-blocks                                  |   153.400 |     0.1%
@typescript-eslint/no-base-to-string                 |   153.256 |     0.1%
@typescript-eslint/non-nullable-type-assertion-style |   152.669 |     0.1%
no-floating-decimal                                  |   148.466 |     0.1%
@typescript-eslint/prefer-optional-chain             |   145.371 |     0.1%
no-multiple-empty-lines                              |   145.180 |     0.1%
no-restricted-properties                             |   142.316 |     0.1%
import/no-duplicates                                 |   136.240 |     0.1%
@typescript-eslint/no-shadow                         |   136.012 |     0.1%
no-eval                                              |   133.947 |     0.1%
no-mixed-spaces-and-tabs                             |   133.860 |     0.1%
linebreak-style                                      |   130.940 |     0.1%

As for the contents of the various linter files, we have many and with many rules, but we attach a screenshot of the package.json to give you an idea of the number of plugins we are using.

"eslint": "8.47.0",
"eslint-config-prettier": "9.0.0",
"eslint-import-resolver-typescript": "3.6.0",
"eslint-plugin-cypress": "2.14.0",
"eslint-plugin-deprecation": "1.5.0",
"eslint-plugin-import": "2.28.1",
"eslint-plugin-jest": "27.2.3",
"eslint-plugin-jsdoc": "46.5.0",
"eslint-plugin-json": "3.1.0",
"eslint-plugin-jsx-a11y": "6.7.1",
"eslint-plugin-no-secrets": "0.8.9",
"eslint-plugin-optimize-regex": "1.2.1",
"eslint-plugin-pii": "1.0.2",
"eslint-plugin-prefer-arrow": "1.2.3",
"eslint-plugin-prettier": "5.0.0",
"eslint-plugin-react": "7.32.2",
"eslint-plugin-react-hooks": "4.6.0",
"eslint-plugin-rxjs": "5.0.3",
"eslint-plugin-unused-imports": "3.0.0",

Finally, about the possibility of doing a joint session, we will talk about it in the team.

Thank you very much!

[github-actions]

This issue has been automatically marked as stale because it hasn't had any recent activity. It will be closed in 14 days if no further activity occurs.
If we missed this issue please reply to keep it active.
Thanks for being a part of the Nx community! 🙏

[meeroslav]

Can you try running lint with with enough memory to produce results when run with @nx/linter and prefix it with TIMING=100 to see what results it produces?

Btw. your command listed above is not the same as what @nx/linter uses for apps is using since you are skipping all the .html files. It's very likely that if you would have the exact match of commands, the pure eslint run would not be significantly faster.

My suggestion is to get rid of import/namespace and deprecation/deprecation and look into other rules that take over 5% of your lint run. Both rules are notoriously slow (check import-js/eslint-plugin-import#2340 and gund/eslint-plugin-deprecation#44).

Slow rules are not worth the check.

[github-actions]

This issue has been closed for more than 30 days. If this issue is still occuring, please open a new issue with more recent context.