Large memory usage, memory leaks

[2-5]

When vscode-pylint introspects an object, it seems to keep a reference to it alive.

If the object is very large, vscode-pylint will quickly consume memory. This happens especially in the context of Jupyter notebooks, where is usual to work with large data objects, but can happen with regular .py files too:

1. create new Python file with this content:

x = b"a" * 4_000_000_000  # 4 GB bytes object

2. open it in VS Code with Pylint extension active
3. notice how Pylint extension process (bundled\tool\lsp_server.py) now uses 4 GB of RAM
4. close above Python file in VS Code
5. notice that Pylint extension didn't release memory, it's still using 4 GB of RAM

[karthiknadig]

@2-5 This is a pylint issue, you may have to file it in the https://github.com/PyCQA/pylint .

If you are using the latest version of pylint extension, then it includes pylint (v2.16.1) which added a flag to clean up caching on each request. We use that flag in the extension. If you are seeing this with the latest stable version of the extension, 2023.2.0, then this should be reported to https://github.com/PyCQA/pylint

There is a workaround, you will need the following:

1. A virtual environment with pylint 2.16.1 installed.
2. Use the following setting, "pylint.path": ["${workspaceFolder}/.venv/bin/pylint"] if your virtual env is in your workspace, or "pylint.path": ["<absolute path to pylint executable>"]

This is less performant, but it should be able to run pylint as you would from the command line, instead of a server like mode like we normally do in the extension.

[bersbersbers]

I can reproduce this issue, and I filed pylint-dev/pylint#8361.

[bersbersbers]

I am seeing this issue several times per day, with memory use approaching several gigabytes even without the OP's reproduction steps. pylint-dev/pylint#8361 is carefully pessimistic this can be solved, I would say:

IIUC Python doesn't always release memory back to the OS even after manual garbage collection.

The workaround using "pylint.path" seems to work for now.

[bersbersbers]

fyi, pylint-dev/pylint#8361 has been closed as a duplicate of pylint-dev/astroid#1780.

[bersbersbers]

Some work in pylint-dev/pylint#8361 has landed in pylint 2.17.1, but that does not seem to have fixed the issue. Let's hope for pylint-dev/astroid#1780.

(Tested with pylint 2.17.1/astroid 2.15.0.)

[bersbersbers]

2. Use the following setting, "pylint.path": ["${workspaceFolder}/.venv/bin/pylint"] if your virtual env is in your workspace, or "pylint.path": ["<absolute path to pylint executable>"]

One problem with this workaround is that pylint ignores my pyproject.toml file when using it. I use

"pylint.path": [
    "C:\\Users\\bers\\.pyenv-win-venv\\envs\\project_3.11\\Scripts\\pylint.exe",
],

and that doesn't seem to use my C:\code\project\pyproject.toml when linting C:\code\project\project\project.py, while it does when linting C:\code\project\project.py.

An extended workaround is to add something like

"pylint.args": [
    "--rcfile=C:\\Code\\project\\pyproject.toml",
],

to each project.

[karthiknadig]

@bersbersbers You can use ${workspaceFolder} inside pylint.args as well.

As for pylint ignoring pyproject.toml, we launch pylint binary with cwd set to the workspace directory. Not sure why it can't find it.

[bersbersbers]

@bersbersbers You can use ${workspaceFolder} inside pylint.args as well.

Good idea, thanks.

Anyway, there is now another workaround from pylint-dev/pylint#8361 (comment), which involves adding only

    "pylint.args": [
        "--clear-cache-post-run",
    ],

As for pylint ignoring pyproject.toml, we launch pylint binary with cwd set to the workspace directory. Not sure why it can't find it.

I haven't investigated that in more depth, but my guess is that this may be related to the same underlying reason why adding --clear-cache-post-run is necessary. In fact, in lsp_server.py's _linting_helper() I can see the following variables:

2023-03-23 03:05:21.579 [info] document = file:///c%3A/Code/project/project/bug.py
2023-03-23 03:05:21.579 [info] code_workspace = c:\Code\project\project
2023-03-23 03:05:21.580 [info] VERSION_TABLE = {'c:\\code\\project': (2, 17, 1)}

leading to VERSION_TABLE.get(code_workspace, None) being None, and no "--clear-cache-post-run=y" being added.

The folder open in VS Code is C:\\Code\\project, so VERSION_TABLE is correct except for capitalization, while code_workspace looks incorrect.

I tried storing bug.py in the workspace - still does not work, probably due to capitalization:

2023-03-23 03:18:13.231 [info] document = file:///c%3A/Code/project/bug.py
2023-03-23 03:18:13.238 [info] code_workspace = c:\Code\project
2023-03-23 03:18:13.238 [info] VERSION_TABLE = {'c:\\code\\project': (2, 17, 1)}

The same is true when I edit an out-of-workspace file:

2023-03-23 03:08:25.312 [info] document = file:///c%3A/Users/bers/.vscode/extensions/ms-python.pylint-2023.5.10801010/bundled/tool/lsp_server.py
2023-03-23 03:08:25.312 [info] code_workspace = c:\Users\bers\.vscode\extensions\ms-python.pylint-2023.5.10801010\bundled\tool
2023-03-23 03:08:25.313 [info] VERSION_TABLE = {'c:\\code\\project': (2, 17, 1)}

Again - VERSION_TABLE.get(code_workspace, None) is None, and no "--clear-cache-post-run=y" is added.

Not sure what a good fix would be, but for all three files, I'd expect "--clear-cache-post-run=y" to be added based on the version of pylint ending up being used for that file.

By the way, if major == 2 and minor >= 16 would not include pylint 3, not sure if that is intended.

[bersbersbers]

More details and repro steps for the above issue can be found in #332.

[karthiknadig]

This PR #349 should address the path issue that caused --clear-cache-post-run to skip in some cases.

The leak in pylint should be addressed with that.

If you have to use the --rcfile setting, then you can set it relative to your project:

"pylint.args": ["--rcfile=${workspaceFolder}/pyproject.toml"]