skip-gitignore will check files not in the git repository

[ucodery]

When isort asks git to check which files it ignores, isort can ask git about files not under revision tracking, due to symlinks. Even though isort gets the root of the git repository from a prior git execution, it then does a recursive glob which can resolve symlinked directories and add files and sub-directories that are not part of the repository.

When this happens, the call to find out what files should be ignored gets very slow, slower even than just running isort over a larger code base. git will spew out fatal: pathspec '/some/external/path' is beyond a symbolic link for all of the extra paths.

[ucodery]

Also, some food for thought, black will ignore any file that is on the other side of a symlink that points outside of the project's root. It might be good for isort to take the same line and always ignore these sorts of files.
isort does have an older userbase, and works in more than just git repositories, so this might break some workflows though.

[timothycrosley]

Closing this as your fix has been merged in the main branch, and will be released soon (in 5.9.2) 🎉 - thank you!

It might be good for isort to take the same line and always ignore these sorts of files.

I'm open to doing this in the future!

[adamjstewart]

I just ran into this same issue! Unfortunately, isort 5.9.2 doesn't seem to solve the problem for me.

Specifically, I have a Spack environment in my project directory. The .gitignore contains .spack-env/, and git correctly ignores everything in this directory. However, when I run isort, it tries to reformat files in a completely different root directory because it follows symlinks in .spack-env/.

I thought #1773 was designed specifically to address this issue? Is my case somehow slightly different? For now, adding .spack-env/ to extend_skip seems to work.

[timothycrosley]

@adamjstewart Another option would be to use the dont_follow_links option.

[ucodery]

@adamjstewart interesting, #1773 was meant to address scenarios like the one you describe as well. Are you explicitly using --skip-gitignore?

[adamjstewart]

Yes, I have skip_gitnore = true in my pyproject.toml. I also tried specifying --skip-gitignore on the command-line.

[ucodery]

It's possible that this fix is incomplete. If you wouldn't mind could you send the full config you are using, as well as the complete output from your run of isort? (Also isort --show-files would be helpful).

[adamjstewart]

Here is a minimal working reproducer. First, we create some files:

$ cd
$ cat > a.py
# Incorrectly ordered imports
import numpy
import os
$ mkdir foo
$ cd foo
$ cp ../a.py b.py
$ ln -s ../a.py c.py
$ ls -l
total 4.0K
-rw-r----- 1 t-astewart t-astewart 53 Jul 12 15:06 b.py
lrwxrwxrwx 1 t-astewart t-astewart  7 Jul 12 15:06 c.py -> ../a.py

Then create a git repo:

$ git init
Initialized empty Git repository in /home/t-astewart/foo/.git/
$ cat > .gitignore
c.py
$ git status
On branch master

No commits yet

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        .gitignore
        b.py
        pyproject.toml

nothing added to commit but untracked files present (use "git add" to track)

Then configure and run isort:

$ cat > pyproject.toml
[tool.isort]
skip_gitignore = true
$ isort --vn
5.9.2
$ isort . --check
ERROR: /home/t-astewart/foo/b.py Imports are incorrectly sorted and/or formatted.
ERROR: /home/t-astewart/a.py Imports are incorrectly sorted and/or formatted.
Skipped 1 files
$ isort . --show-files
./b.py
./c.py

I would expect the symlink c.py to be ignored. It is correctly ignored by git, but not by isort when skip_gitignore is set.

[ucodery]

@adamjstewart I have submitted #1789 which fixes some issues, but opened #1788 to address the larger issue.
The problem is not strictly that gitignore wasn't bing consulted here, but that isort traverses directories passed to it, before even asking gitignore if it should skip. Because the list of files is gathered before checking gitignore --dont-follow-links is really the only way forward right now in some circumstances.

If you have any further concerns let's take it to #1788.