Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

skip-gitignore will check files not in the git repository #1772

Closed
ucodery opened this issue Jul 5, 2021 · 9 comments
Closed

skip-gitignore will check files not in the git repository #1772

ucodery opened this issue Jul 5, 2021 · 9 comments

Comments

@ucodery
Copy link
Collaborator

ucodery commented Jul 5, 2021

When isort asks git to check which files it ignores, isort can ask git about files not under revision tracking, due to symlinks. Even though isort gets the root of the git repository from a prior git execution, it then does a recursive glob which can resolve symlinked directories and add files and sub-directories that are not part of the repository.

When this happens, the call to find out what files should be ignored gets very slow, slower even than just running isort over a larger code base. git will spew out fatal: pathspec '/some/external/path' is beyond a symbolic link for all of the extra paths.

@ucodery
Copy link
Collaborator Author

ucodery commented Jul 5, 2021

Also, some food for thought, black will ignore any file that is on the other side of a symlink that points outside of the project's root. It might be good for isort to take the same line and always ignore these sorts of files.
isort does have an older userbase, and works in more than just git repositories, so this might break some workflows though.

@timothycrosley
Copy link
Member

Closing this as your fix has been merged in the main branch, and will be released soon (in 5.9.2) 🎉 - thank you!

It might be good for isort to take the same line and always ignore these sorts of files.

I'm open to doing this in the future!

@adamjstewart
Copy link

adamjstewart commented Jul 11, 2021

I just ran into this same issue! Unfortunately, isort 5.9.2 doesn't seem to solve the problem for me.

Specifically, I have a Spack environment in my project directory. The .gitignore contains .spack-env/, and git correctly ignores everything in this directory. However, when I run isort, it tries to reformat files in a completely different root directory because it follows symlinks in .spack-env/.

I thought #1773 was designed specifically to address this issue? Is my case somehow slightly different? For now, adding .spack-env/ to extend_skip seems to work.

@timothycrosley
Copy link
Member

@adamjstewart Another option would be to use the dont_follow_links option.

@ucodery
Copy link
Collaborator Author

ucodery commented Jul 12, 2021

@adamjstewart interesting, #1773 was meant to address scenarios like the one you describe as well. Are you explicitly using --skip-gitignore?

@adamjstewart
Copy link

Yes, I have skip_gitnore = true in my pyproject.toml. I also tried specifying --skip-gitignore on the command-line.

@ucodery
Copy link
Collaborator Author

ucodery commented Jul 12, 2021

It's possible that this fix is incomplete. If you wouldn't mind could you send the full config you are using, as well as the complete output from your run of isort? (Also isort --show-files would be helpful).

@adamjstewart
Copy link

Here is a minimal working reproducer. First, we create some files:

$ cd
$ cat > a.py
# Incorrectly ordered imports
import numpy
import os
$ mkdir foo
$ cd foo
$ cp ../a.py b.py
$ ln -s ../a.py c.py
$ ls -l
total 4.0K
-rw-r----- 1 t-astewart t-astewart 53 Jul 12 15:06 b.py
lrwxrwxrwx 1 t-astewart t-astewart  7 Jul 12 15:06 c.py -> ../a.py

Then create a git repo:

$ git init
Initialized empty Git repository in /home/t-astewart/foo/.git/
$ cat > .gitignore
c.py
$ git status
On branch master

No commits yet

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        .gitignore
        b.py
        pyproject.toml

nothing added to commit but untracked files present (use "git add" to track)

Then configure and run isort:

$ cat > pyproject.toml
[tool.isort]
skip_gitignore = true
$ isort --vn
5.9.2
$ isort . --check
ERROR: /home/t-astewart/foo/b.py Imports are incorrectly sorted and/or formatted.
ERROR: /home/t-astewart/a.py Imports are incorrectly sorted and/or formatted.
Skipped 1 files
$ isort . --show-files
./b.py
./c.py

I would expect the symlink c.py to be ignored. It is correctly ignored by git, but not by isort when skip_gitignore is set.

@ucodery
Copy link
Collaborator Author

ucodery commented Jul 20, 2021

@adamjstewart I have submitted #1789 which fixes some issues, but opened #1788 to address the larger issue.
The problem is not strictly that gitignore wasn't bing consulted here, but that isort traverses directories passed to it, before even asking gitignore if it should skip. Because the list of files is gathered before checking gitignore --dont-follow-links is really the only way forward right now in some circumstances.

If you have any further concerns let's take it to #1788.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants