Speeding up Git using a built-in file system watcher

[dscho]

tl;dr If your worktrees have many files, you can make git status, git commit and git add faster by using the (experimental!) config setting core.useBuiltinFSMonitor = true. For users’ convenience, there is a checkbox in the upcoming Git for Windows v2.32.0 installer to do that.

With this config setting, Git commands that want to refresh Git’s index will query the built-in file system watcher (“FSMonitor”) which files have been changed, and only look at those. When there are tens of thousands of files in the worktree, this makes a difference because Git would otherwise have to look at the lastModified times of all files (“full scan”) to figure out which ones were modified since Git looked last. If the FSMonitor is not yet running, it is automatically started, and subsequent Git commands will benefit from it.

Background

Over sixteen years ago, when Git was invented to replace the version control system used by the Linux kernel project, the fastest way to determine which worktree files were modified vs which ones were unchanged was to use Linux’ ultra-fast lstat() call, essentially to compare the lastModified times to the ones recorded in Git’s index (and also taking the lastModified time of the Git index itself into account).

This worked pretty well, even for the source code of the Linux kernel project (which consisted of a little over 17k files at the time).

Today, many projects in the software industry have worktrees containing many, many more files. Combined with the different performance characteristics of NTFS, refreshing the index in such worktrees can take minutes!

Using file system watchers

The obvious idea how to accelerate refreshing Git’s index is to avoid looking at all files. Instead, run a watcher that asks the operating system to be notified whenever a file or directory within a certain directory tree was modified, added or deleted. This watcher can be queried, and Git can look at the lastModified times of just that set of paths.

This changes the time complexity taken by a git status invocation from the order of total file count to the order of the number of modified/added/deleted files.

Git supports the use of file system watchers since v2.16.0, when it introduced the config setting core.fsmonitor. This setting points to a script or executable that talks to the file system watcher.

So why is this core.fsmonitor feature not in more wide-spread use?

There are multiple explanations, the most prominent one being: it takes effort to set up a file system monitor. Watchman, the most common file system monitor, does not regularly release Windows binaries and it is difficult to build from source for many platforms.

Another reason is that the speed still leaves much to be desired, even when using a file system watcher, because Git has to spawn a hook whenever it wants to refresh Git’s index. Spawning processes is expensive on Windows, in particular when the process in question is the Perl interpreter that then has to load a JSON library to manage the Watchman query.

As a consequence of these hurdles, adoption of this feature has been low, and therefore several bugs have been lurking in the code base for years, undetected.

A built-in file system watcher

In 2019, we set out to change the game. We started to implement a file system watcher that is integrated into Git itself (“built-in”). This solves not only the build issues, but also lowers the bar to enable the feature by quite some margin. It is now as easy as setting the config option core.useBuiltinFSMonitor to true.

Another, crucial improvement is that Git can talk to the built-in file system watcher directly, without having to spawn a new process. This speeds up the common case noticeably.

Finally, it allows the file system monitor to be Git-aware. Things like excluding .git from being watched automatically are much, much easier that way.

The built-in file system watcher is a direct outcome of the Scalar project, a project whose mission is to bring as many of the improvements developed and refined in VFS for Git as possible to Git proper. Some things can obviously not be moved: the virtual file system aspect was never accepted in the Git project, and it also turned out to be impractical e.g. with recent macOS versions. Using a file system watcher to accelerate Git index refreshes, however, provided too nice of a benefit to pass up on, hence the idea to implement a Git native one.

Acknowledgements

This feature is the culmination of many years of effort, by many developers. First, Ben Peart introduced the core.fsmonitor setting to allow for running a hook that talks to Watchman (developed at Facebook), and almost as importantly also worked with the Watchman development team to make it run better on Windows. Alex Vandiver contributed to the effort by making the feature more robust. Johannes Schindelin started implementing the built-in file system watcher, Kevin Willford continued the work on it, and then Jeff Hostetler took over, reworking and redesigning the parts that needed it.

[jeffhostetler]

Nicely said! Thank you for writing this up!

Let me add that the file system watcher also greatly speeds up untracked file detection when the "untracked cache" extension is enabled. This is in addition to the lstat()/mtime gains described above for tracked files listed in the index.

[Listwon]

Caveat for Unity users wanting to try it out right now: it breaks Unity Package Manager so you can't add packages from git repositories (via URL) to your project.
https://forum.unity.com/threads/resource-blocked-when-installing-post-processing-package.827199/

[jeffhostetler]

I'm not familiar with Unity, so I'll have to ask some dumb questions here.

Does the package manager create temporary Git repos down in .tmp*, copy out whatever files it needs, and then delete the Git repos?

If so, and if a FSMonitor daemon process got started in that temporary workdir, then yes, I could see the PM getting an error when it tries to delete the workdir root directory -- since the FSMonitor daemon's CWD is there.

I'm working on a fix to have the daemon CD out of the workdir after it gets started to avoid locking the directory. Hopefully, that will take care of the problem. I already have code in there to shutdown the daemon if/when the .git directory is deleted (which should happen before the root dir is deleted), but that just means we're racing to the finish.....

Thanks for the report.

[Listwon]

This is not a dumb question and it works exactly as you described (at least that's what I saw investigating what suddenly broke one of my Unity projects).

[jeffhostetler]

Thanks for the confirmation!

[AshaSalorina]

I'm not familiar with Unity, so I'll have to ask some dumb questions here.

Does the package manager create temporary Git repos down in .tmp*, copy out whatever files it needs, and then delete the Git repos?

If so, and if a FSMonitor daemon process got started in that temporary workdir, then yes, I could see the PM getting an error when it tries to delete the workdir root directory -- since the FSMonitor daemon's CWD is there.

I'm working on a fix to have the daemon CD out of the workdir after it gets started to avoid locking the directory. Hopefully, that will take care of the problem. I already have code in there to shutdown the daemon if/when the .git directory is deleted (which should happen before the root dir is deleted), but that just means we're racing to the finish.....

Thanks for the report.

Hello but, I also encountered the problem in the latest version, is there any progress on it now?

[jeffhostetler]

I have fixed this in my V4 version of FSMonitor.
It will be included in the v2.34 release of git-for-windows.
(We shipped a preview of V4 in v2.33 of git-for-windows.)

[SetTrend]

Out of curiosity: The Windows kernel comes with a built-in file system watcher:

https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-findfirstchangenotificationw

Does the Scalar project use that? If not, wouldn't it have been beneficial to use the built-in Windows watcher?

[dscho]

Quoting from the linked page:

To retrieve information about the specific change as part of the notification, use the
ReadDirectoryChangesW function.

It is the ReadDirectoryChangesW() function that is used by the Windows-specific backend of the built-in FSMonitor.

And: it is not the Scalar project that uses it. The built-in FSMonitor is in Git itself. So "Does the Scalar project use that?" is the wrong question.

[mastercoms]

So an actual bug (maybe, please let me know!): core.useBuiltinFSMonitor=true detects LFS checkout as changes in the working tree. This happens when you disable built in Git LFS filters during a pull (git -c filter.lfs.smudge= -c filter.lfs.process= -c filter.lfs.required=false merge --ff-only origin/main) and then do a git lfs pull (or git lfs fetch;git lfs checkout). This will mark the files which were LFS pointers during the merge, but then updated to LFS objects to be changed in git status. This does not happen with the FS monitor disabled.

git version 2.33.0.vfs.0.0

git-lfs/2.13.3 (GitHub; windows amd64; go 1.16.2; git a5e65851)

I'm not sure if this should be raised to an issue, please let me know.

[dscho]

Hmmm. Maybe if you could provide a minimal example, it might help the discussion.

[mastercoms]

I'm not sure what additional things I should give, I just ran:

git -c filter.lfs.smudge= -c filter.lfs.process= -c filter.lfs.required=false merge --ff-only origin/main
git lfs pull

With FS monitor on, this does not work. With it off, it does work.

If I modify LFS to not run git update-index -q --refresh, then neither works. It's run here: https://github.com/git-lfs/git-lfs/blob/main/commands/pull.go#L144

[dscho]

That description requires a cloned repository with Git LFS objects. Not everybody has such a repository handy, and that makes it cumbersome to reproduce. I, for one, lack the time and the inclination to pursue this any further.

[mastercoms]

Ah ok, so I should make a repo with LFS files along with a script which does this behavior? Will get that to you asap, sorry for not getting what you meant!

[mastercoms]

I will try to get around to this still, but we haven't found a need for this workflow anymore with the latest Git LFS update, so it's a low priority for us to investigate.

[vchirikov]

@dscho

It makes me crazy..
How I can disable this file-locking feature(at least for one repo)???

core.useBuiltinFSMonitor doesn't work at all, any git command runs git ("C:\Program Files\Git\mingw64\libexec\git-core\git.exe" fsmonitor--daemon run --free-console --ipc-threads=8)

argh.

[jeffhostetler]

Try

git config --list --show-origin | grep -i fsmonitor

This will tell you whether the core.useBuiltinFSMonitor value is set and whether it is set locally or globally.

BTW do you have feature.experimental set? That will also turn on FSMonitor.
https://github.com/git-for-windows/git/blob/main/repo-settings.c#L75

[vchirikov]

git config --list --show-origin | grep -i fsmonitor

Did you see the previous screenshot? the command was there :)

[feature]
    experimental = true
    manyFiles = true

Finally! Thanks @jeffhostetler, I will disable it. Could we have a warning about this as we have in git update-index --no-fsmonitor ? I think it's quite useful

--

p.s. now works fine, thanks.

[derrickstolee]

This means we have a bug that features.experimental is overriding the core.useBuiltinFSMonitor, which is backwards. @jeffhostetler can you look into fixing that?

[jeffhostetler]

This was added in 3bcb1a2
in a GFW-only commit as part of the inclusion of this experimental feature.

Looking at the nearby values that use the UPDATE_DEFAULT_BOOL() they only force set the value if unset in the config. We should do that here too.

[jeffhostetler]

I have fixed this in my V4 version of FSMonitor.
It will be included in the v2.34 release of git-for-windows.

[yhvicey]

Sometimes when I tried to remove some files from repo within editors (e.g. vscode), this monitor is blocking system from removing these files from disk, which caused git thinking I haven't remove these files - not sure if it's caused by this monitor only, or the combination of vscode + rename files action + fsmonitor.

[yhvicey]

And after recalled what I observed, I remember the files were deleted from disk actually, however the git status didn't get correct result, until I git rm --cached ... - So maybe this is not caused by file locking, but VS Code impacting Git for Windows' fsmonitor?

[dscho]

So maybe this is not caused by file locking, but VS Code impacting Git for Windows' fsmonitor?

That should not be possible. Unless we lose events, in which case FSMonitor should fall back to a full scan.

[yhvicey]

Is there any log/verbose logging switch for fsmonitor? If so I can try re-produce the issue and attach log here.

[jeffhostetler]

There are a couple of settings/knobs to turn on. The best way to set this up is to stop the daemon in your repo, set these envvars, and explicitly start it in a terminal. You can set both of these envvars to the same path and the output will be properly interleaved.
Then in another terminal window, try modifying files and running various Git commands.

$ git fsmonitor--daemon stop
$ export GIT_TRACE2_PERF=/an/absolute/pathname/for/daemon/log
$ export GIT_TRACE_FSMONITOR=/an/absolute/pathname/for/daemon/log
$ git fsmonitor--daemon run

It might help to do something similar on the client side:

$ export GIT_TRACE2_PERF=/an/absolute/pathname/for/client
$ export GIT_TRACE_FSMONITOR=/an/absolute/pathname/for/client
$ git status

and then you can see both halves of the conversation.

Let me know if you have questions or problems getting this working.
And thanks for your interest in helping make FSMonitor better.
Jeff

[dscho]

Please test the latest snapshot.

[mastercoms]

Seems like using git restore --source commit -- file then git restore -- file keeps the modified state in working tree until git update-index -- file is called. This does not happen without file system monitor. Where commit is a target commit which results in the file being changed.

[dscho]

Please test the latest snapshot.

[jeffhostetler]

@mastercoms I tried your test script with GFW v2.33.0.windows.2 and could not reproduce it. I get identical output whether FSMonitor is enabled or disabled.

I also did not see any effect from the checkout.workers = 2. I wouldn't expect parallel checkout to kick in without a larger test repo or until checkout.thresholdForParallelism is lowered.

Please let me know if the problem persists. Thanks.

[mastercoms]

Hi, apologies for wasting your time. I initially didn't properly disable filesystem monitor while comparing, but did so on my investigation this time around. Now I know this behavior happens whether or not filesystem monitor is enabled, and is solely related to checkout.workers. I'll move this report elsewhere once I've found the right place for it, but I've kept it here for posterity.

$ git version
git version 2.33.0.windows.2

On the current commit which sets checkout.workers=2

$ ./test.sh
Setting checkout.workers=2
Restoring to old commit
On branch main
Your branch is up to date with 'origin/main'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   test.txt

no changes added to commit (use "git add" and/or "git commit -a")
Restoring to HEAD
On branch main
Your branch is up to date with 'origin/main'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   test.txt

no changes added to commit (use "git add" and/or "git commit -a")
Updating index
On branch main
Your branch is up to date with 'origin/main'.

nothing to commit, working tree clean

This is the problematic behavior, as restoring to HEAD shows unstaged changes to test.txt, needing an update index.

On a commit that sets checkout.workers=0 in test.sh:

Setting checkout.workers=0
Restoring to old commit
On branch main
Your branch is ahead of 'origin/main' by 1 commit.
  (use "git push" to publish your local commits)

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   test.txt

no changes added to commit (use "git add" and/or "git commit -a")
Restoring to HEAD
On branch main
Your branch is ahead of 'origin/main' by 1 commit.
  (use "git push" to publish your local commits)

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   test.txt

no changes added to commit (use "git add" and/or "git commit -a")
Updating index
On branch main
Your branch is ahead of 'origin/main' by 1 commit.
  (use "git push" to publish your local commits)

nothing to commit, working tree clean

On a commit that sets checkout.workers=1:

$ ./test.sh
Setting checkout.workers=1
Restoring to old commit
On branch main
Your branch is ahead of 'origin/main' by 1 commit.
  (use "git push" to publish your local commits)

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   test.txt

no changes added to commit (use "git add" and/or "git commit -a")
Restoring to HEAD
On branch main
Your branch is ahead of 'origin/main' by 1 commit.
  (use "git push" to publish your local commits)

nothing to commit, working tree clean
Updating index
On branch main
Your branch is ahead of 'origin/main' by 1 commit.
  (use "git push" to publish your local commits)

nothing to commit, working tree clean

This is the expected behavior, as restoring to HEAD shows no unstaged changes to test.txt, and there is no need to update index.

I wouldn't expect parallel checkout to kick in without a larger test repo or until checkout.thresholdForParallelism is lowered.

Me neither, but my config option for it is unset, so very strange to me.

This exact behavior also happens on the latest snapshot:

$ git version
git version 2.33.0.windows.2.126.g988794267c.20211004113703

$ ./test.sh
Setting checkout.workers=2
Restoring to old commit
On branch main
Your branch is up to date with 'origin/main'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   test.txt

no changes added to commit (use "git add" and/or "git commit -a")
Restoring to HEAD
On branch main
Your branch is up to date with 'origin/main'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   test.txt

no changes added to commit (use "git add" and/or "git commit -a")
Updating index
On branch main
Your branch is up to date with 'origin/main'.

nothing to commit, working tree clean

This exact behavior also happens on the latest microsoft/git release:

$ git version
git version 2.33.0.vfs.0.0

$ ./test.sh
Setting checkout.workers=2
Restoring to old commit
On branch main
Your branch is up to date with 'origin/main'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   test.txt

no changes added to commit (use "git add" and/or "git commit -a")
Restoring to HEAD
On branch main
Your branch is up to date with 'origin/main'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   test.txt

no changes added to commit (use "git add" and/or "git commit -a")
Updating index
On branch main
Your branch is up to date with 'origin/main'.

nothing to commit, working tree clean

[jeffhostetler]

Thanks for the follow up. You could report it via "git bugreport". It should open VIM and let you fill in the details. Then you can send the resulting file to the Git mailing list. See https://git.wiki.kernel.org/index.php/GitCommunity

I'd also suggest adding a "set -x" to your test script to echo the commands as executed. That makes it easier to follow along.

Thanks again!

[mastercoms]

Oh, thank you for the guidance! I have never submitted a bug to Git before 😅

[hello-smile6]

Nice feature!

[jeffhostetler]

Note: upstream has decided on core.fsmonitor = true instead of core.useBuiltinFSMonitor = 1, so we should retool some of the documentation here.

[jeffhostetler]

In the port of 2.36.0-rc0 we added a patch to continue to support core.useBuiltinFSMonitor and add a deprecation warning.
So we don't necessarily need to reword this discussion topic.

[OannesGPT]

Thank you!! This is just amazing.

[xd003]

Just so i am understanding this correctly, i just need to add the following in my .gitconfig right ?

[core]
	fsmonitor = true

[dscho]

Better do it in a more fine-grained way, repository by repository. But yes, you can also configure it in the user-wide config.

[HabunoGD1809]

great!