Git add with auto_lfs_track should track binary files

[osanseviero]

When pushing binary files, users get the following error

remote: Your push was rejected because it contains binary files.
remote: Please use https://git-lfs.github.com/ to store binary files.

With auto LFS track this should be tracked by default.

WDYT @LysandreJik @julien-c ?

[julien-c]

i don't remember the auto_lfs_track implementation, is there a file size threshold somewhere? We changed the file size threshold for binary files only to 1MB on the hub side recently. Non binary files still 10MB

[osanseviero]

Yes, we set a 10mb auto track at https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/repository.py#L849-L853

We could do some similar check for binary files for 1MB?

[julien-c]

yes!

and sorry about not thinking of this when we changed the thresholds on the hub side (@Pierrci @SBrandeis @coyotte508 )

[LysandreJik]

Hey @osanseviero, indeed, happy to do that. Do you have a reproducible code example I can play with?

[osanseviero]

Yes @LysandreJik.

from huggingface_hub import Repository, HfApi
import os

repo_url = HfApi().create_repo("test-bin-bug")
repo = Repository("local_repo", clone_from=repo_url)
with open(os.path.join("local_repo", "file"), "wb") as out:
    out.truncate(1024*1024)
repo.push_to_hub("Commit #1")

[Pierrci]

@LysandreJik you can check https://github.com/huggingface/moon-landing/blob/5fa4de96cb48dc0613933d2aa7e3311bbd0abaac/server/scripts/git/pre-receive#L143 for what we use server-side to detect binary files (cc @coyotte508 who implemented it)

[LysandreJik]

Will be closed by #828