Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MSN (Masked Siamese Networks) for ViT #18815

Merged
merged 30 commits into from Sep 22, 2022
Merged

MSN (Masked Siamese Networks) for ViT #18815

merged 30 commits into from Sep 22, 2022

Conversation

sayakpaul
Copy link
Member

@sayakpaul sayakpaul commented Aug 30, 2022 •
edited

What does this PR do?

Adds the MSN checkpoints for ViT. MSN shines in the few-shot regimes which would benefit real-world use cases. Later we could add a pre-training script so that people can actually perform pre-training with MSN with their own datasets.

Closes #18758

Who can review?

@sgugger @NielsRogge @amyeroberts

TODO

  • Add documentation
  • Add rest of the files for repo consistency
  • Host MSN weights on the Facebook org on HF Hub (@NielsRogge ?)
  • Change the checkpoint paths wherever needed

@HuggingFaceDocBuilderDev
Copy link

HuggingFaceDocBuilderDev commented Aug 30, 2022 •
edited

The documentation is not available anymore as the PR was closed or merged.

@sayakpaul
Copy link
Member Author

@NielsRogge, after studying the pretraining script of MSN thoroughly I am still unsure of how to put together a ViTMSNForPretraining similar to ViTMAEForPreTraining. There are multiple moving pieces that I think are best off residing inside a standalone pretraining script:

Both the EMA and sharpening components operate with their own schedules.

Given this, I think it's best to resort to a separate pre-training script and use this model for feature extraction and fine-tuning.

There's an ongoing discussion around releasing the weights of the linear classification layers and fine-tuned models. So when that's available, we could directly support those via ViTMSNForImageClassification. Regardless, I am happy to add a ViTMSNForImageClassification for easy access.

What do you think?

@sgugger
Copy link
Collaborator

sgugger commented Sep 1, 2022

Thanks for your PR! It would be great to have the ViTMSNForImageClassification even if there are no released weights for image classification, so users can already fine-tune the main checkpoint if they want.

For pretraining, if multiple new pieces are needed, maybe it could go in a research project at first, where you can add more modules?

@sayakpaul
Copy link
Member Author

For pretraining, if multiple new pieces are needed, maybe it could go in a research project at first, where you can add more modules?

Sounds good to me.

Thanks for your PR! It would be great to have the ViTMSNForImageClassification even if there are no released weights for image classification, so users can already fine-tune the main checkpoint if they want.

Sure, I will continue the work from here on then. Thank you!

sayakpaul
sayakpaul commented Sep 13, 2022
View reviewed changes
tests/models/vit_msn/test_modeling_vit_msn.py

@slow
def test_inference_image_classification_head(self):
torch.manual_seed(2)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is here so as to ensure the classification head params are always initialized using the same init. Since we don't have classification head params from the authors yet, this is needed to ensure the tests are passing.

@sayakpaul sayakpaul marked this pull request as ready for review September 13, 2022 04:01
@sayakpaul sayakpaul changed the title [WIP] MSN (Masked Siamese Networks) for ViT MSN (Masked Siamese Networks) for ViT Sep 13, 2022
@sayakpaul
Copy link
Member Author

@sgugger @NielsRogge @amyeroberts ready for review.

@sayakpaul
Copy link
Member Author

@sgugger @NielsRogge @amyeroberts a friendly nudge on the PR.

sgugger
sgugger approved these changes Sep 20, 2022
View reviewed changes
Copy link
Collaborator

@sgugger sgugger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple more comments, then the weights should be transferred to the right org before we merge this PR.

src/transformers/models/vit_msn/configuration_vit_msn.py Outdated Show resolved Hide resolved
src/transformers/models/vit_msn/modeling_vit_msn.py Outdated Show resolved Hide resolved
src/transformers/models/vit_msn/modeling_vit_msn.py Outdated Show resolved Hide resolved
sayakpaul and others added 2 commits September 20, 2022 22:29
@sayakpaul @sgugger
Update src/transformers/models/vit_msn/configuration_vit_msn.py
4c2d732 
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
@sayakpaul
address PR comments.
0da81a4
@sayakpaul
Copy link
Member Author

@sgugger addressed your comments. After the weights are transferred to the right org, I will open a PR there adding README.

amyeroberts
amyeroberts approved these changes Sep 22, 2022
View reviewed changes
Copy link
Collaborator

@amyeroberts amyeroberts left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice PR ❤️ All LGTM.

Thanks for your work adding this model 💪

src/transformers/models/vit_msn/modeling_vit_msn.py Outdated Show resolved Hide resolved
src/transformers/models/vit_msn/modeling_vit_msn.py
)


# Caution: We don't have the weights for the classification head yet. This class
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice comment :)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I am contacting the model author Mido to check if they can release the classification params before we merge. Will keep y'all posted.

sayakpaul and others added 2 commits September 22, 2022 15:08
@sayakpaul @amyeroberts
Update src/transformers/models/vit_msn/modeling_vit_msn.py
b7b17cb 
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
@sayakpaul
chore: put model in no_grad() and formatting.
89c24cc
@sgugger sgugger merged commit 2d9853b into huggingface:main Sep 22, 2022
@sayakpaul sayakpaul deleted the feat/msn branch September 22, 2022 11:20
@sayakpaul sayakpaul mentioned this pull request Sep 22, 2022
Merged
@ydshieh
Copy link
Collaborator

ydshieh commented Sep 23, 2022

Hi @sayakpaul . First, thank you for this PR 🤗 .

The doctest for this model is currently failing, as

transformers/src/transformers/models/vit_msn/modeling_vit_msn.py

Line 646 in 7e84723

>>> print(model.config.id2label[predicted_label])

this outputs the predicted label, but there is no expected value provided.

The config has LABEL_0 ... LABEL_999 in id2label, but I feel it should be the actual labels for the COCO dataset.

Could you take a look for this config, as well as the missing expected outputs for the doctest? Thank you!

Here is the failing doctest job:

https://github.com/huggingface/transformers/actions/runs/3109562462/jobs/5039877349

@sayakpaul
Copy link
Member Author

The config has LABEL_0 ... LABEL_999 in id2label, but I feel it should be the actual labels for the COCO dataset.

The model was trained on ImageNet-1k.

I will add the expected outputs. Thanks for flagging it.

@sayakpaul sayakpaul mentioned this pull request Sep 25, 2022
Merged
oneraghavan pushed a commit to oneraghavan/transformers that referenced this pull request Sep 26, 2022
@sayakpaul @NielsRogge
@sgugger @amyeroberts @oneraghavan
MSN (Masked Siamese Networks) for ViT (huggingface#18815)
c43afb4 
* feat: modeling and conversion scripts for msn.

* chore: change license year.

* chore: remove unneeded modules.

* feat: direct loading of state_dict from remote url.

* fix: import paths.

* add: rest of the files.

* add and fix rest of the files.

Co-authored-by: Niels <niels.rogge1@gmail.com>

* chore: formatting.

* code quality fix.

* chore: remove pooler.

* feat: add classification top.

* fix: configuration object.

* add: initial test cases (one failing).

* fix: basemodeloutput.

* add: caution on using the classification head.

* add: rest of the model related files.

* add: vit msn readme.

* fix: copied from statement.

* fix: dummy objects.

* add: ViTMSNPreTrainedModel to inits.

* fix: repo consistency.

* minor change in the model doc.

* fix: tests.

* Empty-Commit

* Update src/transformers/models/vit_msn/configuration_vit_msn.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* address PR comments.

* Update src/transformers/models/vit_msn/modeling_vit_msn.py

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

* chore: put model in no_grad() and formatting.

Co-authored-by: Niels <niels.rogge1@gmail.com>
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@amyeroberts amyeroberts amyeroberts approved these changes

@sgugger sgugger sgugger approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add MSN checkpoints to ViT
5 participants
@sayakpaul @HuggingFaceDocBuilderDev @sgugger @ydshieh @amyeroberts