Add BEiT #12994
Add BEiT #12994
Conversation
src/transformers/models/detr/convert_detr_original_pytorch_checkpoint_to_pytorch.py
Outdated
Show resolved
Hide resolved
tests/test_modeling_beit.py
Outdated
|
|
||
|
|
||
| @require_torch | ||
| class BEiTModelTest(ModelTesterMixin, unittest.TestCase): |
There was a problem hiding this comment.
Quick question. The tests are different for a bunch of vision models now, maybe we should have a special tester class for them and refactor the common tests of vision models there? I'm not familiar enough with how similar those tests are to be sure it's worth it, so tell me if it makes no sense.
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
|
I've uploaded all checkpoints to the hub: https://huggingface.co/models?search=microsoft/beit I've renamed the checkpoints which are fine-tuned on ImageNet-1k (after being intermediately fine-tuned on ImageNet-22k) to be just @donglixp if you're interested, could you write model cards for these models? Model cards are READMEs that describe the models in detail. You can take inspiration from ViT's model card. Also, I do have a notebook for |
|
@NielsRogge great work, any news on the future PR, to add the semantic segmentation model and the pretrained Ade20k? Thanks! |
|
@JStumpp say no more, it's added ;) |
What does this PR do?
It adds BEiT: BERT Pre-Training of Image Transformers to the library. It's the first paper that enables self-supervised pre-trained Vision Transformers (ViTs) to outperform their supervised pre-training counterparts. As a picture says more than a thousand (or 16x16?) words, this is a good summary of the approach:
The authors used OpenAI's DALL-E's encoder to map images to tokens, which the model then needs to predict based on masked patches. There are 3 models defined:
BEiTModel,BEiTForMaskedImageModelingandBEiTForImageClassification.This PR also cleans up some scripts from the library, namely those that defined id2label dicts for several datasets. I have removed
imagenet_classes.pyandcoco_classes.pyfrom the utils directory. Instead, id2label's are now defined on the hub in their own repository. These can then be used in conversion scripts using thehuggingface_hublibrary.To do
microsoft/beit_base_patch16_224_pt22k_ft22k_to_1kis getting out of handBEiTForMaskedImageModelingmodel. For this, tagging one of the original authors: @donglixpIn a future PR, I also plan to add the semantic segmentation model, which obtains SOTA on Ade20k.