Add Chinese-CLIP implementation

[yangapku]

What does this PR do?

This PR adds Chinese-CLIP model into Transformers repo. The Chinese-CLIP model was introduced in Chinese CLIP: Contrastive Vision-Language Pretraining in Chinese. Chinese CLIP is an implementation and adaptation of CLIP (Radford et al., 2021) on a large-scale dataset of Chinese image-text pairs. It is capable of performing Chinese-based cross-modal retrieval and also playing as a vision backbone for vision tasks like zero-shot image classification, open-domain object detection, etc. This model was contributed by OFA-Sys. The original Github repo of Chinese-CLIP can be found at this link. Currently we have released our model weights on the Huggingface ModelHub. Compared with original OpenAI CLIP, we changed the text encoder to Chinese Roberta encoder, thus we reimplemented the config, modeling and preprocessor modules of Chinese-CLIP. Necessary unit tests and documents have been added.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[yangapku]

@ydshieh All the comments mentioned above have been addressed.

[sgugger]

A couple of final comments on my side and it should be good to merge!

[yangapku]

@ydshieh Hi, is the PR able to be merged now? Thank you very much! ❤️

[ydshieh]

@sgugger I feel there must be very good reason we want to have CLIPTextTransformer and CLIPVisionTransformer, and use these components in CLIPTextModel, CLIPVisionModel and CLIPModel.

(Potentially to avoid CLIPPreTrainedModel in another CLIPPreTrainedModel which might cause some issues - at least if we ever want to have a TF port).

Do you think here we need to avoid this line

self.text_model = ChineseCLIPTextModel(text_config, add_pooling_layer=False)

and to create ChineseCLIPTextTransformer and use it?

[ydshieh]

Hi @yangapku , other than the above comment, LGTM! But let's wait @sgugger 's response.

There are a few places where I believe we can still use # copied from (probably need some tweaks) - I can help on this before merge.

[sgugger]

@ydshieh I think this was done this way just to be more aligned with the original checkpoints in the CLIP case. Here it works fine with the checkpoint, so I wouldn't over-complexify things

[ydshieh]

@yangapku Before we merge, could you run

RUN_SLOW=1 python -m pytest -v tests/models/chinese_clip/

I got 5 failures. You can focus on the first/last ones at this moment. Let me know if you need help on fixing them 🙏

FAILED tests/models/chinese_clip/test_modeling_chinese_clip.py::ChineseCLIPTextModelTest::test_model_from_pretrained - AttributeError: 'ChineseCLIPConfig' object has no attribute 'vocab_size'
FAILED tests/models/chinese_clip/test_modeling_chinese_clip.py::ChineseCLIPModelTest::test_torchscript_output_attentions - AssertionError: Items in the second set but not the first:
FAILED tests/models/chinese_clip/test_modeling_chinese_clip.py::ChineseCLIPModelTest::test_torchscript_output_hidden_state - AssertionError: Items in the second set but not the first:
FAILED tests/models/chinese_clip/test_modeling_chinese_clip.py::ChineseCLIPModelTest::test_torchscript_simple - AssertionError: Items in the second set but not the first:
FAILED tests/models/chinese_clip/test_modeling_chinese_clip.py::ChineseCLIPModelIntegrationTest::test_inference - OSError: Can't load tokenizer for 'OFA-Sys/chinese-clip-vit-base-patch16'. If you were trying to load it

[yangapku]

@yangapku Before we merge, could you run

RUN_SLOW=1 python -m pytest -v tests/models/chinese_clip/

I got 5 failures. You can focus on the first/last ones at this moment. Let me know if you need help on fixing them 🙏

FAILED tests/models/chinese_clip/test_modeling_chinese_clip.py::ChineseCLIPTextModelTest::test_model_from_pretrained - AttributeError: 'ChineseCLIPConfig' object has no attribute 'vocab_size'
FAILED tests/models/chinese_clip/test_modeling_chinese_clip.py::ChineseCLIPModelTest::test_torchscript_output_attentions - AssertionError: Items in the second set but not the first:
FAILED tests/models/chinese_clip/test_modeling_chinese_clip.py::ChineseCLIPModelTest::test_torchscript_output_hidden_state - AssertionError: Items in the second set but not the first:
FAILED tests/models/chinese_clip/test_modeling_chinese_clip.py::ChineseCLIPModelTest::test_torchscript_simple - AssertionError: Items in the second set but not the first:
FAILED tests/models/chinese_clip/test_modeling_chinese_clip.py::ChineseCLIPModelIntegrationTest::test_inference - OSError: Can't load tokenizer for 'OFA-Sys/chinese-clip-vit-base-patch16'. If you were trying to load it

Okay I will try to fix them today.

[yangapku]

@yangapku Before we merge, could you run

RUN_SLOW=1 python -m pytest -v tests/models/chinese_clip/

I got 5 failures. You can focus on the first/last ones at this moment. Let me know if you need help on fixing them 🙏

FAILED tests/models/chinese_clip/test_modeling_chinese_clip.py::ChineseCLIPTextModelTest::test_model_from_pretrained - AttributeError: 'ChineseCLIPConfig' object has no attribute 'vocab_size'
FAILED tests/models/chinese_clip/test_modeling_chinese_clip.py::ChineseCLIPModelTest::test_torchscript_output_attentions - AssertionError: Items in the second set but not the first:
FAILED tests/models/chinese_clip/test_modeling_chinese_clip.py::ChineseCLIPModelTest::test_torchscript_output_hidden_state - AssertionError: Items in the second set but not the first:
FAILED tests/models/chinese_clip/test_modeling_chinese_clip.py::ChineseCLIPModelTest::test_torchscript_simple - AssertionError: Items in the second set but not the first:
FAILED tests/models/chinese_clip/test_modeling_chinese_clip.py::ChineseCLIPModelIntegrationTest::test_inference - OSError: Can't load tokenizer for 'OFA-Sys/chinese-clip-vit-base-patch16'. If you were trying to load it

Okay I will try to fix them today.

@ydshieh The first and last failed cases have been fixed. Now only the failed test cases with Torchscript still remain. Meanwhile, to fix the first failed case, I have to remove the copied from comment for ChineseCLIPTextModel, since it has diverged from BertModel with our customed config_class ChineseCLIPTextConfig.

[yangapku]

@ydshieh Hi, is the PR able to be merged now? Do I have to fix the test cases related with Torchscript? If so, more help is needed since I am not so familiar with it 😢 .

[ydshieh]

I will take a look on those 3 tests @yangapku .

[ydshieh]

@yangapku I pushed the remaining fix. Will merge once the final CI is green 🚀 🚀 🚀

非常感謝您的工作! 💯

[ydshieh]

Thanks again!

[yangapku]

Thank you very much for your brilliant support! @ydshieh @sgugger

[ydshieh]

Hi @yangapku Just a follow up. From your branch , I see the file

convert_chinese_clip_original_pytorch_to_hf.py

is last modified on Nov 22. (The change on Nov 29 doesn't count). However, the modeling file changed quite a lot since then due to our review comments. I just want to make sure the conversion script still works correctly, and the original checkpoints and the the converted HF checkpoints still have the same outputs on some test examples.

It would be super nice if you can double check, but it's your call, it's just a suggestion.
(It's always good to make sure the users get the right checkpoints to use :-))

[yangapku]

@ydshieh Hi, I have ensured that this conversion script works correctly 😄 . In fact, today we have also updated the other 3 model scales (ViT-L/14, ViT-L/14@336px, ViT-H/14) on our HF model hub, during which I have used this script to convert our original model to HF format. After the conversion, I have tested all the converted HF checkpoints (the pytorch_model.bin), all of them works in expectation.

[ydshieh]

Thank you @yangapku !