Add Data2Vec for Vision in TF

[sayakpaul]

This PR adds the data2vec [1] model for vision in TensorFlow.

Todo:

* Fix cross-loading.
* Add integration test.
* Add remaining tests.
* Rest of the files remaining for the PR.
* TF weight uploading to Hub (to be done by someone from the 🤗 team)

Notes

• This PR does not add ...ForSegmentation. This can be done in a separate PR I think.
• Locally, I ran the tests using: RUN_SLOW=1 python -m pytest tests/data2vec/test_modeling_tf_data2vec_vision.py.

References

[1] data2vec: https://arxiv.org/abs/2202.03555

@sgugger @Rocketknight1 @ydshieh

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[sayakpaul]

I used these steps for styling: #16255 (comment).

On my end, when I am running make style I get the following:

...

doc-builder style src/transformers docs/source --max_len 119 --path_to_docs docs/source
Overwriting content of src/transformers/models/speech_to_text/modeling_tf_speech_to_text.py.
Overwriting content of src/transformers/models/luke/modeling_luke.py.
Overwriting content of src/transformers/models/xlm_prophetnet/modeling_xlm_prophetnet.py.
Overwriting content of src/transformers/models/tapas/modeling_tapas.py.
Overwriting content of src/transformers/models/tapas/modeling_tf_tapas.py.
Overwriting content of src/transformers/models/data2vec/modeling_tf_data2vec_vision.py.
Overwriting content of src/transformers/models/t5/modeling_flax_t5.py.
Overwriting content of src/transformers/models/t5/modeling_t5.py.
Overwriting content of src/transformers/models/t5/modeling_tf_t5.py.
Overwriting content of src/transformers/models/rag/modeling_rag.py.
Overwriting content of src/transformers/models/rag/retrieval_rag.py.
Overwriting content of src/transformers/models/wav2vec2/modeling_flax_wav2vec2.py.
Overwriting content of src/transformers/models/encoder_decoder/modeling_tf_encoder_decoder.py.
Overwriting content of src/transformers/models/encoder_decoder/modeling_encoder_decoder.py.
Overwriting content of src/transformers/models/xlm/modeling_xlm.py.
Overwriting content of src/transformers/models/vision_encoder_decoder/modeling_tf_vision_encoder_decoder.py.
Overwriting content of src/transformers/models/imagegpt/modeling_imagegpt.py.
Overwriting content of src/transformers/models/longformer/modeling_longformer.py.
Overwriting content of src/transformers/models/xlnet/modeling_xlnet.py.
Overwriting content of src/transformers/models/xlnet/modeling_tf_xlnet.py.
Overwriting content of src/transformers/models/gpt2/modeling_tf_gpt2.py.
Overwriting content of src/transformers/models/prophetnet/modeling_prophetnet.py.
Overwriting content of src/transformers/models/realm/modeling_realm.py.
Overwriting content of src/transformers/models/openai/modeling_tf_openai.py.
Overwriting content of src/transformers/models/openai/modeling_openai.py.
Overwriting content of docs/source/en/model_doc/luke.mdx.
Overwriting content of docs/source/en/model_doc/bert-generation.mdx.
Cleaned 27 files!

The CI console is also suggestive of this change. Should I add these cleaned files to the PR?

[sgugger]

Make sure you update hf-doc-builder to its latest version with pip install hf-doc-builder -U. We had a new release last week to fix some bugs in the example styling in our docs :-)

[sgugger]

Very nice, thanks a lot for adding this model in TensorFlow!

[sayakpaul]

Make sure you update hf-doc-builder to its latest version with pip install hf-doc-builder -U. We had a new release last week to fix some bugs in the example styling in our docs :-)

@sgugger I see that hf-doc-builder is already up to date on my end (Version: 0.3.0).

[sgugger]

You'll probably need to rebase your PR on master to get the changes in the setup for the quality check to pass (otherwise the CI uses the cached installed libraries).

[sayakpaul]

You'll probably need to rebase your PR on master to get the changes in the setup for the quality check to pass (otherwise the CI uses the cached installed libraries).

Thanks, @sgugger! I first rebased my main with the upstream and then merged the main into the PR branch. And then I force-pushed. Let's see.

[ydshieh]

Hi, @sayakpaul

• You can ignore Model templates runner / run_tests_templates (pull_request). (You can even cancel that workflow run)
• I have just merged a (big) PR that moved model test folders, like tests/bert to tests/models/bert. When you have time, could you
  • pull the changes (from upstream main) to your main
  • rebase your working branch on the main (better to avoid using merge in this case, I believe)
  • move your new test file tests/data2vec/test_modeling_tf_data2vec_vision.py from tests/data2vec to tests/models/data2vec
    • You might need to fix a few lines of import.
    • For example, from ..test_configuration_common import ConfigTester --> from ...test_configuration_common import ConfigTester

please? 🙏 Thank you!

[sayakpaul]

@ydshieh after rebasing, won't I need to merge the main into my PR branch so that it has the full effect?

[ydshieh]

@ydshieh after rebasing, won't I need to merge the main into my PR branch so that it has the full effect?

In order to incorporate the changes in main into your PR branch, you can either use merge or rebase. I am in favor of using rebase as it might be cleaner in some cases (won't introduce a lot of file changes).

Once you have latest changes from upstream main in your local main, you can checkout to your PR branch, and do something like

git rebase main

(sometimes there might be conflicts to fix, but I think there won't be conflict in this case)

Then you will have to force push.

[sayakpaul]

@ydshieh oops looks like I have made things worse instead of making them work. I am not sure how I can revert to a mergeable state now. Any suggestion?

[ydshieh]

@ydshieh oops looks like I have made things worse instead of making them work. I am not sure how I can revert to a mergeable state now. Any suggestion?

Let me give it a try - I am definitely NOT a Git Pro 😢
(No guarantee though - hope 🤞 )

Could you let me know what steps you have done, please?

[sayakpaul]

I just followed your suggestions:

• Rebased my main with the upstream main.
• Checked out to my PR branch and ran git rebase main.
• Made the necessary changes you suggested regarding moving the test file.

I think you mistakenly made a push to my PR branch which is what may have caused the unnecessary changes to reflect in this PR.

I am happy to work on the necessary steps per your suggestions too.

[sgugger]

Currently, the follow checkpoint crashes (after the two suggestions I have made on the PR):

from transformers import TFAutoModel

tf_model = TFAutoModel.from_pretrained("facebook/data2vec-vision-base", from_pt=True)

Same for "facebook/data2vec-vision-large", therefore I can't convert those checkpoints (and it looks like something needs fixing?)

Here is the traceback:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/tmp/ipykernel_2749758/3199004601.py in <module>
----> 1 tf_model = TFAutoModel.from_pretrained("facebook/data2vec-vision-large", from_pt=True)

~/git/transformers/src/transformers/models/auto/auto_factory.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    444         elif type(config) in cls._model_mapping.keys():
    445             model_class = _get_model_class(config, cls._model_mapping)
--> 446             return model_class.from_pretrained(pretrained_model_name_or_path, *model_args, config=config, **kwargs)
    447         raise ValueError(
    448             f"Unrecognized configuration class {config.__class__} for this kind of AutoModel: {cls.__name__}.\n"

~/git/transformers/src/transformers/modeling_tf_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
   1794 
   1795             # Load from a PyTorch checkpoint
-> 1796             return load_pytorch_checkpoint_in_tf2_model(model, resolved_archive_file, allow_missing_keys=True)
   1797 
   1798         # we might need to extend the variable scope for composite models

~/git/transformers/src/transformers/modeling_tf_pytorch_utils.py in load_pytorch_checkpoint_in_tf2_model(tf_model, pytorch_checkpoint_path, tf_inputs, allow_missing_keys)
    122     logger.info(f"PyTorch checkpoint contains {sum(t.numel() for t in pt_state_dict.values()):,} parameters")
    123 
--> 124     return load_pytorch_weights_in_tf2_model(
    125         tf_model, pt_state_dict, tf_inputs=tf_inputs, allow_missing_keys=allow_missing_keys
    126     )

~/git/transformers/src/transformers/modeling_tf_pytorch_utils.py in load_pytorch_weights_in_tf2_model(tf_model, pt_state_dict, tf_inputs, allow_missing_keys)
    153 
    154     if tf_inputs is not None:
--> 155         tf_model(tf_inputs, training=False)  # Make sure model is built
    156     # Adapt state dict - TODO remove this and update the AWS weights files instead
    157     # Convert old format to new format if needed from a PyTorch state_dict

~/anaconda3/lib/python3.9/site-packages/keras/utils/traceback_utils.py in error_handler(*args, **kwargs)
     65     except Exception as e:  # pylint: disable=broad-except
     66       filtered_tb = _process_traceback_frames(e.__traceback__)
---> 67       raise e.with_traceback(filtered_tb) from None
     68     finally:
     69       del filtered_tb

~/git/transformers/src/transformers/modeling_tf_utils.py in run_call_with_unpacked_inputs(self, *args, **kwargs)
    381         main_input = fn_args_and_kwargs.pop(main_input_name, None)
    382         unpacked_inputs = input_processing(func, self.config, main_input, **fn_args_and_kwargs)
--> 383         return func(self, **unpacked_inputs)
    384 
    385     # Keras enforces the first layer argument to be passed, and checks it through `inspect.getfullargspec()`. This

~/git/transformers/src/transformers/models/data2vec/modeling_tf_data2vec_vision.py in call(self, pixel_values, bool_masked_pos, head_mask, output_attentions, output_hidden_states, return_dict, training)
    893     ) -> Union[tuple, TFData2VecVisionModelOutputWithPooling]:
    894 
--> 895         outputs = self.data2vec_vision(
    896             pixel_values=pixel_values,
    897             bool_masked_pos=bool_masked_pos,

~/git/transformers/src/transformers/modeling_tf_utils.py in run_call_with_unpacked_inputs(self, *args, **kwargs)
    381         main_input = fn_args_and_kwargs.pop(main_input_name, None)
    382         unpacked_inputs = input_processing(func, self.config, main_input, **fn_args_and_kwargs)
--> 383         return func(self, **unpacked_inputs)
    384 
    385     # Keras enforces the first layer argument to be passed, and checks it through `inspect.getfullargspec()`. This

~/git/transformers/src/transformers/models/data2vec/modeling_tf_data2vec_vision.py in call(self, pixel_values, bool_masked_pos, head_mask, output_attentions, output_hidden_states, return_dict, training)
    712         embedding_output = self.embeddings(pixel_values, bool_masked_pos, training=training)
    713 
--> 714         encoder_outputs = self.encoder(
    715             embedding_output,
    716             head_mask=head_mask,

~/git/transformers/src/transformers/models/data2vec/modeling_tf_data2vec_vision.py in call(self, hidden_states, head_mask, output_attentions, output_hidden_states, return_dict)
    625             layer_head_mask = head_mask[i] if head_mask is not None else None
    626 
--> 627             relative_position_bias = self.relative_position_bias() if self.relative_position_bias is not None else None
    628             layer_outputs = layer_module(hidden_states, layer_head_mask, output_attentions, relative_position_bias)
    629 

ValueError: Exception encountered when calling layer "encoder" (type TFData2VecVisionEncoder).

The first argument to `Layer.call` must always be passed.

Call arguments received:
  • hidden_states=tf.Tensor(shape=(3, 197, 1024), dtype=float32)
  • head_mask=['None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None']
  • output_attentions=False
  • output_hidden_states=False
  • return_dict=True

I have converted facebook/data2vec-vision-base-ft1k and am doing facebook/data2vec-vision-large-ft1k now.

[sayakpaul]

@sgugger thanks for providing the update. Let me check from my end once.

[sayakpaul]

@sgugger should be all good now.

I have verified from my end too:

[sgugger]

Can confirm it works. TF weights added for all 4 facebook Data2Vec Image models.

[sayakpaul]

@sgugger thanks! I have also run the tests on my end locally, they're passing.

Over to you.

[sgugger]

Repinging @Rocketknight1 to have another set of eyes on this :-)

[ydshieh]

Thanks @sayakpaul for this work!

I will review the part regarding the test more thoroughly today, but so far having 2 questions.

[Rocketknight1]

Overall the model code looks very solid! I had some comments but they've already been addressed, so I'm happy to approve this once tests are green.

[sayakpaul]

Thanks! Looks like they are all green now.

[sgugger]

Thanks again for your contribution!