TensorFlow MobileViT #18555

sayakpaul · 2022-08-10T12:58:55Z

This PR implements the MobileViT model in TensorFlow.

Interesting points

The classification and segmentation models provided with MobileViT are fully compatible with TensorFlow Lite. Therefore, I have included sample code in the model documentation showing how to perform the TensorFlow Lite conversion (~4 lines of code).
TFLlite versions of the smallest checkpoints for classification and semantic segmentation are 1MB and 2MBs, respectively. I believe this will be quite beneficial to the TinyML community.

TODOs

Hosting of the TF checkpoints on the Hub. (Can I do it now? If so, I need resources that show how to do that.)
Remove from_pt wherever needed.

@amyeroberts @gante @sgugger up for review!

HuggingFaceDocBuilderDev · 2022-08-10T13:09:50Z

The documentation is not available anymore as the PR was closed or merged.

Co-authored-by: Amy <aeroberts4444@gmail.com>

Coo-authored-by: Yih <2521628+ydshieh@users.noreply.github.com>

gante

Thank you for the TF implementation of MobileViT 🙌 The TFLite demo is great, especially because it should be covered by our doctests 🚀

In addition to the comments throughout the code, I have the following notes:

I will share the instructions to open the Hub PR when this PR is approved by all (everyone has permission to do it now 🎉 )
The training argument is missing in the layer's call (and in places like the dropout calls)

src/transformers/models/mobilevit/__init__.py

src/transformers/models/mobilevit/modeling_tf_mobilevit.py

tests/models/mobilevit/test_modeling_tf_mobilevit.py

sayakpaul · 2022-08-25T04:46:42Z

Didn't realize that re-requesting a review from @gante would result in removing @amyeroberts and @sgugger from the reviewer list. Please know that it was completely unintentional.

gante · 2022-08-25T10:23:43Z

@sayakpaul no worries :)

gante

Thank you for the changes 🔥

amyeroberts

LGTM! 📱

src/transformers/models/mobilevit/__init__.py

src/transformers/models/mobilevit/modeling_tf_mobilevit.py

amyeroberts · 2022-08-25T12:23:27Z

src/transformers/models/mobilevit/modeling_tf_mobilevit.py

+        patch_width, patch_height = self.patch_width, self.patch_height
+        patch_area = tf.cast(patch_width * patch_height, "int32")
+
+        batch_size, orig_height, orig_width, channels = shape_list(features)


Suggested change

batch_size, orig_height, orig_width, channels = shape_list(features)

batch_size, orig_height, orig_width, channels = tf.shape(features)

Having it in one line leads to:

OperatorNotAllowedInGraphError: Iterating over a symbolic tf.Tensor is not allowed in Graph execution. Use Eager execution or decorate this function with @tf.function.

That's why I separated it.

amyeroberts · 2022-08-25T14:32:31Z

tests/models/mobilevit/test_modeling_tf_mobilevit.py

+    def test_attention_outputs(self):
+        pass
+
+    @unittest.skip("Test was written for TF 1.x and isn't really relevant here")


If this is the case - should it even be in the test suite? cc @gante

Nope, me and @Rocketknight1 talked about it a few weeks ago. We should remove this test, it's heavy and the only new thing it tests is that we can build a functional TF model with the model class (which it's kinda obvious we can)

I assume it will be phased out in a separate PR from the main TF testing suite?

Ofc, as a separate PR 👍 Leave it be as it is in this PR :)

src/transformers/models/mobilevit/modeling_tf_mobilevit.py

amyeroberts · 2022-08-25T15:00:18Z

Thanks for another great model addition @sayakpaul !

gante · 2022-08-25T15:24:50Z

@sayakpaul assuming it is passing the slow tests, it is ready for the TF weights.

The super complex instructions to do it are as follows:

Make sure you have the latest version of the hub installed (pip install huggingface_hub -U) and that you are logged in to HF with a write token (huggingface-cli login)
Run transformers-cli pt-to-tf --model-name foo/bar from this branch :D
In the Hub PR, tag @joaogante, @lysandre

sayakpaul · 2022-08-25T15:26:52Z

Super simple (complex?) question:

What is the format of foo/bar?

gante · 2022-08-25T15:28:15Z

The same as the model name on the hub, e.g. this model would be apple/mobilevit-small

P.S.: I edited the comment above with a 3rd step :D

hollance · 2022-08-29T09:59:25Z

[...] the output consistency of nn.functional.interpolate and tf.image.resize with the same argument values.

This might be due to the align_corners option. I once wrote a long blog post about this difference between PyTorch and TF. https://machinethink.net/blog/coreml-upsampling/ Not sure if that's the same issue but it seems likely.

sayakpaul · 2022-08-29T11:05:06Z

[...] the output consistency of nn.functional.interpolate and tf.image.resize with the same argument values.

This might be due to the align_corners option. I once wrote a long blog post about this difference between PyTorch and TF. https://machinethink.net/blog/coreml-upsampling/ Not sure if that's the same issue but it seems likely.

Very well! If we need to deal with the inconsistencies between tf.image.resize and nn.functional.interpolate I suggest we do that in a separate PR 'cause various vision models would benefit from that (ViT for example).

sayakpaul · 2022-08-30T02:09:44Z

@gante WDYT?

gante · 2022-08-30T08:39:38Z

@sayakpaul regarding the PR, all good on my end, but we still need approval from @sgugger :D

As for the tf.image.resize -- yeah, it would be nice to standardize for all models. Would you be interested in working on it? In any case, I'd like to ask you to open an issue, so we don't forget to track it!

sayakpaul · 2022-08-30T08:40:26Z

As for the tf.image.resize -- yeah, it would be nice to standardize for all models. Would you be interested in working on it? In any case, I'd like to ask you to open an issue, so we don't forget to track it!

On it, sir!

sayakpaul · 2022-08-31T07:05:17Z

@amyeroberts @gante

Please take note of the changes in 32cfd30.

Initially, when I tested TFLite conversion it didn't require any spec for SELECT operations but now they're failing with a specification for the SELECT ops. What is more surprising is that the TFLite interpreter is treating tf.Conv2D to be a SELECT op. Hence I have raised tensorflow/tensorflow#57550.

sgugger

Thanks a lot for your PR! Left a couple of nits then we can merge this.

src/transformers/models/mobilevit/modeling_tf_mobilevit.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

gante · 2022-09-01T12:14:32Z

(retriggered failing job, seems like a spurious failure)

sayakpaul · 2022-09-01T12:27:16Z

Yeah probably nothing related to the PR?

sgugger · 2022-09-01T12:27:41Z

The build doc job failure is not spurious. There seems to be a problem with an example bloc introduced by this PR.

sayakpaul · 2022-09-01T12:31:56Z

Let me see if removing comments from the example block does the trick. Because when the job wasn't failing the example block didn't have any comments.

sayakpaul · 2022-09-01T12:42:00Z

No, it didn't help :( Any suggestions to try out?

gante · 2022-09-01T12:47:03Z

docs/source/en/model_doc/mobilevit.mdx

+  You can use the following code to convert a MobileViT checkpoint (be it image classification or semantic segmentation) to generate a 
+  TensorFlow Lite model:
+
+  ```py


could it be because the example is indented?

Looks like it, for some reason. The failure seems to disappear locally when I remove it. In any case its place is probably closer to the TF models doc?

This was the culprit it seems :3

gante · 2022-09-01T12:48:25Z

The build doc job failure is not spurious. There seems to be a problem with an example bloc introduced by this PR.

My bad :D read the failure bottom to top, so I didn't notice the mobilevit errors

* initial implementation. * add: working model till image classification. * add: initial implementation that passes intg tests. Co-authored-by: Amy <aeroberts4444@gmail.com> * chore: formatting. * add: tests (still breaking because of config mismatch). Coo-authored-by: Yih <2521628+ydshieh@users.noreply.github.com> * add: corrected tests and remaning changes. * fix code style and repo consistency. * address PR comments. * address Amy's comments. * chore: remove from_pt argument. * chore: add full-stop. * fix: TFLite model conversion in the doc. * Update src/transformers/models/mobilevit/modeling_tf_mobilevit.py Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> * Update src/transformers/models/mobilevit/modeling_tf_mobilevit.py Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> * Update src/transformers/models/mobilevit/modeling_tf_mobilevit.py Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> * Update src/transformers/models/mobilevit/modeling_tf_mobilevit.py Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> * Update src/transformers/models/mobilevit/modeling_tf_mobilevit.py Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> * apply formatting. * chore: remove comments from the example block. * remove identation in the example. Co-authored-by: Amy <aeroberts4444@gmail.com> Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

initial implementation.

712429b

sayakpaul and others added 5 commits August 11, 2022 18:14

add: working model till image classification.

2251838

add: initial implementation that passes intg tests.

db3ac6d

Co-authored-by: Amy <aeroberts4444@gmail.com>

chore: formatting.

f08fdaa

add: tests (still breaking because of config mismatch).

8569374

Coo-authored-by: Yih <2521628+ydshieh@users.noreply.github.com>

add: corrected tests and remaning changes.

6fcc70f

sayakpaul changed the title ~~[WIP] TensorFlow MobileViT~~ TensorFlow MobileViT Aug 22, 2022

sayakpaul marked this pull request as ready for review August 22, 2022 07:10

sayakpaul added 2 commits August 22, 2022 12:44

fix code style and repo consistency.

cd72a53

Merge branch 'main' into feat/tf-mobilevit

7c51b81

gante requested review from sgugger, amyeroberts and gante August 24, 2022 14:56

gante reviewed Aug 24, 2022

View reviewed changes

sayakpaul added 2 commits August 25, 2022 10:08

address PR comments.

cc634b7

Merge branch 'main' into feat/tf-mobilevit

6e419e4

sayakpaul requested review from gante and removed request for amyeroberts and sgugger August 25, 2022 04:45

gante requested review from sgugger and amyeroberts August 25, 2022 10:23

gante approved these changes Aug 25, 2022

View reviewed changes

amyeroberts approved these changes Aug 25, 2022

View reviewed changes

Merge branch 'main' into feat/tf-mobilevit

35d4303

sayakpaul mentioned this pull request Aug 30, 2022

Inconsistencies between nn.functional.interpolate and tf.image.resize #18811

Closed

4 tasks

gante mentioned this pull request Aug 30, 2022

[TensorFlow] Adding GroupViT #18020

Merged

5 tasks

fix: TFLite model conversion in the doc.

32cfd30

Merge branch 'main' into feat/tf-mobilevit

e81539a

sgugger approved these changes Sep 1, 2022

View reviewed changes

sayakpaul and others added 6 commits September 1, 2022 16:59

Update src/transformers/models/mobilevit/modeling_tf_mobilevit.py

b5593b9

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

Update src/transformers/models/mobilevit/modeling_tf_mobilevit.py

4365320

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

Update src/transformers/models/mobilevit/modeling_tf_mobilevit.py

7c93be0

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

Update src/transformers/models/mobilevit/modeling_tf_mobilevit.py

06cb368

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

Update src/transformers/models/mobilevit/modeling_tf_mobilevit.py

560d7ca

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

apply formatting.

127a0f1

chore: remove comments from the example block.

43ce94d

gante reviewed Sep 1, 2022

View reviewed changes

remove identation in the example.

9b00370

sgugger merged commit 954e18a into huggingface:main Sep 1, 2022

sayakpaul deleted the feat/tf-mobilevit branch September 1, 2022 15:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TensorFlow MobileViT #18555

TensorFlow MobileViT #18555

sayakpaul commented Aug 10, 2022 •

edited

HuggingFaceDocBuilderDev commented Aug 10, 2022 •

edited

gante left a comment

sayakpaul commented Aug 25, 2022

gante commented Aug 25, 2022

gante left a comment

amyeroberts left a comment

amyeroberts Aug 25, 2022

sayakpaul Aug 26, 2022

amyeroberts Aug 25, 2022

gante Aug 25, 2022

sayakpaul Aug 26, 2022 •

edited

gante Aug 26, 2022

amyeroberts commented Aug 25, 2022

gante commented Aug 25, 2022 •

edited

sayakpaul commented Aug 25, 2022

gante commented Aug 25, 2022

hollance commented Aug 29, 2022

sayakpaul commented Aug 29, 2022

sayakpaul commented Aug 30, 2022

gante commented Aug 30, 2022

sayakpaul commented Aug 30, 2022

sayakpaul commented Aug 31, 2022

sgugger left a comment

gante commented Sep 1, 2022

sayakpaul commented Sep 1, 2022

sgugger commented Sep 1, 2022

sayakpaul commented Sep 1, 2022

sayakpaul commented Sep 1, 2022

gante Sep 1, 2022

sgugger Sep 1, 2022

sayakpaul Sep 1, 2022

gante commented Sep 1, 2022

	batch_size, orig_height, orig_width, channels = shape_list(features)
	batch_size, orig_height, orig_width, channels = tf.shape(features)

TensorFlow MobileViT #18555

TensorFlow MobileViT #18555

Conversation

sayakpaul commented Aug 10, 2022 • edited

Interesting points

TODOs

HuggingFaceDocBuilderDev commented Aug 10, 2022 • edited

gante left a comment

Choose a reason for hiding this comment

sayakpaul commented Aug 25, 2022

gante commented Aug 25, 2022

gante left a comment

Choose a reason for hiding this comment

amyeroberts left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sayakpaul Aug 26, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amyeroberts commented Aug 25, 2022

gante commented Aug 25, 2022 • edited

sayakpaul commented Aug 25, 2022

gante commented Aug 25, 2022

hollance commented Aug 29, 2022

sayakpaul commented Aug 29, 2022

sayakpaul commented Aug 30, 2022

gante commented Aug 30, 2022

sayakpaul commented Aug 30, 2022

sayakpaul commented Aug 31, 2022

sgugger left a comment

Choose a reason for hiding this comment

gante commented Sep 1, 2022

sayakpaul commented Sep 1, 2022

sgugger commented Sep 1, 2022

sayakpaul commented Sep 1, 2022

sayakpaul commented Sep 1, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gante commented Sep 1, 2022

sayakpaul commented Aug 10, 2022 •

edited

HuggingFaceDocBuilderDev commented Aug 10, 2022 •

edited

sayakpaul Aug 26, 2022 •

edited

gante commented Aug 25, 2022 •

edited