Add ConvNeXt models #16421

sayakpaul · 2022-04-16T10:29:05Z

Conversion scripts and ImageNet-1k evaluation are available here: https://github.com/sayakpaul/keras-convnext-conversion.

Comparison to the actual reported numbers

name	original acc@1	keras acc@1
convnext_tiny_1k_224	82.1	81.312
convnext_small_1k_224	83.1	82.392
convnext_base_21k_1k_224	85.8	85.364
convnext_large_21k_1k_224	86.6	86.36
convnext_xlarge_21k_1k_224	87.0	86.732

@LukeWood

keras/applications/convnext.py

fchollet

Thanks for the PR! This is a great addition to applications.

Comments may not be exhaustive.

keras/applications/convnext.py

sayakpaul · 2022-04-17T12:50:05Z

Once the implementation looks good, I will add the other components. Please find the model conversion and the evaluation details in this comment: #16421 (comment).

By "conversion" I mean the following:

I first implemented the models in Keras.
I then populated them with the pre-trained parameters.

Implementation correctness needs to be ensured in these cases, hence the evaluation.

keras/applications/convnext.py

sayakpaul · 2022-04-18T01:56:16Z

keras/applications/convnext.py

+  "xlarge":
+    ("da65d1294d386c71aebd81bc2520b8d42f7f60eee4414806c60730cd63eb15cb",
+      "2bfbf5f0c2b3f004f1c32e9a76661e11a9ac49014ed2a68a49ecd0cd6c88d377"),
+}


I have also converted the ImageNet-21 checkpoints which would likely be better for transfer learning than the checkpoints from ImageNet-1k pre-training.

But to add those checkpoints we need the following:

ImageNet-21k models are supposed to be multi-label classifiers. So the activation should be "sigmoid". So when weights="imagenet21k" && include_top=True, classifier_activation is supposed to be sigmoid.

This would require changes to imagenet_utils.validate_activation. As I understand it, it only supports softmax at the moment.

IMO, a better idea would be to keep the PR as it is. Once it's done, we could work on another PR setting up validation for sigmoid when loading pre-trained models like the one mentioned above. After that, I'm happy to work on another PR to incorporate the ImageNet-21k checkpoints making the changes necessary.

Sounds good, it's fine to only have support for imagenet1k for now (this is consistent with the other applications). We can add more checkpoints in the future.

keras/applications/convnext.py

fchollet · 2022-04-18T21:19:00Z

keras/applications/convnext.py

+  "xlarge":
+    ("da65d1294d386c71aebd81bc2520b8d42f7f60eee4414806c60730cd63eb15cb",
+      "2bfbf5f0c2b3f004f1c32e9a76661e11a9ac49014ed2a68a49ecd0cd6c88d377"),
+}


Sounds good, it's fine to only have support for imagenet1k for now (this is consistent with the other applications). We can add more checkpoints in the future.

sayakpaul · 2022-04-19T02:13:58Z

@fchollet just incorporated the changes you asked for.

These changes will require a re-conversion of the pre-trained parameters since the model structure has been changed a bit now. I will do that (as well as the ImageNet-1k evaluation) after others have had a chance to review the PR.

LukeWood

Looks good to me - just need to figure out the right end action per Francois' comments and add tests when that is done. Thanks @sayakpaul for the contribution!

LukeWood

Performed a more thorough pass to get this moving along. A few minor changes, then a big question regarding normalization that I think @fchollet will have context on.

keras/applications/convnext.py

LukeWood · 2022-04-29T17:20:46Z

keras/applications/convnext.py

+
+  def apply(x):
+    x = layers.Normalization(
+      mean=[0.485 * 255, 0.456 * 255, 0.406 * 255],


So, these are initialized based on imagenet: this is required for use with the pretrained weights. Is there a way we can allow users to configure this for custom datasets?

@fchollet

It's inspired from ResNet-RS and RegNets. For non-ImageNet weights, it's necessary to disable it.

keras/applications/convnext.py

sayakpaul · 2022-05-06T05:12:13Z

@LukeWood with the current setup, we have a problem.

https://github.com/sayakpaul/keras/blob/feat/convnext/keras/applications/applications_test.py#L129 will fail being unable to get instantiated from the config.

https://github.com/sayakpaul/keras/blob/feat/convnext/keras/applications/applications_test.py#L134-#L137 does not help.

Error trace (used separately to test the component separately):

Traceback (most recent call last):
  File "convert.py", line 249, in <module>
    main(args)
  File "convert.py", line 111, in main
    reconstructed_model = convnext_model_tf.__class__.from_config(config)
  File "/Users/sayakpaul/.local/bin/.virtualenvs/pytorch/lib/python3.8/site-packages/keras/engine/functional.py", line 708, in from_config
    input_tensors, output_tensors, created_layers = reconstruct_from_config(
  File "/Users/sayakpaul/.local/bin/.virtualenvs/pytorch/lib/python3.8/site-packages/keras/engine/functional.py", line 1326, in reconstruct_from_config
    process_layer(layer_data)
  File "/Users/sayakpaul/.local/bin/.virtualenvs/pytorch/lib/python3.8/site-packages/keras/engine/functional.py", line 1308, in process_layer
    layer = deserialize_layer(layer_data, custom_objects=custom_objects)
  File "/Users/sayakpaul/.local/bin/.virtualenvs/pytorch/lib/python3.8/site-packages/keras/layers/serialization.py", line 207, in deserialize
    return generic_utils.deserialize_keras_object(
  File "/Users/sayakpaul/.local/bin/.virtualenvs/pytorch/lib/python3.8/site-packages/keras/utils/generic_utils.py", line 679, in deserialize_keras_object
    deserialized_obj = cls.from_config(
  File "/Users/sayakpaul/.local/bin/.virtualenvs/pytorch/lib/python3.8/site-packages/keras/engine/training.py", line 2641, in from_config
    functional.reconstruct_from_config(config, custom_objects))
  File "/Users/sayakpaul/.local/bin/.virtualenvs/pytorch/lib/python3.8/site-packages/keras/engine/functional.py", line 1325, in reconstruct_from_config
    for layer_data in config['layers']:
KeyError: 'layers'

I am currently trying to wrap LayerScale as a separate layer so that ConvNeXtBlock class could be turned into a nested function such as done in RegNets (example).

If this works out well, then the problem is solved otherwise we'll have to brainstorm more.

feat: convnext with functional api.

sayakpaul · 2022-05-06T07:51:17Z

@LukeWood I think I was able to make things work. I ran bazel test keras/applications/applications_test and it's passing successfully (Colab Notebook).

Here's what changed:

LayerScale has now become a layer so that we can stay with the Functional API. This simplifies the model design and stays in line with the other Keras applications.

One thing I couldn't understand is that without this with context, the test fails. It should have also complained about the StochasticDepth layer since it's also custom. Both these custom layers have get_config() overridden. So, I'm not sure if I defined the get_config() for LayerScale in the wrong way.

Let me know your thoughts on the recent changes.

P.S.: Weight conversion and verification have been performed on ImageNet-1k as well and they are all good. Refer here.

chore: spacing fix/

keras/applications/convnext.py

LukeWood · 2022-05-09T17:45:44Z

Hey @sayakpaul ! I uploaded the weights to our bucket. Now we can update the path in your code and merge the PR. Thanks!

sayakpaul · 2022-05-10T00:46:00Z

@LukeWood done. Thank you!

AdityaKane2001 · 2022-05-11T04:30:37Z

This is great to have in keras.applications!

LukeWood · 2022-05-11T05:07:47Z

This is great to have in keras.applications!

super excited to have it in keras.applications!

sayakpaul added 2 commits April 15, 2022 15:27

feat: initial implementation of convnext.

ce8d99b

chore: added config and cleaned up some code.

2470a0e

google-ml-butler bot added the size:L label Apr 16, 2022

google-ml-butler bot assigned gbaned Apr 16, 2022

sayakpaul commented Apr 16, 2022

View reviewed changes

keras/applications/convnext.py Show resolved Hide resolved

sayakpaul commented Apr 16, 2022

View reviewed changes

keras/applications/convnext.py Show resolved Hide resolved

sayakpaul commented Apr 16, 2022

View reviewed changes

keras/applications/convnext.py Outdated Show resolved Hide resolved

fix: initial convnext implementation.

4d49ca2

fchollet reviewed Apr 17, 2022

View reviewed changes

sayakpaul added 2 commits April 17, 2022 17:56

chore: added doc instantiator.

bfd1af8

chore: applied initial PR feedback.

9844e27

sayakpaul marked this pull request as ready for review April 17, 2022 12:48

sayakpaul mentioned this pull request Apr 17, 2022

Add ConvNeXt family of models to keras.applications #16321

Closed

fchollet reviewed Apr 17, 2022

View reviewed changes

keras/applications/convnext.py Outdated Show resolved Hide resolved

LukeWood self-requested a review April 17, 2022 20:12

google-ml-butler bot added the keras-team-review-pending Pending review by a Keras team member. label Apr 17, 2022

chore: Block -> ConvNeXtBlock.

3b990ae

sayakpaul commented Apr 18, 2022

View reviewed changes

chore: corrected repo link and indentation.

2c1b1e4

gbaned requested a review from fchollet April 18, 2022 10:32

fchollet reviewed Apr 18, 2022

View reviewed changes

feat: added config to convnext block, simplied staging.

dfe6a6b

sayakpaul requested a review from fchollet April 20, 2022 04:22

LukeWood suggested changes Apr 20, 2022

View reviewed changes

gbaned added this to Assigned Reviewer in PR Queue via automation Apr 20, 2022

rchao removed the keras-team-review-pending Pending review by a Keras team member. label Apr 21, 2022

LukeWood suggested changes Apr 29, 2022

View reviewed changes

PR Queue automation moved this from Assigned Reviewer to Reviewer Requested Changes Apr 29, 2022

fix: reconstruction of convnext models from config.

d3ee960

sayakpaul added 2 commits May 6, 2022 12:15

feat: convnext with functional api.

e691392

Merge pull request #2 from sayakpaul/feat/convnext-functional

dd84bc3

feat: convnext with functional api.

sayakpaul requested a review from LukeWood May 6, 2022 07:51

sayakpaul added 2 commits May 6, 2022 13:46

chore: spacing fix/

4b24dc0

Merge pull request #3 from sayakpaul/feat/convnext-functional

ea5f829

chore: spacing fix/

gbaned requested review from fchollet and removed request for fchollet May 6, 2022 14:52

chore: datatype to float.

aa7b4c3

LukeWood added the ready to pull Ready to be merged into the codebase label May 9, 2022

LukeWood approved these changes May 9, 2022

View reviewed changes

google-ml-butler bot added the kokoro:force-run label May 9, 2022

kokoro-team removed the kokoro:force-run label May 9, 2022

LukeWood reviewed May 9, 2022

View reviewed changes

keras/applications/convnext.py Outdated Show resolved Hide resolved

LukeWood removed the ready to pull Ready to be merged into the codebase label May 9, 2022

chore: change weight path.

c9e5b0d

sayakpaul requested a review from LukeWood May 10, 2022 01:16

LukeWood added the ready to pull Ready to be merged into the codebase label May 10, 2022

LukeWood approved these changes May 10, 2022

View reviewed changes

google-ml-butler bot added the kokoro:force-run label May 10, 2022

kokoro-team removed the kokoro:force-run label May 10, 2022

copybara-service bot merged commit 3ff21f8 into keras-team:master May 10, 2022

PR Queue automation moved this from Reviewer Requested Changes to Merged May 10, 2022

sayakpaul deleted the feat/convnext branch May 11, 2022 00:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ConvNeXt models #16421

Add ConvNeXt models #16421

sayakpaul commented Apr 16, 2022 •

edited

fchollet left a comment

sayakpaul commented Apr 17, 2022

sayakpaul Apr 18, 2022

sayakpaul Apr 18, 2022

fchollet Apr 18, 2022

fchollet Apr 18, 2022

sayakpaul commented Apr 19, 2022 •

edited

LukeWood left a comment

LukeWood left a comment

LukeWood Apr 29, 2022

sayakpaul Apr 29, 2022

sayakpaul commented May 6, 2022 •

edited

sayakpaul commented May 6, 2022

LukeWood commented May 9, 2022

sayakpaul commented May 10, 2022

AdityaKane2001 commented May 11, 2022

LukeWood commented May 11, 2022

Add ConvNeXt models #16421

Add ConvNeXt models #16421

Conversation

sayakpaul commented Apr 16, 2022 • edited

fchollet left a comment

Choose a reason for hiding this comment

sayakpaul commented Apr 17, 2022

sayakpaul Apr 18, 2022

Choose a reason for hiding this comment

sayakpaul Apr 18, 2022

Choose a reason for hiding this comment

fchollet Apr 18, 2022

Choose a reason for hiding this comment

fchollet Apr 18, 2022

Choose a reason for hiding this comment

sayakpaul commented Apr 19, 2022 • edited

LukeWood left a comment

Choose a reason for hiding this comment

LukeWood left a comment

Choose a reason for hiding this comment

LukeWood Apr 29, 2022

Choose a reason for hiding this comment

sayakpaul Apr 29, 2022

Choose a reason for hiding this comment

sayakpaul commented May 6, 2022 • edited

sayakpaul commented May 6, 2022

LukeWood commented May 9, 2022

sayakpaul commented May 10, 2022

AdityaKane2001 commented May 11, 2022

LukeWood commented May 11, 2022

sayakpaul commented Apr 16, 2022 •

edited

sayakpaul commented Apr 19, 2022 •

edited

sayakpaul commented May 6, 2022 •

edited