Add Donut

[NielsRogge]

What does this PR do?

This PR adds Donut to the library. Donut is to LayoutLM what T5 is to BERT. :D

The model is implemented as an instance of our existing VisionEncoderDecoderModel.

See also clovaai/donut#10 (comment)

To do:

• move repos to appropriate organization in
• update niels to appropriate organization in test_modeling_vision_encoder_decoder.py

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[sgugger]

Thanks for adding this model but:

1. we do not accept configuration parameters to switch layernorm on and off as it hurts readability of the original model. Thus a new model file should be created here.
2. the processor shouldn't implement as_target_processor and rely on the new API instead.

[NielsRogge]

I have implemented a new DonutSwinModel, that copies everything of SwinModel, except the final layer norm. I've added it in a file called modeling_donut_swin.py (and implemented a corresponding DonutSwinConfig in configuration_donut_swin.py).

I went with modeling_donut_swin.py (and configuration_donut_swin.py) in the "donut" folder rather than modeling_donut.py (and configuration_donut.py) since it only implements the model and configuration of the encoder part (Swin Transformer). For the decoder, BART is leveraged. Let me know if this is ok.

[sgugger]

This one should be "DonutSwin", you can leave another one for the full model (("donut", "Donut")).

[NielsRogge]

Ok, so when I add ("donut", "Donut") and ("donut-swin", "DonutSwin") here, the big table gets updated to DonutSwin (which should actually be Donut)

[NielsRogge]

Gently pinging @sgugger here, as it's my last question before this PR can be merged.

[WaterKnight1998]

Hi @NielsRogge , do you plan on supporting: Document Parsing modality?

[alaradirik]

Great work! I think the PR is in good shape but I have some general comments about the structuring.

It'd be great to move the utility methods in feature_extraction_donut.py to image_utils and also to rethink the naming conventions in test_modeling_vision_encoder_decoder.py if we want to add new models with different base models.

[alaradirik]

Please see the comment above, this might be better suited for the image_utils file

[alaradirik]

I think the reshaped_hidden_states argument might be a bit redundant, do we really need it?

[NielsRogge]

These were added to the original Swin Transformer, to make sure the model can be used as a backbone (to return hidden states of shape (batch_size, num_channels, height, width) à la convolutional models rather than (batch_size, sequence_length, hidden_size) à la Transformer models.

However, for Donut these may not be necessary, so I can remove them if required.

[alaradirik]

Please see the comment above

[gwkrsrch]

@NielsRogge I found a typo here, "s_cord-v2>"->"<s_cord-v2>".