Add YOLOS

[NielsRogge]

What does this PR do?

This PR adds YOLOS, an awesome and simple object detector.

YOLOS is just a single Transformer encoder (ViT), trained using DETR's objective.

For now, I've used "vit" as base_model_prefix, in order to easily load weights from ViT and ViTMAE checkpoints on the hub.

[sgugger]

Thanks for adding this new model! There are a few badly named variables left and some docstrings to fix, but overall it's in great shape!

[NielsRogge]

Addressed most comments. The remaining comments are about badly formatted docstrings, however these are all copied from DETR (so I can't change them due to #Copied from statements). Is it ok if I address these docstrings in a separate PR for both models?

Also pinging @Narsil as the pipeline test for YOLOS is failing. This is because YOLOS doesn't take pixel_mask as input, whereas DETR does. This makes YOLOS fail for the object detection pipeline.

[sgugger]

I'd advocate to make the changes in docstrings in DETR to be propagated to YOLOS in this PR, just to make sure we don't forget.

[Narsil]

Also pinging @Narsil as the pipeline test for YOLOS is failing. This is because YOLOS doesn't take pixel_mask as input, whereas DETR does. This makes YOLOS fail for the object detection pipeline.

Then the feature_extractor should not output them. The image pipeline are pretty simple and roughly simply do

model(**feature_extractor(image)) so if the feature extractor only outputs what's needed then it should work.
That or pixel_mask should be handled (doesn't seem to be making sense for this model reading your comment).

[NielsRogge]

Then the feature_extractor should not output them.

Yeah the problem is, YOLOS uses the same feature extractor as DETR, which outputs both pixel_values and pixel_mask. Hence, I've just added ("yolos", "DetrFeatureExtractor") to the Auto Feature Extractor API.

I think the easiest here is to add pixel_mask=None to the forward of YOLOS, in order to make model(**feature_extractor(image)) work.

[sgugger]

I think the easiest here is to add pixel_mask=None to the forward of YOLOS, in order to make model(**feature_extractor(image)) work.

We're not adding an argument that will be ignored all the time, that's just confusing to users. Especially if they end up passing one and don't get why it's not used.

If the feature extractor should not return pixel_mask then either use a class attribute on DetrFeatureExtractor to make it not return that in certain cases, or create a YolosFeatureExtractor that removes that field from the output of the feature extractor.

[NielsRogge]

Ok so I created a new YolosFeatureExtractor, however the pipeline test is still failing:

def _call_impl(self, *input, **kwargs):
        forward_call = (self._slow_forward if torch._C._get_tracing_state() else self.forward)
        # If we don't have any hooks, we want to skip the rest of the logic in
        # this function, and just call forward.
        if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
                or _global_forward_hooks or _global_forward_pre_hooks):
>           return forward_call(*input, **kwargs)
E           TypeError: forward() got an unexpected keyword argument 'pixel_mask'

@Narsil could you help me debug this? It's weird cause YolosFeatureExtractor doesn't create a pixel mask. Also, I added doc tests which are passing.

[Narsil]

@Narsil could you help me debug this? It's weird cause YolosFeatureExtractor doesn't create a pixel mask. Also, I added doc tests which are passing.

I have checked and the reason is that the tested Feature extractor is actually a detr one, not a Yolo one:

https://github.com/huggingface/transformers/pull/16848/files#diff-fcbe32a3a065f97b00f1c242ecd45858b8d5680a2437b65af183eb0c439e2be9R347

The pipeline tests rely on ModelTester to create the base objects

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[NielsRogge]

Failing test is unrelated, merging.