NLLB tokenizer

[LysandreJik]

Adds the NLLB tokenizer. In order to run:

>>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, pipeline

>>> model = AutoModelForSeq2SeqLM.from_pretrained("facebook/nllb-200-distilled-600M")
>>> tokenizer = AutoTokenizer.from_pretrained("facebook/nllb-200-distilled-600M")

>>> translator = pipeline('translation', model=model, tokenizer=tokenizer, src_lang="eng_Latn", tgt_lang='ron_Latn')
>>> translator("UN Chief says there is no military solution in Syria")
[{'translation_text': 'Şeful ONU spune că nu există o soluţie militară în Siria'}]

Closes #18043

[LysandreJik]

All models are now public, feel free to try it out @stefan-it. The generation seems good, have not tried fine-tuning yet.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[vmarsel]

I don't know of a better place to post (issue?), so I'll do it here :)

@LysandreJik Thank you so much for adding support for the NLLB dense models! I pulled out this branch and tried all of them and they work awesome!

There is the following place in the readme
"This implementation contains dense models available in release. Let us know via GitHub if you want to see MoE models as well."

So it would be really great if you could add MoE models! I tried to figure out the original repo, but it turned out to be unexpectedly difficult. I couldn't get MoE to run. So if you add MoE models, I'm sure it will make a lot of people happier, at least me :)

[TonyMas]

@LysandreJik Thanks a lot for your promt work! I tried using NLLB model from HuggingFace and noticed one problem:

max_length does not set in config.json for any of the NLLB models, so it uses default value of max_length (20).

transformers/src/transformers/configuration_utils.py

Line 125 in 33028f4

max_length (`int`, *optional*, defaults to 20):

As the result, your example code cannot generate more than 20 tokens. it is possible to set max_length higher when calling translation method, but it will be great to have meaningful default as well.

For comparison, both for M2M and MBart50 models max_length set in config.json file to 200.

[TroyZuroske]

@LysandreJik Thanks a lot for your promt work! I tried using NLLB model from HuggingFace and noticed one problem:

max_length does not set in config.json for any of the NLLB models, so it uses default value of max_length (20).

transformers/src/transformers/configuration_utils.py

Line 125 in 33028f4

max_length (`int`, *optional*, defaults to 20):

As the result, your example code cannot generate more than 20 tokens. it is possible to set max_length higher when calling translation method, but it will be great to have meaningful default as well.
For comparison, both for M2M and MBart50 models max_length set in config.json file to 200.

How is the default max_length determined per model? Or is it documented in their white papers? With this PR, I have started evaluating the extremely large model (facebook/nllb-200-3.3B) against GCP translation and so far it is doing really well despite the length of text I give it but I want to give it the best chance to perform so knowing the ideal max_length would help.

[TonyMas]

@LysandreJik Thanks a lot for your promt work! I tried using NLLB model from HuggingFace and noticed one problem:
max_length does not set in config.json for any of the NLLB models, so it uses default value of max_length (20).

transformers/src/transformers/configuration_utils.py

Line 125 in 33028f4

max_length (`int`, *optional*, defaults to 20):

As the result, your example code cannot generate more than 20 tokens. it is possible to set max_length higher when calling translation method, but it will be great to have meaningful default as well.
For comparison, both for M2M and MBart50 models max_length set in config.json file to 200.

How is the default max_length determined per model? Or is it documented in their white papers? With this PR, I have started evaluating the extremely large model (facebook/nllb-200-3.3B) against GCP translation and so far it is doing really well despite the length of text I give it but I want to give it the best chance to perform so knowing the ideal max_length would help.

I think usual default for max_length is to be equal to max input length. Translation pipeline in transformers are checking that max_length at higher than 90% of input length.

transformers/src/transformers/pipelines/text2text_generation.py

Lines 272 to 278 in 33028f4

	def check_inputs(self, input_length: int, min_length: int, max_length: int):
	if input_length > 0.9 * max_length:
	logger.warning(
	f"Your input_length: {input_length} is bigger than 0.9 * max_length: {max_length}. You might consider "
	"increasing your max_length manually, e.g. translator('...', max_length=400)"
	)
	return True

[sgugger]

Thanks for adding this model! Would it make sense to add a default model for NLLB in the auto mappings?

[vince62s]

as mentionned here #19943
where did you guys see that the " Langtoken" is added AFTER the tokens ?
In the NLLB paper, it says only the "Langtoken" is placed BEFORE the tokens. (mBart does the opposite)

[stefan-it]

I've just seen this example - where the lang-token is prepended:

https://github.com/facebookresearch/fairseq/blob/nllb/fairseq/data/multilingual/multilingual_data_manager.py#L78-L101

from original code base 🤔

[vince62s]

right. Also I am wondering why they use "" which is "eos" as the start token of the source sequence. (in fact same for the target sequence). I would have expected:
SRC = LangTok + tokens
TGT = BOS + LangTok, tokens + EOS

It seems they use EOS instead of BOS and that they put a EOS as the SRC start.