AttributeError: 'NoneType' object has no attribute 'from_pretrained'

[louisabraham]

This code was working yesterday but doesn't work today:

from transformers import AutoTokenizer
AutoTokenizer("Helsinki-NLP/opus-mt-en-fr")

[jacampo]

Same here a couple of hours ago

[LysandreJik]

1. Hi, could you please provide the information related to your environment?

2. When you say it was working yesterday but was working before, do you mean to say you've upgraded to version v4.0.0 released yesterday? If this is so, you may be obtaining the following error message: AttributeError: 'NoneType' object has no attribute 'from_pretrained'. This would be because you do not have sentencepiece installed.

3. Are you sure this worked previously? This should never have worked, as AutoTokenizer cannot be initialized like this, but has to be instantiated from the from_pretrained method:

from transformers import AutoTokenizer
AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-en-fr")

which works on v4.0.0 and on master, as long as you have SentencePiece installed.

[LysandreJik]

Putting a better error message in #8881.

[louisabraham]

Right, I was using

AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-en-fr")

Thanks, pip install sentencepiece fixed the issue!

It looks that previously the tokenizer outputted torch tensors and now lists. Is this intended? It breaks existing code.

[LysandreJik]

Yes, this was a bug. Tokenizers are framework-agnostic and should not output a specific framework's tensor. The implementation of the Marian tokenizer was not respecting the API in that regard.

Tokenizers can still handle torch tensors, you need to specify that you want them though:

tokenizer(xxx, return_tensors="pt")

I guess in your situation it has to do with the prepare_seq2seq_batch:

tokenizer.prepare_seq2seq_batch(xxx, return_tensors="pt")

[louisabraham]

Thanks!