PegasusTokenizer returning None

[akashe]

Environment info

• transformers version:
• Platform: Ubuntu 20.04
• Python version: Python 3.7.10
• PyTorch version (GPU?): 1.8.1+cu101
• Tensorflow version (GPU?):
• Using GPU in script?: Problem in both CPU and GPU
• Using distributed or parallel set-up in script?: No

Who can help @patrickvonplaten @LysandreJik

Information

Model I am using (Bert, XLNet ...): Pegasus

The problem arises when using:

• the official example scripts: (give details below)
• my own modified scripts: (give details below)

The tasks I am working on is:

• an official GLUE/SQUaD task: (give the name)
• my own task or dataset: (give details below)

To reproduce

Steps to reproduce the behavior:

1. Go to https://huggingface.co/transformers/model_doc/pegasus.html#pegasusforconditionalgeneration
2. Run the summarization example in the section
3. PegasusTokenizer.from_pretrained('google/pegasus-xsum') returns None. PegasusTokenizer also returns None for 'google/pegasus-large'

Expected behavior

Should return a non None value.

[patrickvonplaten]

Hey @akashe,

Think this error is analogs to this one: #8864.

Installing sentencepiece should solve the problem :-)

#8864

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[nikhilmenon06]

Hey @akashe,

Think this error is analogs to this one: #8864.

Installing sentencepiece should solve the problem :-)

#8864

Still does not seem to work, even after installing sentencepiece

[crackedpotato007]

Same here ;(

[patrickvonplaten]

Could you please update to the newest transformers version and check again? I cannot reproduce the error sadly

[akashe]

Hi @patrickvonplaten, checked with the newest transformers. Tokenizer is not returning None.

[smahm094]

@akashe did you solve the problem later? I am having the same issue.

[akashe]

Update to the newest version. It worked after that.

[Ahmedniz1]

I got the same issue first, of getting Nonetype. To solve this, just install sentencepiece, and make sure to restart runtime.