Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Better warning when loading a tokenizer with AutoTokenizer w/o Sneten… #8881

Merged
merged 1 commit into from Dec 1, 2020
Merged

Better warning when loading a tokenizer with AutoTokenizer w/o Sneten… #8881

merged 1 commit into from Dec 1, 2020

Conversation

LysandreJik
Copy link
Member

…cePiece

Currently, initializing a sentencepiece AutoTokenizer without having sentencepiece installed results in the following error:

AttributeError: 'NoneType' object has no attribute 'from_pretrained'

This improves the error message to:

 This tokenizer cannot be instantiated. Please make sure you have `sentencepiece` installed in order to use this tokenizer.

Fix #8864

@LysandreJik LysandreJik mentioned this pull request Dec 1, 2020
Closed
sgugger
sgugger approved these changes Dec 1, 2020
View reviewed changes
Copy link
Collaborator

@sgugger sgugger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@LysandreJik LysandreJik merged commit a947386 into master Dec 1, 2020
@LysandreJik LysandreJik deleted the better-sentencepiece-warning branch December 1, 2020 18:13
stas00 pushed a commit to stas00/transformers that referenced this pull request Dec 5, 2020
@LysandreJik @stas00
Better warning when loading a tokenizer with AutoTokenizer w/o Sneten…
5cefe85 
…cePiece (huggingface#8881)
@thomwolf
Copy link
Member

thomwolf commented Dec 9, 2020

Thanks!

LysandreJik added a commit that referenced this pull request Dec 9, 2020
@LysandreJik
Better warning when loading a tokenizer with AutoTokenizer w/o Sneten…
b90c684 
…cePiece (#8881)
LysandreJik added a commit that referenced this pull request Dec 9, 2020
@LysandreJik
Better warning when loading a tokenizer with AutoTokenizer w/o Sneten…
e639932 
…cePiece (#8881)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sgugger sgugger sgugger approved these changes

@thomwolf thomwolf Awaiting requested review from thomwolf

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

AttributeError: 'NoneType' object has no attribute 'from_pretrained'
3 participants
@LysandreJik @thomwolf @sgugger