New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix issues with LitGPT Tokenizer (SentencePiece and HF Tokenizers) #1396

Draft

Andrei-Aksionov wants to merge 2 commits into Lightning-AI:main from Andrei-Aksionov:tokenizer_fix_hf_and_sentencepiece

Collaborator

Andrei-Aksionov commented May 7, 2024

Hi there 👋

While I was working on #1341, I noticed a couple of strange things with the tokenizer:

If to make a prompt a bit more complex (by adding special tokens from the prompt) then the output doesn't match to what HF AutoTokenizer outputs. It is true for both SentencePiece and Tokenizers.
If to force Phi-3 to use Tokenizers library, then generate.base outputs something useful, while chat.base doesn't.
Test for tokenizers checks only SentencePiece. We need to test both SentencePiece and Tokenizers.

Andrei-Aksionov added 2 commits

May 6, 2024 13:07


          Add special characters from prompt template


          Merge branch 'main' into tokenizer_fix_hf_and_sentencepiece

9b3f371

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment