Add Qwen support #850
Add Qwen support #850
Conversation
chenht2026
requested review from
awaelchli,
carmocca and
lantiga
as code owners
chenht2026
force-pushed
the
main
branch
3 times, most recently
from
4823c6a
to
2fe1ec3
Compare
|
Thank you @chenht2026, sorry for the wait. A few of us took a break :-) |
| @@ -27,6 +27,8 @@ class Config: | |||
| rotary_percentage: float = 0.25 | |||
| parallel_residual: bool = True | |||
| bias: bool = True | |||
| # just for Qwen | |||
| is_Qwen: Optional[bool] = None | |||
There was a problem hiding this comment.
I think we should avoid this in favor of something that characterizes Qwen, like having bias only in c_attn.
For the time being we could rename this as attn_bias, and then in the future turn bias into a Option[bool, List[str]] if there's a need for it.
| # Copyright (c) Alibaba Cloud. | ||
| # | ||
| # This source code is licensed under the license found in the | ||
| # LICENSE file in the root directory of this source tree. |
There was a problem hiding this comment.
If you didn't write the code from this file (which I assume you were not since you did add this license), you should link it to the original source. Is the original from PaddlePaddle? https://github.com/PaddlePaddle/PaddleNLP/blob/develop/paddlenlp/transformers/qwen/tokenizer.py
I would advice that you create a version that only implements that few methods required by tokenizer.py
There was a problem hiding this comment.
I just copy it from their huggingface repo. tokenization_qwen.py
| from typing import Collection, Dict, List, Set, Tuple, Union | ||
|
|
||
| import tiktoken | ||
| from transformers import PreTrainedTokenizer, AddedToken |
There was a problem hiding this comment.
This project doesn't have transformers as a dependency, so this import is not possible
| @@ -91,6 +95,8 @@ def encode( | |||
| tokens = self.processor.encode(string).ids | |||
| elif self.backend == "sentencepiece": | |||
| tokens = self.processor.encode(string) | |||
| elif self.backend == "tiktoken": | |||
| tokens = self.processor.encode(string) | |||
There was a problem hiding this comment.
It doesn't seem like the new processor implements this method. Also, what about decoding?
tutorials/download_Qwen.md
Outdated
There was a problem hiding this comment.
It would be good mention qwen's recommended languages.
There was a problem hiding this comment.
Chinese and English. I'll add it.
Would it work with Qwen 2 (Qwen/Qwen1.5-7B-Chat)? if not what needs to be added? |
It works.
Qwen's tokenizer is based on tiktoken, I add the tokenizer(tokenization_qwen.py) from its huggingface repo without any revision. This make the code a little complicate, so maybe do not merge.
May be some one needs it.
Closes #840