Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TVM is expecting tokenizer.json for HF models. But HF repo (e.g. : Helsinki-NLP/opus-mt-it-en) contains only .spm files based on sentencepiece model #16671

Open
kamal-kg opened this issue Mar 4, 2024 · 0 comments
Open

TVM is expecting tokenizer.json for HF models. But HF repo (e.g. : Helsinki-NLP/opus-mt-it-en) contains only .spm files based on sentencepiece model #16671

kamal-kg opened this issue Mar 4, 2024 · 0 comments
Labels
needs-triage PRs or issues that need to be investigated by maintainers to find the right assignees to address it type: bug

Comments

@kamal-kg
Copy link

kamal-kg commented Mar 4, 2024

How can I upload tokenizer.json file in TVM for translation inference (e.g. : Helsinki-NLP/opus-mt-it-en). Because .json file does not contain merge information and sentencepiece learned weights.
Please help to resolve this issue.
@kamal-kg kamal-kg added needs-triage PRs or issues that need to be investigated by maintainers to find the right assignees to address it type: bug labels Mar 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
needs-triage PRs or issues that need to be investigated by maintainers to find the right assignees to address it type: bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@kamal-kg