Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Downloading tokenizers from Docker #299

Open
wants to merge 1 commit into
base: dev
Choose a base branch
from
Open

Downloading tokenizers from Docker #299

wants to merge 1 commit into from

Conversation

valaises
Copy link
Member

No description provided.

@valaises valaises self-assigned this Feb 13, 2024
mitya52
mitya52 requested changes Feb 14, 2024
View reviewed changes
Copy link
Member

@mitya52 mitya52 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry wrong approve, I request changes

mitya52
mitya52 previously approved these changes Feb 14, 2024
View reviewed changes
@valaises
init
8ac530d 
tokenizer_rewrite_path is relative

implemented HF format for get_tokenizer

changelist:
* used aiofiles
* get-tokenizer -> tokenizer
* simplified tokenizer_rewrite_path
* added few TODOs

removed print
@valaises valaises force-pushed the feb13-tokenizer-from-docker branch from 98884d2 to 8ac530d Compare April 16, 2024 15:16
@mitya52 mitya52 mentioned this pull request May 14, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mitya52 mitya52 mitya52 left review comments

@hazratisulton hazratisulton Awaiting requested review from hazratisulton

At least 1 approving review is required to merge this pull request.

Assignees

@valaises valaises

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@valaises @mitya52