New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Disable ASCII encoding when building the model file #98

Merged

tushuhei merged 3 commits into main from ensure_ascii

Dec 2, 2022

Member

tushuhei commented Dec 2, 2022 •

edited

Disabling ASCII encoding when dumping model files for following reasons:

File size reduction
- Japanese: 7.7KB → 6.2KB
- Simplified Chinese: 26KB → 22KB
Better debugging

tushuhei added 3 commits

December 2, 2022 11:13


          ensure_ascii=False in outputing the model

f65b4ec


          Convert existing model files

37d01b5


          Ensure reading/writing files in UTF-8

ff718a4

tushuhei merged commit 393791f into main

tushuhei deleted the ensure_ascii branch

December 14, 2022 02:49

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment