Commits on Dec 12, 2020

1. Add first working version of SBCS model training code

   

   dan-blanchard committed Dec 12, 2020
   Copy the full SHA

   5d91d93 View commit details

   Browse the repository at this point in the history

2. Greatly improved language model training code. Can now use Wikipedia

   

   dan-blanchard committed Dec 12, 2020
   Copy the full SHA

   c530b58 View commit details

   Browse the repository at this point in the history

3. Add SingleByteCharSetModel for use by new model training code

   

   dan-blanchard committed Dec 12, 2020
   Copy the full SHA

   b72e0d3 View commit details

   Browse the repository at this point in the history

4. Fix a bunch of minor issues with create_language_model

   

   dan-blanchard committed Dec 12, 2020
   Copy the full SHA

   36cdf8c View commit details

   Browse the repository at this point in the history

5. Don't crawl more than 20k wiki articles per language in create_langua… …

   …ge_model
   
   This is in case we encouter some really crazy article with millions of links,
   but it's also nice for debugging.

   

   dan-blanchard committed Dec 12, 2020
   Copy the full SHA

   51c13a2 View commit details

   Browse the repository at this point in the history

6. Retrain all single byte charset models and fix some training and dete… …

   …ction issues

   

   dan-blanchard committed Dec 12, 2020
   Copy the full SHA

   d0f72fc View commit details

   Browse the repository at this point in the history

7. Remove codecs from MISSING_ENCODINGS in test.py now that we're not mi… …

   …ssing them

   

   dan-blanchard committed Dec 12, 2020
   Copy the full SHA

   ef79d4c View commit details

   Browse the repository at this point in the history

8. Fix potential crash in create_language_model.py

   

   dan-blanchard committed Dec 12, 2020
   Copy the full SHA

   382ad9a View commit details

   Browse the repository at this point in the history

9. Add helour's Latin-1 prober from abandoned PR, possibly temporarily

   

   dan-blanchard committed Dec 12, 2020
   Copy the full SHA

   41732cd View commit details

   Browse the repository at this point in the history

10. Temporarily comment out Czech model because it seems wrong.

    

    dan-blanchard committed Dec 12, 2020
    Copy the full SHA

    b0de2d6 View commit details

    Browse the repository at this point in the history

11. Fix a bad merge conflict resolution

    

    dan-blanchard committed Dec 12, 2020
    Copy the full SHA

    67bc4bb View commit details

    Browse the repository at this point in the history

12. Make create_language_model a bit more reliable, and make it dump wiki… …

    …pedia text so we do not have to download it a bunch of times

    

    dan-blanchard committed Dec 12, 2020
    Copy the full SHA

    7a3636c View commit details

    Browse the repository at this point in the history

13. Try retraining again after some bugfixes to training script

    

    dan-blanchard committed Dec 12, 2020
    Copy the full SHA

    d45f781 View commit details

    Browse the repository at this point in the history

14. Count symbols and digits for total characters in SBCS prober

    

    dan-blanchard committed Dec 12, 2020
    Copy the full SHA

    4503b74 View commit details

    Browse the repository at this point in the history

15. Black formatting

    

    dan-blanchard committed Dec 12, 2020
    Copy the full SHA

    eac7414 View commit details

    Browse the repository at this point in the history