Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fixed several TreebankWordTokenizer and NLTKWordTokenizer bugs #2877

Merged
merged 7 commits into from Nov 6, 2021
Merged

Fixed several TreebankWordTokenizer and NLTKWordTokenizer bugs #2877

merged 7 commits into from Nov 6, 2021

Commits on Nov 4, 2021

  1. Fixed issue with quote-tokenization, small regression for detokenization

    @tomaarsen
    tomaarsen committed Nov 4, 2021
    Copy the full SHA
    dfeb8ef View commit details
    Browse the repository at this point in the history

  2. Updated double-quote to single quote in doctest output

    @tomaarsen
    tomaarsen committed Nov 4, 2021
    Copy the full SHA
    9a8622f View commit details
    Browse the repository at this point in the history

  3. Resolved issue with 'wanna' absorbing a space too much in (de)tokeniz… 

    …ation
    @tomaarsen
    tomaarsen committed Nov 4, 2021
    Copy the full SHA
    ef4739e View commit details
    Browse the repository at this point in the history

  4. Allow importing TreebankWordDetokenizer from nltk.tokenize

    @tomaarsen
    tomaarsen committed Nov 4, 2021
    Copy the full SHA
    9766dde View commit details
    Browse the repository at this point in the history

  5. Added additional test for span_tokenize

    @tomaarsen
    tomaarsen committed Nov 4, 2021
    Copy the full SHA
    1b854de View commit details
    Browse the repository at this point in the history

  6. Add span_tokenize to NLTKWordTokenizer, like in TreebankWordTokenizer

    @tomaarsen
    tomaarsen committed Nov 4, 2021
    Copy the full SHA
    5c42d83 View commit details
    Browse the repository at this point in the history

  7. Added credits for modifications

    @tomaarsen
    tomaarsen committed Nov 4, 2021
    Copy the full SHA
    0e4ad31 View commit details
    Browse the repository at this point in the history