Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improved Tokenize documentation + added TokenizerI as superclass for TweetTokenizer #2878

Merged
merged 7 commits into from Nov 21, 2021
Merged

Improved Tokenize documentation + added TokenizerI as superclass for TweetTokenizer #2878

merged 7 commits into from Nov 21, 2021

Commits on Nov 4, 2021

  1. Add span_tokenize to NLTKWordTokenizer, just like in TreebankWordToke… 

    …nizer
    @tomaarsen
    tomaarsen committed Nov 4, 2021
    Copy the full SHA
    2c57d1c View commit details
    Browse the repository at this point in the history

Commits on Nov 7, 2021

  1. Merge branch 'develop' of https://github.com/nltk/nltk into develop

    @tomaarsen
    tomaarsen committed Nov 7, 2021
    Copy the full SHA
    e232fe8 View commit details
    Browse the repository at this point in the history

  2. Added documentation for core tokenization modules

    @tomaarsen
    tomaarsen committed Nov 7, 2021
    Copy the full SHA
    f49107b View commit details
    Browse the repository at this point in the history

  3. Added tokenize_sents method to TweetTokenizer 

    By subclassing it with TokenizerI
    @tomaarsen
    tomaarsen committed Nov 7, 2021
    Copy the full SHA
    6cf7c0a View commit details
    Browse the repository at this point in the history

  4. Resolved documentation indentation issue in tokenize/casual.py

    @tomaarsen
    tomaarsen committed Nov 7, 2021
    Copy the full SHA
    24af735 View commit details
    Browse the repository at this point in the history

  5. Fixed copy-paste issue in tokenize docstring

    @tomaarsen
    tomaarsen committed Nov 7, 2021
    Copy the full SHA
    2218952 View commit details
    Browse the repository at this point in the history

Commits on Nov 20, 2021

  1. Merge branch 'develop' of https://github.com/nltk/nltk into enhanceme… 

    …nt/tokenize_documentation
    @tomaarsen
    tomaarsen committed Nov 20, 2021
    Copy the full SHA
    1de3f72 View commit details
    Browse the repository at this point in the history