Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor LanguageModel class, adding split functionality and unit tests #3209

Open
wants to merge 1 commit into
base: develop
Choose a base branch
from
Open

Refactor LanguageModel class, adding split functionality and unit tests #3209

wants to merge 1 commit into from

Commits on Nov 29, 2023

  1. Refactor LanguageModel class, adding split functionality and unit tests 

    Enhanced LanguageModel class to include a split method for text processing based on English word frequencies. The split function breaks text into words and punctuations using a language model based on English word frequencies. Additionally, added comprehensive pytest unit tests to validate the split functionality under various scenarios, including punctuation and contractions.

- Implemented split method in LanguageModel for text processing
- Added unit tests for split functionality covering punctuation and contractions
- Ensured adherence to PEP8 standards and code readability

Resolves: nltk#3195
    @venkat1924
    venkat1924 committed Nov 29, 2023
    Configuration menu
    Copy the full SHA
    95eab3a View commit details
    Browse the repository at this point in the history