Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Resolve IndexError in sent_tokenize #2922

Merged
merged 2 commits into from Dec 21, 2021
Merged

Resolve IndexError in sent_tokenize #2922

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4c046a2
Prevent IndexError if input starts with an endline character
tomaarsen Dec 21, 2021
d9dbb52
Add doctest for Punkt sent_tokenize issue
tomaarsen Dec 21, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
5 changes: 5 additions & 0 deletions nltk/test/tokenize.doctest
View file
Open in desktop
Expand Up @@ -310,6 +310,11 @@ Testing mutable default arguments for https://github.com/nltk/nltk/pull/2067
>>> type(pst._lang_vars)
<class 'nltk.tokenize.punkt.PunktLanguageVars'>

Testing that inputs can start with dots.

>>> pst = PunktSentenceTokenizer(lang_vars=None)
>>> pst.tokenize(". This input starts with a dot. This used to cause issues.")
['.', 'This input starts with a dot.', 'This used to cause issues.']

Regression Tests: align_tokens
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Expand Down
2 changes: 1 addition & 1 deletion nltk/tokenize/punkt.py
View file
Open in desktop
Expand Up @@ -1379,7 +1379,7 @@ def _match_potential_end_contexts(self, text):
# Find the word before the current match
split = text[: match.start()].rsplit(maxsplit=1)
before_start = len(split[0]) if len(split) == 2 else 0
before_words[match] = split[-1]
before_words[match] = split[-1] if split else ""
matches.append(match)

return [
Expand Down