Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Resolve IndexError in sent_tokenize #2922

Merged
merged 2 commits into from Dec 21, 2021
Merged

Resolve IndexError in sent_tokenize #2922

merged 2 commits into from Dec 21, 2021

Conversation

tomaarsen
Copy link
Member

Closes #2921

Hello!

Pull request overview

  • Prevent IndexError when the sent_tokenize input starts with a dot.
  • Wrote tests to prevent this from failing again.

Details

It is possible that split is empty whenever . is matched, which can happen whenever the input of sent_tokenize starts with a ., in the following snippet:

nltk/nltk/tokenize/punkt.py

Lines 1380 to 1382 in dd1494e

split = text[: match.start()].rsplit(maxsplit=1)
before_start = len(split[0]) if len(split) == 2 else 0
before_words[match] = split[-1]

This PR ought to resolve this issue.

  • Tom Aarsen

@tomaarsen tomaarsen merged commit d4d99b4 into nltk:develop Dec 21, 2021
@tomaarsen tomaarsen deleted the bugfix/punkt-patch branch December 21, 2021 14:08
@tomaarsen tomaarsen mentioned this pull request Dec 21, 2021
Closed
ExplorerFreda added a commit to ExplorerFreda/nltk that referenced this pull request Dec 24, 2021
@ExplorerFreda
Merge branch 'develop' of https://github.com/ExplorerFreda/nltk into …
6c99522 
…develop

* 'develop' of https://github.com/ExplorerFreda/nltk:
  Temporarily pause Python 3.10 CI tests due to scikit-learn issues with Windows
  Resolve IndexError in `sent_tokenize` (nltk#2922)
  Drop support for Python 3.6, support Python 3.10 (nltk#2920)
  updates for 3.6.6
  minor clean ups
  updates for 3.6.6
@tomaarsen tomaarsen mentioned this pull request Dec 24, 2021
Closed
@tomaarsen tomaarsen mentioned this pull request Jan 21, 2022
Closed
@tianjianjiang tianjianjiang mentioned this pull request Jan 21, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Sentence tokenizer fails when sentence.startswith('. ')
1 participant
@tomaarsen