Add reference to entropy implementation used #3229
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Making it more clear that the entropy implementation in NLTK is the one based on the Shannon-McMillan-Breiman theorem, as used and referenced by Jurafsky and Jordan Boyd-Graber.
iliakur
requested changes
Jan 26, 2024
Co-authored-by: Ilia Kurenkov <ilia.kurenkov@gmail.com>
Co-authored-by: Ilia Kurenkov <ilia.kurenkov@gmail.com>
mbauwens
commented
Jan 26, 2024
add entropy_extended with length normalisation and relative frequency weighting. Also adapt perplexity_extended to work with this new entropy measure.
iliakur
reviewed
Jan 26, 2024
|
@iliakur yeah, so a bit messy here, but I've reverted the PR to just the update to the docs. I now have a separate branch on my fork that I'll be getting ready for a PR for an alternative entropy/perplexity method, including tests :) |
iliakur
approved these changes
Jan 27, 2024
|
@stevenbird @tomaarsen this PR is ready to merge. I've been away from nltk development long enough that I figured I'd check in with you two about any formalities that may be missing. |
|
I'm afraid I've also not had much time for NLTK recently. That said, I think this is good to go. Thanks for taking care of this, and nice to see you here again @mbauwens.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is in reference to issue #2961.
This is just a simple addition to the docs, making it more clear that the entropy implementation in NLTK is the one based on the Shannon-McMillan-Breiman theorem, as used and referenced by Jurafsky (Chapter 3 page 23 - 3.49) and Jordan Boyd-Graber (
lm.txtfile in this comment).This PR does not include the alternative implementation of the entropy function that was proposed in the issue.