Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add reference to entropy implementation used #3229

Merged
merged 9 commits into from
Jan 29, 2024
Merged

Add reference to entropy implementation used #3229

merged 9 commits into from
Jan 29, 2024

Commits on Jan 26, 2024

  1. Add reference to entropy implementation used 

    Making it more clear that the entropy implementation in NLTK is the one based on the Shannon-McMillan-Breiman theorem, as used and referenced by Jurafsky and Jordan Boyd-Graber.
    @mbauwens
    mbauwens committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    29269e8 View commit details
    Browse the repository at this point in the history

  2. Consistently use full names 

    Co-authored-by: Ilia Kurenkov <ilia.kurenkov@gmail.com>
    @mbauwens @iliakur
    mbauwens and iliakur committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    b4de24f View commit details
    Browse the repository at this point in the history

  3. Consistency with other docstrings in the module 

    Co-authored-by: Ilia Kurenkov <ilia.kurenkov@gmail.com>
    @mbauwens @iliakur
    mbauwens and iliakur committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    460ed15 View commit details
    Browse the repository at this point in the history

  4. first draft of extended entropy method 

    add entropy_extended with length normalisation and relative frequency weighting. Also adapt perplexity_extended to work with this new entropy measure.
    @mbauwens
    mbauwens committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    d645382 View commit details
    Browse the repository at this point in the history

  5. Added unk flag and rel freq weighting

    @mbauwens
    mbauwens committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    90fdd00 View commit details
    Browse the repository at this point in the history

  6. Cleaning up draft

    @mbauwens
    mbauwens committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    7bea41f View commit details
    Browse the repository at this point in the history

  7. remove draft from PR

    @mbauwens
    mbauwens committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    8518e58 View commit details
    Browse the repository at this point in the history

  8. Update api.py

    @mbauwens
    mbauwens committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    bdf2410 View commit details
    Browse the repository at this point in the history

  9. Update api.py

    @mbauwens
    mbauwens committed Jan 26, 2024
    Configuration menu
    Copy the full SHA
    a4ac97e View commit details
    Browse the repository at this point in the history