Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: Allow category-wise Fleiss' kappas. #9242

Draft
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

jseabold
Copy link
Member

@jseabold jseabold commented May 9, 2024

The API is a bit of a mess with many conditional returns.

Probably worth adding a bunch return and maybe a separate function with bells and whistles and deprecating the old one maybe?

Includes #9241, so it should go in after that. Will leave as draft until that's merged.

  • tests added / passed.
  • code/documentation is well formatted.
  • properly formatted commit message. See
    NumPy's guide.

@josef-pkt
Copy link
Member

josef-pkt commented May 10, 2024

Thanks @jseabold

We have Holder and HolderTuple which are now returned by most newer functions.
HolderTuple is backwards compatible with a tuple return (allows indexing and assigning to tuple) but not when previous return was only one value.

For transition and ambiguous cases where extra information might not be a priority, I had used return_results=False as keyword.
Otherwise, I now often return always extra results (if they are cheap) and intermediate results in a Holder instance.
If extra information is more expensive, I have been adding them recently as methods to a subclass of Holder, similar to the model results instance.

The Holder classes have a __repr__ that list all attributes with values.

Aside: Holder and HolderTuple do not subclass python dict or tuple, because I only wanted to support/maintain explicitly defined methods and not to have to worry about inherited methods from dict or tuple.

@jseabold
Copy link
Member Author

jseabold commented May 15, 2024

Rebased on PR with results class returned.

Made it an error to request Randolph's Kappa with details. Not sure it makes much sense to assume a marginal 50/50 chance between a single category vs. all others?

If that makes sense, will need to remove that code path from the detail helper function, and I don't think I have a test yet for method = 'fleiss' and return_results = False.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants