Calibration: added expected calibration error (ECE)

[yanielc]

What does this implement/fix? Explain your changes.

This PR implements the Expected Calibration Error (ECE) as a function in the sklearn.calibration module. ECE is a widely used metric used to quantify the calibration performance of probabilistic machine learning models, particularly in binary classification tasks. It measures the discrepancy between predicted probabilities and empirical probabilities across different confidence levels.

It is used both in "classical" machine learning and deep learning, for example:

• "Obtaining Well Calibrated Probabilities Using Bayesian Binning", see PDF at https://ojs.aaai.org/index.php/AAAI/article/view/9602

• The widely cited "On Calibration of Modern Neural Networks" , see PDF at https://arxiv.org/pdf/1706.04599.pdf. The ECE is defined in equations 2 and 3.

Computing the ECE follows the same initial steps as the calibration curve. So I place the common code into a new function called calibration_stats which is then called by both calibration_curve and expected_calibration_error.

Any other comments?

I have not added unit tests yet. Sending the PR to get some feedback before proceeding.

[github-actions]

✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

_{Generated for commit: 0c5ab33. Link to the linter CI: here}

[adrinjalali]

cc @glemaitre @ogrisel @lorentzenchr

[lorentzenchr]

This is a duplicate of #11096, see also the discussion in the issue #18268.

[yanielc]

Ah ok, I completely missed that other PR. Should I close my PR? On the other hand, #11096 hasn’t had work in a long while.

…

On Mon 15. Apr 2024 at 21.53, Christian Lorentzen ***@***.***> wrote: This is a duplicate of #11096 <#11096>, see also the discussion in the issue #18268 <#18268>. — Reply to this email directly, view it on GitHub <#28831 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHYMNFZDTVUTHXZ7725MXGDY5QO23AVCNFSM6AAAAABGGD2UKOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANJXGU4TEOBVG4> . You are receiving this because you authored the thread.Message ID: ***@***.***>