Lower precision custom estimator checks

[adriangb]

Hi folks,

I'm the (sole 😢) maintainer of https://github.com/adriangb/scikeras. I'm trying to update the library to the latest version of Keras and Scikit-Learn but am running into precision issues with the (very useful, thank you!) check_estimator functionality: https://github.com/adriangb/scikeras/actions/runs/8589370291/job/23535429541?pr=317#step:6:1440

There's several places where the precision (correct me if this is the wrong term here) of Keras just doesn't match what Scikit-Learn expects. I suspect the added complexity of a framework like Keras and loss of precision in some places internally where it may be using float32, etc. compound to cause this, but I think this is okay for real world applications.

Is there any way I can tune check_estimator to relax the tolerances so I can continue to use it in tests? If there isn't currently, would it be reasonable to discuss a way to provide that?

Thanks!

[glemaitre]

We don't have this granularity but this is an issue on our side. We should distinguish API checks and numerical checks and make it opt-in/out for third-party library. We are going to work on that and I know @adrinjalali is also interested in it.

For the moment, the only work around is to use the xfail tag and revisit it in the future.

[adriangb]

Gotcha, thank you for confirming. It would be nice to have some way to still run the numeric tests just at lower precision.

[adrinjalali]

we'll be working on separating API compatibility tests from other common tests, many of which admittedly very scikit-learn specific. For for now, you can do as we do internally and skip those tests:

scikit-learn/sklearn/svm/_classes.py

Lines 356 to 364 in d1d1596

	def _more_tags(self):
	return {
	"_xfail_checks": {
	"check_sample_weights_invariance": (
	"zero sample_weight is not equivalent to removing samples"
	),
	}
	}