Test decision tree pickle for different endianness

[lesteve]

Reference Issues/PRs

Close #21359

What does this implement/fix? Explain your changes.

This adds a test that checks that pickles containing big endian arrays can be loaded on a little-endian machine.

The way the pickles are created is to change the Pickle dispatch_table or to reimplement NumpyPickle.save for joblib pickles. This creates pickles with numpy arrays in big endian, which does not match exactly a pickle created on a big endian machine. The main advantage IMO is that we have some test for #17644 ...

I thought it was simpler that using qemu inside a docker image (look at #21237 (comment) for more details) as mentioned in #19602 (comment).

Doing something similar for more (or even all estimators) as mentioned in #19602 (comment) is left for a future PR.

[ogrisel]

LGTM!

And +1 for a similar test (+fix) for #19602 in a follow-up PR to support pickling of tree structures that have system specific precision levels in their integer fields.

[lesteve]

And +1 for a similar test (+fix) for #19602 in a follow-up PR to support pickling of tree structures that have system specific precision levels in their integer fields.

Yep I was working on it in parallel since this touches the same code, I opened a draft PR with a similar approach: #21552. This is better to merge this one first.

[rth]

Very nice! Thanks a lot @lesteve !

Maybe this should be put somewhere in common tests? Or at least in some folder where it would make sense to run it parametrized for a few estimators (I'm thinking also KDTree/BallTree #21553 ). Otherwise LGTM.

[lesteve]

Maybe this should be put somewhere in common tests? Or at least in some folder where it would make sense to run it parametrized for a few estimators (I'm thinking also KDTree/BallTree #21553 ). Otherwise LGTM.

I have this in mind eventually but for now my current strategy is:

• have this PR merged
• get Support cross 32bit/64bit pickles for decision tree #21552 merged (for now there seems to be some subtleties on Windows 32 bit that I need to investigate)
• look at Cross endianness and bitness pickle issues with KNeighborsClassifier / KDTree #21553 in more details and fix it
• once this is done and with the experience of having fixed a similar issue in two different places, revisit adding a common test about this kind of endianness 32bit/64bit issues

[rth]

OK, fair enough. Thanks!