[MRG] Add partial_fit function to DecisionTreeClassifier

[PSSF23]

Reference Issues/PRs

First step for #18888

What does this implement/fix? Explain your changes.

• Add the partial_fit function to DecisionTreeClassifier

Any other comments?

Collaboration of @neurodata

• Repo for ensemble attempts and benchmarks: https://github.com/PSSF23/SDTF/
• arXiv preprint: https://arxiv.org/abs/2110.08483

Thank you for feedback!

[amueller]

Thanks for the PR. Can you show that this is faster than building the tree from scratch?

[PSSF23]

@amueller I don't think speed is what I have in mind. The VFDT name might cause some confusion, but this PR is more like a preliminary step that allows future algorithms to have a focus on streaming data. In those cases, data samples would come continuously and saving all of them to wait for a batch fitting would be quite expensive.

I will check the time differences in benchmarks though. Thanks for the advice!

[jnothman]

If the goal is online learning, this should be implemented as partial_fit, but you would need to show that multiple calls to partial_fit roughly equate to fitting in batch.

[PSSF23]

@jnothman Thank you for the advice. So another partial_fit function would be better than a parameter in fit? Hope I understand your words right.

I am working on benchmarking with CIFAR-10 and will get back to you when I have satisfactory results!

[glemaitre]

@jnothman Thank you for the advice. So another partial_fit function would be better than a parameter in fit? Hope I understand your words right.

Yes, partial_fit is our current API form online learning.

[thomasjpfan]

Hi @thomasjpfan, I saw that you added feature_names_in to tree attributes, but there is currently no implementations for it in the tree module, right?

The estimators in the tree module sets feature_names_in_ when _validate_data is called:

scikit-learn/sklearn/tree/_classes.py

Lines 165 to 167 in f33fb0a

	X, y = self._validate_data(
	X, y, validate_separately=(check_X_params, check_y_params)
	)

[PSSF23]

Have some tests been disabled? The code should be covered earlier...

[PSSF23]

The test error in test_different_endianness_joblib_pickle doesn't seem to be related:

ValueError: Big-endian buffer not supported on little-endian compiler

[lesteve]

The test error in test_different_endianness_joblib_pickle doesn't seem to be related:

This is actually related although in a not so trivial way. One likely fix would be to change ClassificationCriterion.__cinit__ similarly to https://github.com/scikit-learn/scikit-learn/pull/21552/files#diff-ecb2f5fb06ba7e14c6d06a8fbc811d684eaa534640d2e2f8f0102a1c4d4afca2R588.

More details:

• the test was added recently in main to test deployment cases where the machine you train and pickles your model does not have the same endianness as the machine you unpickle your model to do predictions, see Pickle portability little 🡒 big endian #21237 for more details. There is a variation of this for cross-bitness, you train on a 64 bit machine and deploy on a 32bit machine e.g. Support cross 32bit/64bit pickles for decision tree #21552.
• the change in your PR that triggered this error is that you added self.builder_ to the DecisionTreeClassifier object, so the builder object (and so the criterion object as well) needs to be pickled when pickling the DecisionTreeClassifier object. ClassificationCriterion.__cinit__ is too specific about the dtype of the n_classes it takes.
• this is quite likely that your PR would fail in cross-bitness deployment use cases. The thing is that for cross-bitness, the test is done in a more manual way (basically tweaking the output of __reduce__ to make it look like it has been generated with a different bitness) and the test pass.

[PSSF23]

@lesteve Thanks! I'll look into it. I think the problem is fixed~

[PSSF23]

This line:

X, y = fetch_california_housing(return_X_y=True)

causes the following error, which is definitely unrelated this time:

urllib.error.HTTPError: HTTP Error 403: Forbidden

[glemaitre]

This might be a temporary issue. We are planning to make a release with a retry mechanism at some point.