Apply dmatrix iteration iterface in PySpark xgboost and support external memory mode

[WeichenXu123]

Apply dmatrix iteration iterface in PySpark xgboost and support external memory mode.

[WeichenXu123]

@trivialfis When will you create first PR "Apply dmatrix iteration iterface in PySpark xgboost " ? If you are busy, I can help to create PR. Thank you.

[trivialfis]

@WeichenXu123 Thank you for the work. Feel free to continue the work on external memory. I just didn't want to have too many conflicts between the implementation of ext memory and the quantile dmatrix. (See the WIP iterator in #8088 ).

[WeichenXu123]

@trivialfis To reduce conflicts, I would like to wait #8088 merged first. :)

[trivialfis]

I need to apply the quantile dmatrix to spark as well.

[WeichenXu123]

I think external memory is useless in most cases.

if memory is not efficient per task, we can increase num_workers param.

[trivialfis]

I agree. I kept the feature here mostly for experimental support.

[trivialfis]

But quantile DMatrix can still be used for reducing memory usage without sacrificing performance.