DeviceQuantileDMatrix Improvements

[jamescolless]

When training on large amounts of data using a gpu the DeviceQuantileDMatrix is extremely useful in helping to avoid memory issues. In such situations it is however quite unfortunate that it is limited to 2^31-1000 elements (e.g. it cannot handle 10 million samples with 250 features). Is there any way this limitation might be removed?

Similarly it would be great if DeviceQuantileDMatrix was able to be used during external memory training (it seems as if only DMatrix is currently supported).

[hcho3]

The Thrust library (a dependency of XGBoost) currently has an integer overflow bug that prevents us from creating DeviceQuantileDMatrix that's larger than 2^31 elements. See #6228. To enable DeviceQuantileDMatrix larger than 2^31 elements, the integer overflow bug in Thrust needs to be fixed.

[jamescolless]

I see, thanks! Hopefully that gets resolved soon.

[hcho3]

Will be fixed in #6234