[Roadmap] external memory support for tree_method='hist'

[CodingCat]

in the doc we mentioned that Distributed and external memory version only support tree_method=approx

but is it valid now? what's the problem with external memory + hist? I tested but didn't see any issue, did I miss anything?

[CodingCat]

@hcho3

[CodingCat]

hmmm....looks like the training converged too early....but didn't get the idea why it is related to external memory or the row.page has some problem....

[hcho3]

@CodingCat When I wrote the fast hist, I assumed that each DMatrix would have a single block, i.e. entire data fit in the memory. External memory support has not been tested. Is external memory high in your priority?

[trivialfis]

There's open issue related to external memory #4037. I'm still struggling to understand it...

[CodingCat]

@hcho3 , yes, our data size makes external memory a must-have......but I am a bit confused which part makes hist does not work with external memory....

@trivialfis what I observed is that when I setup external memory with hist tree_method, the metrics (train/test) doesn't change after 5 - 6 iterations, though the training still moves forward.....

[CodingCat]

I have started looking into why external memory version of hist is messed up with accuracy

[hcho3]

@CodingCat Is this issue blocking?

[CodingCat]

no, it just means you need more memory to use distributed hist,

[rongou]

@CodingCat Is this still an issue? As part of my work on gpu external memory support, I tested hist on the Higgs dataset and got identical AUC metrics. The external memory version is actually slightly faster.

Mode	Time(seconds)	AUC
hist	1309.64	0.8393
hist external memory	1228.53	0.8393

[CodingCat]

I didn’t test it for a while...last time what I saw is the metrics stays the same since the first few iteration Maybe it has been fixed in some changes regarding dmatrix, etc.

…

On Fri, Feb 21, 2020 at 10:16 AM Rong Ou ***@***.***> wrote: @CodingCat <https://github.com/CodingCat> Is this still an issue? As part of my work on gpu external memory support, I tested hist on the Higgs dataset and got identical AUC metrics. The external memory version is actually slightly faster. Mode Time(seconds) AUC hist 1309.64 0.8393 hist external memory 1228.53 0.8393 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4093?email_source=notifications&email_token=AAFFQ6ANJ3KHR5CGRZP4OHTREALBLA5CNFSM4GTOJ7ZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMTTFHQ#issuecomment-589771422>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFFQ6AK3DU4SQOYZTUSA5TREALBLANCNFSM4GTOJ7ZA> .

[PeterPanonGit]

@rongou In my experiments external memory seems to run much slower than the in-memory counterpart regardless of the tree_method I use. However in your benchmark it seems that hist and hist external memory speed is almost the same. Could you post a script that we can use to re-produce your benchmark?

[rongou]

@PeterPanonGit I don't have the script anymore, but I wasn't doing anything special, just the standard approach for enabling external memory. Since in external memory mode a lot of data is written out to disk and read back, the speed of your disk might be a factor. Are you on a local desktop with a late model SSD? In the cloud environment I/O might be more constrained.

[PeterPanonGit]

@rongou I used an SSD but external memory version is still ~3x slower than in memory version. Maybe my SSD is not good. Is there a way to check how much time is spent on disk I/O and how much time is spend on in-memory computation?

[rongou]

There should be some logging output when the files are written out, before tree construction starts.

[trivialfis]

Closing in favor of #7354 .