Too much memory usage

[terrancewong]

How to represent

fallocate -x -l 1T test1t
/usr/bin/time -f "%M kBpeak, %Us user, %I FSI, %O FSO, %P CPU, %es real" b3sum test1t
9387bd9b2ff4c3d9baa8c65d491f51789b6ed2a000aa45b679f80d21c3cc5013  test1t      
504482044 kBpeak, 1078.56s user, 88 FSI, 0 FSO, 3439% CPU, 703.66s real 

500GB memory used. rendering very high load, system less usable.

[oconnor663]

My guess is that this is just what happens when we mmap the sparse file and then read the mmap?

[terrancewong]

sounds like that, file t is around 61GB, mem peak around 60GB.

 % /usr/bin/time -f "%M kBpeak, %Us user, %I FSI, %O FSO, %P CPU, %es real" b3sum t                    
ab5d590789635ed6444bbf901f81fa8611ecbd8a6581b156ccb3752cc46dbe49  t
59595172 kBpeak, 36.24s user, 0 FSI, 0 FSO, 2735% CPU, 1.44s real

does not happen when hashing through pipe.

[oconnor663]

Yes that's expected. b3sum doesn't try to mmap standard input, so hashing with | or < works around this. Passing --no-mmap should have the same effect. It's also possible that --num-threads=1 would cut down on the allocations, but that's mostly up to your OS and not something we control.

[terrancewong]

but --no-mmap makes it significantly slower, only 100% cpu utilized.

[terrancewong]

and in theory it could get away with only storing log(N) scale Merkel tree.

[oconnor663]

Yep :(