add BlockedBitpacker #1030
add BlockedBitpacker #1030
Conversation
fix memory_usage calculation
calc mem_usage of more structs in index creation add some comments
bitpacker/src/blocked_bitpacker.rs
Outdated
| } | ||
|
|
||
| pub fn flush(&mut self) { | ||
| if self.cache.is_empty() { |
There was a problem hiding this comment.
nitpick...
nowadays I like to avoid the unwrap using the following pattern instead of line 90 and line 86-88
let base_value = if let Some(min_value) = self.cache.iter().min() {
min_value
} else {
return;
};There was a problem hiding this comment.
I can do that, but there is still a unwrap() in line 96
bitpacker/src/blocked_bitpacker.rs
Outdated
| .cache | ||
| .iter() | ||
| .map(|val| compute_num_bits(*val - base_value)) | ||
| .max() |
There was a problem hiding this comment.
there is minmax in common. You can move it into bitpacker maybe?
There was a problem hiding this comment.
hmm I think the current version is easier to understand, since we don't use min here
let num_bits_block = self
.buffer
.iter()
.map(|val| compute_num_bits(*val - base_value))
.max()
.unwrap();
let (_, num_bits_block) = minmax(
self.buffer
.iter()
.map(|val| compute_num_bits(*val - base_value)),
).unwrap();
There was a problem hiding this comment.
i meant combine with the idea above of course
let (base_val, max_val) = if let Some((min_val, val)) = min_max() {
(min_val, max_val)
} else {
return
};
let num_bits = compute_num_bits(max_val - base_val);
There was a problem hiding this comment.
ah sure that makes sense
bitpacker/src/blocked_bitpacker.rs
Outdated
| num_bits_block, | ||
| &mut self.compressed_blocks, | ||
| ) | ||
| .expect("cannot write bitpacking to output"); // write to im can't fail |
move minmax to bitpacker use minmax in blocked bitpacker
| pub fn flush(&mut self) { | ||
| if let Some((min_value, max_value)) = minmax(self.buffer.iter()) { | ||
| let mut bit_packer = BitPacker::new(); | ||
| let num_bits_block = compute_num_bits(*max_value - min_value); |
There was a problem hiding this comment.
I'm surprise this * compiles... what is it about?
There was a problem hiding this comment.
It's a residue from before. It seems usize implements sub on &usize
https://doc.rust-lang.org/std/primitive.usize.html#impl-Sub%3C%26%27_%20usize%3E
move tantivy bitpacker to subcrate
remove byteorder dependency
use BlockedBitpacker in IntFastFieldWriter