Fix slowdown for LatestChunkStateWitnesses

[robin-near]

        for item in self.store().iter(DBCol::LatestChunkStateWitnesses) {
            if info.is_within_limits() {
                break;
            }

            let (key_bytes, witness_bytes) = item?;
            store_update.delete(DBCol::LatestChunkStateWitnesses, &key_bytes);
            info.count -= 1;
            info.total_size -= witness_bytes.len() as u64;
        }

For some reason, this loop is very slow relative to other code in neard. It hogs 100% CPU and slows everything else to a crawl.

Not sure why, but we should investigate. I think at least, we can avoid iterating at all if the limit is not exceeded in the first place. Ideally, we should not be iterating, but using a deterministic point-wise algorithm such as the one used in DelayedReceipts.

[jancionear]

Ah crap, I'll take a look at it. Maybe store().iter() loads a whole page of results? 0_o

I think at least, we can avoid iterating at all if the limit is not exceeded in the first place.

The set of saved witnesses is always almost full, so most of the time the info will be exceeded, I think it won't help much.

Ideally, we should not be iterating, but using a deterministic point-wise algorithm such as the one used in DelayedReceipts.

👀

[jancionear]

Ideally, we should not be iterating, but using a deterministic point-wise algorithm such as the one used in DelayedReceipts.

The problem is that we'd like to be able to query for a witness with specific height and shard_id, this is difficult to do efficiently with an algorithm like the one for DelayedReceipts. I guess we could scan everything and print out the desired witnesses, but scanning 4GB of data could take a moment 0_o

[jancionear]

Maybe store().iter() loads a whole page of results? 0_o

I tried to set iteration readahead to 4KB, but iter() still takes like 100ms 0_o. Weird stuff...

read_options.set_readahead_size(4096);