Skip to content

Commit

Permalink
fix(builder): put the upper limit on reallocation (#1748)
Browse files Browse the repository at this point in the history
z.Allocator imposes an upper bound of 1GB on the size of the allocator. In builder, we double the allocation while reallocating to amortize the cost over the future allocations.

Consider a scenario where the size of KV is 700MB. The size of the allocator would be 700MB. Now we want to finish the block, so we will need some bytes for the metadata. That can easily fit into the remaining 324MB. But we were earlier doing over-allocation that causes panic.

(cherry picked from commit f762055)
  • Loading branch information
NamanJain8 authored and mangalaman93 committed Feb 14, 2023
1 parent 30b976e commit 212fa1e
Showing 1 changed file with 5 additions and 1 deletion.
6 changes: 5 additions & 1 deletion table/builder.go
Expand Up @@ -100,8 +100,12 @@ type Builder struct {
func (b *Builder) allocate(need int) []byte {
bb := b.curBlock
if len(bb.data[bb.end:]) < need {
// We need to reallocate.
// We need to reallocate. 1GB is the max size that the allocator can allocate.
// While reallocating, if doubling exceeds that limit, then put the upper bound on it.
sz := 2 * len(bb.data)
if sz > (1 << 30) {
sz = 1 << 30
}
if bb.end+need > sz {
sz = bb.end + need
}
Expand Down

0 comments on commit 212fa1e

Please sign in to comment.