From 4298c583a4658bccedb3ec3e1794a4e92a99c271 Mon Sep 17 00:00:00 2001 From: Naman Jain Date: Thu, 23 Sep 2021 13:44:36 +0530 Subject: [PATCH] fix(builder): put the upper limit on reallocation (#1748) z.Allocator imposes an upper bound of 1GB on the size of the allocator. In builder, we double the allocation while reallocating to amortize the cost over the future allocations. Consider a scenario where the size of KV is 700MB. The size of the allocator would be 700MB. Now we want to finish the block, so we will need some bytes for the metadata. That can easily fit into the remaining 324MB. But we were earlier doing over-allocation that causes panic. (cherry picked from commit f762055820704ed8e2cc58f4dd6f6952ff7d30c6) --- table/builder.go | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/table/builder.go b/table/builder.go index 8322bb86f..5bab48a72 100644 --- a/table/builder.go +++ b/table/builder.go @@ -100,8 +100,12 @@ type Builder struct { func (b *Builder) allocate(need int) []byte { bb := b.curBlock if len(bb.data[bb.end:]) < need { - // We need to reallocate. + // We need to reallocate. 1GB is the max size that the allocator can allocate. + // While reallocating, if doubling exceeds that limit, then put the upper bound on it. sz := 2 * len(bb.data) + if sz > (1 << 30) { + sz = 1 << 30 + } if bb.end+need > sz { sz = bb.end + need }