[BUG]: "Arena too small" after enlarging memtable size, then re-opening with prior size

[BatmanAoD]

What version of Badger are you using?

v3.2103.2
v3.2103.5
v4.2.0

What version of Go are you using?

go version go1.20.1 linux/amd64

Have you tried reproducing the issue with the latest release?

Yes

What is the hardware spec (RAM, CPU, OS)?

We run the Go service in a Docker container.

Container OS: CentOS Linux release 7.9.2009 (Core)
Container CPU: 100 Mhz
Container memory: 16 Gb

What steps will reproduce the bug?

• create a database with the default memtable size.
• close the process; re-open the on-disk database with a larger memtable size (we used 256 Mb, i.e. WithMemTableSize(256 * 1024 * 1024))
• close the process; re-open with the default memtable size (i.e. no WithMemTableSize option)

Expected behavior and actual result.

Expected behavior: database is usable

Actual behavior: runtime.Goexit is called, with this stack-trace:

	/usr/local/go/src/runtime/asm_amd64.s:1598
runtime.goexit
	/usr/local/go/src/runtime/proc.go:250
runtime.main
	/go/src/gitlab.com/rigetti/qcs/services/controller/main.go:10
main.main
	/root/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:902
github.com/spf13/cobra.(*Command).Execute
	/root/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:974
github.com/spf13/cobra.(*Command).ExecuteC
	/root/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:856
github.com/spf13/cobra.(*Command).execute
	/go/src/gitlab.com/rigetti/qcs/services/controller/cmd/serve.go:45
gitlab.com/rigetti/qcs/services/controller/cmd.NewRunServerCmd.func1
	/go/src/gitlab.com/rigetti/qcs/services/controller/cmd/serve.go:105
gitlab.com/rigetti/qcs/services/controller/cmd.runServer
	/go/src/gitlab.com/rigetti/qcs/services/controller/storage/backend.go:75
gitlab.com/rigetti/qcs/services/controller/storage.New
	/go/src/gitlab.com/rigetti/qcs/services/controller/storage/badger.go:35
gitlab.com/rigetti/qcs/services/controller/storage.NewBadgerBackend
	/root/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.2/db.go:333
github.com/dgraph-io/badger/v3.Open
	/root/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.2/memtable.go:88
github.com/dgraph-io/badger/v3.(*DB).openMemTables
	/root/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.2/memtable.go:146
github.com/dgraph-io/badger/v3.(*DB).openMemTable
	/root/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.2/memtable.go:219
github.com/dgraph-io/badger/v3.(*memTable).UpdateSkipList
	/root/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.2/memtable.go:521
github.com/dgraph-io/badger/v3.(*logFile).iterate
	/root/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.2/memtable.go:258
github.com/dgraph-io/badger/v3.(*memTable).replayFunction.func1
	/root/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.2/skl/skl.go:306
github.com/dgraph-io/badger/v3/skl.(*Skiplist).Put
	/root/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.2/skl/skl.go:114
github.com/dgraph-io/badger/v3/skl.newNode
	/root/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.2/skl/arena.go:83
github.com/dgraph-io/badger/v3/skl.(*Arena).putVal
	/root/go/pkg/mod/github.com/dgraph-io/badger/v3@v3.2103.2/y/error.go:62
github.com/dgraph-io/badger/v3/y.AssertTruef
Arena too small, toWrite:6797 newTotal:87242763 limit:87241465

Additional information

This is possibly related to #64.

If changing the memtable size for an existing on-disk store is not permitted, this should be documented, and there should be a way to gracefully handle this error. Ideally, if WithMemTableSize is not provided, the Open function should detect the last-used size of the on-disk store.

[BatmanAoD]

Hi, is anyone able to look at this issue?

[dannybabbev]

I have the same issue, reproducible in the same way