Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

loop run backup, out of memory #2054

Open
cxcel opened this issue Apr 3, 2024 · 0 comments
Open

loop run backup, out of memory #2054

cxcel opened this issue Apr 3, 2024 · 0 comments
Labels
kind/bug Something is broken.

Comments

@cxcel
Copy link

cxcel commented Apr 3, 2024

What version of Badger are you using?

v4

What version of Go are you using?

1.21.5

Have you tried reproducing the issue with the latest release?

No

What is the hardware spec (RAM, CPU, OS)?

8G, x86_64, linux

What steps will reproduce the bug?

package main

import (
"bufio"
"context"
"fmt"
"log"
"os"
"os/signal"
"runtime/pprof"
"time"

badger "github.com/dgraph-io/badger/v4"

)

func setBadgerOptions(filePath string) badger.Options {
opts := badger.DefaultOptions(filePath)
opts.NumMemtables = 2 << 20
opts.NumLevelZeroTables = 1
opts.NumLevelZeroTablesStall = 2
opts.SyncWrites = false

return opts

}

func backup(db *badger.DB, dbName string) error {
os.MkdirAll("/tmp/backup", 0755)
backupFile := fmt.Sprintf("/tmp/backup/%s_%s.bak", dbName, time.Now().Format("2006-01-02_15:04:05"))
log.Println("backup db", backupFile)

// Create File
f, err := os.Create(backupFile)
if err != nil {
	return err
}

bw := bufio.NewWriterSize(f, 64<<20)
if _, err = db.Backup(bw, 0); err != nil {
	return err
}

// stream := db.NewStream()
// stream.NumGo = 4 // Default is 16
// stream.Backup(bw, 0)

if err = bw.Flush(); err != nil {
	return err
}

if err = f.Sync(); err != nil {
	return err
}

return f.Close()

}

func waitQuit() {
c := make(chan os.Signal, 1)
signal.Notify(c, os.Interrupt)
<-c
log.Println("Exiting")
}

func run(ctx context.Context, db *badger.DB) {
ticker := time.NewTicker(5 * time.Second)
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
if err := backup(db, "BackupSince"); err != nil {
log.Println("backup error", err)
}
}
}
}
func main() {

/* go tool pprof -http=:8000 cpu.out */
cfile, _ := os.Create("cpu.out")
pprof.StartCPUProfile(cfile)
defer pprof.StopCPUProfile()

mfile, _ := os.Create("mem.out")
defer pprof.WriteHeapProfile(mfile)

db, err := badger.Open(setBadgerOptions("/tmp/BackupSince"))
if err != nil {
	log.Fatal(err)
}
defer db.Close()

ctx, cancel := context.WithCancel(context.Background())

go func() {
	run(ctx, db)
}()

waitQuit()
cancel()

log.Println("done")

}

Expected behavior and actual result.

No response

Additional information

No response

@cxcel cxcel added the kind/bug Something is broken. label Apr 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Something is broken.
Development

No branches or pull requests

1 participant