{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":80087836,"defaultBranch":"main","name":"badger","ownerLogin":"dgraph-io","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2017-01-26T05:09:49.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/13958706?v=4","public":true,"private":false,"isOrgOwned":true},"refInfo":{"name":"","listCacheKey":"v0:1714761319.0","currentOid":""},"activityList":{"items":[{"before":"958ef077401204c8921e584e485a853b73337313","after":null,"ref":"refs/heads/rft-ci-cd-optimization","pushedAt":"2024-05-03T18:35:19.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"ryanfoxtyler","name":"Ryan Fox-Tyler","path":"/ryanfoxtyler","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/60440289?s=80&v=4"}},{"before":"fece30f57aa77ff4ed515cf20b0f0e776f305e3f","after":"a09e9837cd271284f43fba45b94c2847a4a58901","ref":"refs/heads/main","pushedAt":"2024-05-03T18:35:18.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"ryanfoxtyler","name":"Ryan Fox-Tyler","path":"/ryanfoxtyler","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/60440289?s=80&v=4"},"commit":{"message":"ci/cd optimization (#2051)\n\n• updating Actions versions\r\n• adding WarpBuild runners for execution without concurrency limits\r\n• tuning triggers/schedules","shortMessageHtmlLink":"ci/cd optimization (#2051)"}},{"before":"483799562de49dcc0976b200fd6e126e43092518","after":null,"ref":"refs/heads/rft-ci-update","pushedAt":"2024-04-08T13:32:26.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"ryanfoxtyler","name":"Ryan Fox-Tyler","path":"/ryanfoxtyler","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/60440289?s=80&v=4"}},{"before":"6acc8e801739f6702b8d95f462b8d450b9a0455b","after":"fece30f57aa77ff4ed515cf20b0f0e776f305e3f","ref":"refs/heads/main","pushedAt":"2024-04-08T13:32:25.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"ryanfoxtyler","name":"Ryan Fox-Tyler","path":"/ryanfoxtyler","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/60440289?s=80&v=4"},"commit":{"message":"fix(CI): Update to pull_request trigger (#2056)\n\n## Problem\r\nUsing `pull_request_target` can expose secrets based on a quirk in how\r\nGitHub applies permissions to forks. See\r\nhttps://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#pull_request_target\r\n\r\n## Solution\r\nChange trigger from `pull_request_target` to `pull_request`","shortMessageHtmlLink":"fix(CI): Update to pull_request trigger (#2056)"}},{"before":null,"after":"483799562de49dcc0976b200fd6e126e43092518","ref":"refs/heads/rft-ci-update","pushedAt":"2024-04-08T13:02:58.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"ryanfoxtyler","name":"Ryan Fox-Tyler","path":"/ryanfoxtyler","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/60440289?s=80&v=4"},"commit":{"message":"Update ci-badger-tests-coverage.yml","shortMessageHtmlLink":"Update ci-badger-tests-coverage.yml"}},{"before":"5b6c513f9c63a1a34d9c6e92e26ad9c386aa4e96","after":null,"ref":"refs/heads/rft-ci-updates","pushedAt":"2024-03-23T15:55:12.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"ryanfoxtyler","name":"Ryan Fox-Tyler","path":"/ryanfoxtyler","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/60440289?s=80&v=4"}},{"before":null,"after":"5b6c513f9c63a1a34d9c6e92e26ad9c386aa4e96","ref":"refs/heads/rft-ci-updates","pushedAt":"2024-03-23T15:54:27.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"ryanfoxtyler","name":"Ryan Fox-Tyler","path":"/ryanfoxtyler","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/60440289?s=80&v=4"},"commit":{"message":"Update ci-dgraph-tests.yml","shortMessageHtmlLink":"Update ci-dgraph-tests.yml"}},{"before":"d1e5c411ad88e7f43137120d2398963d69cc2725","after":"958ef077401204c8921e584e485a853b73337313","ref":"refs/heads/rft-ci-cd-optimization","pushedAt":"2024-03-14T23:32:42.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ryanfoxtyler","name":"Ryan Fox-Tyler","path":"/ryanfoxtyler","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/60440289?s=80&v=4"},"commit":{"message":"Update ci-golang-lint.yml","shortMessageHtmlLink":"Update ci-golang-lint.yml"}},{"before":"9c21b11b9fe79a6f49ef9564eb99f9e053a2953a","after":"d1e5c411ad88e7f43137120d2398963d69cc2725","ref":"refs/heads/rft-ci-cd-optimization","pushedAt":"2024-03-14T23:31:58.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ryanfoxtyler","name":"Ryan Fox-Tyler","path":"/ryanfoxtyler","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/60440289?s=80&v=4"},"commit":{"message":"Update ci-dgraph-tests.yml","shortMessageHtmlLink":"Update ci-dgraph-tests.yml"}},{"before":"e11fa55d3a998234ff74123b004abb1bbee63863","after":"9c21b11b9fe79a6f49ef9564eb99f9e053a2953a","ref":"refs/heads/rft-ci-cd-optimization","pushedAt":"2024-03-14T23:31:31.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ryanfoxtyler","name":"Ryan Fox-Tyler","path":"/ryanfoxtyler","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/60440289?s=80&v=4"},"commit":{"message":"Update ci-badger-tests.yml","shortMessageHtmlLink":"Update ci-badger-tests.yml"}},{"before":"109022a6cdd2c91cf516190ca57034e46a79dabc","after":"e11fa55d3a998234ff74123b004abb1bbee63863","ref":"refs/heads/rft-ci-cd-optimization","pushedAt":"2024-03-14T23:31:12.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ryanfoxtyler","name":"Ryan Fox-Tyler","path":"/ryanfoxtyler","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/60440289?s=80&v=4"},"commit":{"message":"Update ci-badger-tests-coverage.yml","shortMessageHtmlLink":"Update ci-badger-tests-coverage.yml"}},{"before":"fa8e18657846a65aa5379603b99a1db727ca51f5","after":"109022a6cdd2c91cf516190ca57034e46a79dabc","ref":"refs/heads/rft-ci-cd-optimization","pushedAt":"2024-03-14T23:30:54.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ryanfoxtyler","name":"Ryan Fox-Tyler","path":"/ryanfoxtyler","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/60440289?s=80&v=4"},"commit":{"message":"Update ci-badger-bank-tests.yml","shortMessageHtmlLink":"Update ci-badger-bank-tests.yml"}},{"before":"e355801a644c885a02cddcb136967b83a383f356","after":"fa8e18657846a65aa5379603b99a1db727ca51f5","ref":"refs/heads/rft-ci-cd-optimization","pushedAt":"2024-03-14T23:30:31.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ryanfoxtyler","name":"Ryan Fox-Tyler","path":"/ryanfoxtyler","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/60440289?s=80&v=4"},"commit":{"message":"Update ci-badger-bank-tests-nightly.yml","shortMessageHtmlLink":"Update ci-badger-bank-tests-nightly.yml"}},{"before":"14c3d21899460a434dbec850c3415d481083633e","after":"e355801a644c885a02cddcb136967b83a383f356","ref":"refs/heads/rft-ci-cd-optimization","pushedAt":"2024-03-14T23:29:59.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ryanfoxtyler","name":"Ryan Fox-Tyler","path":"/ryanfoxtyler","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/60440289?s=80&v=4"},"commit":{"message":"Update ci-aqua-security-trivy-tests.yml","shortMessageHtmlLink":"Update ci-aqua-security-trivy-tests.yml"}},{"before":null,"after":"14c3d21899460a434dbec850c3415d481083633e","ref":"refs/heads/rft-ci-cd-optimization","pushedAt":"2024-03-14T23:29:20.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"ryanfoxtyler","name":"Ryan Fox-Tyler","path":"/ryanfoxtyler","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/60440289?s=80&v=4"},"commit":{"message":"Update cd-badger.yml","shortMessageHtmlLink":"Update cd-badger.yml"}},{"before":"384ec9770265cd77a14cfd72cd694636f41591d9","after":null,"ref":"refs/heads/rft-codeowners","pushedAt":"2024-01-19T13:55:27.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"ryanfoxtyler","name":"Ryan Fox-Tyler","path":"/ryanfoxtyler","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/60440289?s=80&v=4"}},{"before":"1c417aa3799cb5010cfc4d520647c769b4b46ba6","after":"6acc8e801739f6702b8d95f462b8d450b9a0455b","ref":"refs/heads/main","pushedAt":"2024-01-19T13:55:26.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"ryanfoxtyler","name":"Ryan Fox-Tyler","path":"/ryanfoxtyler","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/60440289?s=80&v=4"},"commit":{"message":"Update CODEOWNERS (#2043)\n\nUpdating codeowners to map to GitHub Team for easier management","shortMessageHtmlLink":"Update CODEOWNERS (#2043)"}},{"before":"8771e477fe89ff088c13700586c28b8c11c04ad9","after":null,"ref":"refs/heads/joshua/testing","pushedAt":"2024-01-16T14:06:45.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"ryanfoxtyler","name":"Ryan Fox-Tyler","path":"/ryanfoxtyler","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/60440289?s=80&v=4"}},{"before":null,"after":"384ec9770265cd77a14cfd72cd694636f41591d9","ref":"refs/heads/rft-codeowners","pushedAt":"2024-01-16T13:53:04.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"ryanfoxtyler","name":"Ryan Fox-Tyler","path":"/ryanfoxtyler","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/60440289?s=80&v=4"},"commit":{"message":"Update CODEOWNERS","shortMessageHtmlLink":"Update CODEOWNERS"}},{"before":"7b5baa11879cdf9d8608fc77ae3033c30a68b972","after":"1c417aa3799cb5010cfc4d520647c769b4b46ba6","ref":"refs/heads/main","pushedAt":"2024-01-06T09:44:58.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"mangalaman93","name":"Aman Mangal","path":"/mangalaman93","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/2542340?s=80&v=4"},"commit":{"message":"chore: MaxTableSize has been renamed to BaseTableSize (#2038)\n\nIt seems there were some mentions of MaxTableSize around, which does not\r\nexist anymore.","shortMessageHtmlLink":"chore: MaxTableSize has been renamed to BaseTableSize (#2038)"}},{"before":"589c786e1cc670534795757c7b24c6ead9d0b1c6","after":"7b5baa11879cdf9d8608fc77ae3033c30a68b972","ref":"refs/heads/main","pushedAt":"2023-12-18T06:51:12.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"mangalaman93","name":"Aman Mangal","path":"/mangalaman93","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/2542340?s=80&v=4"},"commit":{"message":"docs: update README with project LLS using badger (#2032)","shortMessageHtmlLink":"docs: update README with project LLS using badger (#2032)"}},{"before":"09b73f7a3db7e5642ba9eb00d196a3bb212296be","after":"589c786e1cc670534795757c7b24c6ead9d0b1c6","ref":"refs/heads/main","pushedAt":"2023-12-18T06:50:33.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"mangalaman93","name":"Aman Mangal","path":"/mangalaman93","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/2542340?s=80&v=4"},"commit":{"message":"fix(levelHandler): use lock for levelHandler sort tables instead of rlock (#2034)\n\nCo-authored-by: guangzhixu ","shortMessageHtmlLink":"fix(levelHandler): use lock for levelHandler sort tables instead of r…"}},{"before":"fb1b009595813ae6dabd11628281568c8537f7fd","after":"09b73f7a3db7e5642ba9eb00d196a3bb212296be","ref":"refs/heads/main","pushedAt":"2023-12-18T06:49:27.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"mangalaman93","name":"Aman Mangal","path":"/mangalaman93","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/2542340?s=80&v=4"},"commit":{"message":"fix(txn): discard empty transactions on CommitWith (#2031)\n\n## Problems\r\n* Transactions with empty `pendingWrites` were never discarded(and\r\nmarked as done) for `CommitWith`\r\n* This is similar to https://github.com/dgraph-io/badger/pull/2018,\r\nwhich solved the same problem for `Commit`, but not for `CommitWith`\r\n* The `CommitWith` is used by `WriteBatch`, so flushing an empty batch\r\nnever discarded the inner shadowed transaction, causing a multitude of\r\nissues.\r\n\r\n## Solution\r\nMake sure `Discard` is called for `CommitWith` when `pendingWrites` are\r\nempty. The existing unit test is updated to assert that.","shortMessageHtmlLink":"fix(txn): discard empty transactions on CommitWith (#2031)"}},{"before":null,"after":"103900b0e2a4dff0b922cd0cfaf53f2152012186","ref":"refs/heads/dependabot/go_modules/google.golang.org/grpc-1.56.3","pushedAt":"2023-10-25T21:20:58.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"dependabot[bot]","name":null,"path":"/apps/dependabot","primaryAvatarUrl":"https://avatars.githubusercontent.com/in/29110?s=80&v=4"},"commit":{"message":"chore(deps): bump google.golang.org/grpc from 1.20.1 to 1.56.3\n\nBumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.20.1 to 1.56.3.\n- [Release notes](https://github.com/grpc/grpc-go/releases)\n- [Commits](https://github.com/grpc/grpc-go/compare/v1.20.1...v1.56.3)\n\n---\nupdated-dependencies:\n- dependency-name: google.golang.org/grpc\n dependency-type: indirect\n...\n\nSigned-off-by: dependabot[bot] ","shortMessageHtmlLink":"chore(deps): bump google.golang.org/grpc from 1.20.1 to 1.56.3"}},{"before":"b84bc01e234c7b9fd8ea1ddcef633786b2239489","after":"fb1b009595813ae6dabd11628281568c8537f7fd","ref":"refs/heads/main","pushedAt":"2023-10-13T07:44:11.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"mangalaman93","name":"Aman Mangal","path":"/mangalaman93","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/2542340?s=80&v=4"},"commit":{"message":"optimize allocations by using pre allocated buffer for priorities (#2006)\n\n## Problem\r\nBadger allocates a lot of objects over time. I created a simple\r\nreproducer and measured allocations after 10 minutes of running it.\r\n\r\n```\r\n(pprof) top\r\nShowing nodes accounting for 267006, 99.54% of 268253 total\r\nDropped 71 nodes (cum <= 1341)\r\nShowing top 10 nodes out of 14\r\n flat flat% sum% cum cum%\r\n 155255 57.88% 57.88% 155255 57.88% github.com/dgraph-io/badger/v4.(*levelsController).pickCompactLevels.func1 (inline)\r\n 65539 24.43% 82.31% 65539 24.43% github.com/dgraph-io/badger/v4.(*levelsController).levelTargets\r\n 43691 16.29% 98.60% 264485 98.60% github.com/dgraph-io/badger/v4.(*levelsController).pickCompactLevels\r\n 2521 0.94% 99.54% 2521 0.94% os.(*File).Stat\r\n 0 0% 99.54% 264485 98.60% github.com/dgraph-io/badger/v4.(*levelsController).runCompactor\r\n 0 0% 99.54% 264485 98.60% github.com/dgraph-io/badger/v4.(*levelsController).runCompactor.func3\r\n 0 0% 99.54% 2521 0.94% github.com/dgraph-io/badger/v4.(*logFile).open\r\n 0 0% 99.54% 2521 0.94% github.com/dgraph-io/badger/v4.(*valueLog).open\r\n 0 0% 99.54% 2528 0.94% github.com/dgraph-io/badger/v4.Open\r\n 0 0% 99.54% 2521 0.94% github.com/dgraph-io/ristretto/z.OpenMmapFile\r\n(pprof) sample_index=alloc_space\r\n(pprof) top\r\nShowing nodes accounting for 238.72MB, 98.59% of 242.14MB total\r\nDropped 51 nodes (cum <= 1.21MB)\r\nShowing top 10 nodes out of 34\r\n flat flat% sum% cum cum%\r\n 166.41MB 68.72% 68.72% 166.41MB 68.72% github.com/dgraph-io/badger/v4/skl.newArena (inline)\r\n 59.04MB 24.38% 93.10% 59.04MB 24.38% github.com/dgraph-io/badger/v4.(*levelsController).pickCompactLevels.func1 (inline)\r\n 4MB 1.65% 94.75% 4MB 1.65% github.com/dgraph-io/ristretto/z.Calloc (inline)\r\n 4MB 1.65% 96.41% 4MB 1.65% github.com/dgraph-io/badger/v4.(*levelsController).levelTargets\r\n 3.01MB 1.24% 97.65% 3.01MB 1.24% github.com/google/flatbuffers/go.NewBuilder (inline)\r\n 1.27MB 0.52% 98.17% 1.27MB 0.52% github.com/dgraph-io/ristretto.newCmRow\r\n 1MB 0.41% 98.59% 64.04MB 26.45% github.com/dgraph-io/badger/v4.(*levelsController).pickCompactLevels\r\n 0 0% 98.59% 7.01MB 2.89% github.com/dgraph-io/badger/v4.(*DB).flushMemtable\r\n 0 0% 98.59% 7.01MB 2.89% github.com/dgraph-io/badger/v4.(*DB).handleMemTableFlush\r\n 0 0% 98.59% 83.20MB 34.36% github.com/dgraph-io/badger/v4.(*DB).newMemTable\r\n```\r\n\r\nWe see that pickCompactLevels makes a pretty high number of allocations\r\ndue to appending to slice over and over again:\r\n```\r\n(pprof) list pickCompactLevels\r\nTotal: 268253\r\nROUTINE ======================== github.com/dgraph-io/badger/v4.(*levelsController).pickCompactLevels in /Users/deff/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.2.0/levels.go\r\n 43691 264485 (flat, cum) 98.60% of Total\r\n . . 539:func (s *levelsController) pickCompactLevels() (prios []compactionPriority) {\r\n . 65539 540:\tt := s.levelTargets()\r\n . . 541:\taddPriority := func(level int, score float64) {\r\n . . 542:\t\tpri := compactionPriority{\r\n . . 543:\t\t\tlevel: level,\r\n . . 544:\t\t\tscore: score,\r\n . . 545:\t\t\tadjusted: score,\r\n . . 546:\t\t\tt: t,\r\n . . 547:\t\t}\r\n . . 548:\t\tprios = append(prios, pri)\r\n . . 549:\t}\r\n . . 550:\r\n . . 551:\t// Add L0 priority based on the number of tables.\r\n . 42134 552:\taddPriority(0, float64(s.levels[0].numTables())/float64(s.kv.opt.NumLevelZeroTables))\r\n . . 553:\r\n . . 554:\t// All other levels use size to calculate priority.\r\n . . 555:\tfor i := 1; i < len(s.levels); i++ {\r\n . . 556:\t\t// Don't consider those tables that are already being compacted right now.\r\n . . 557:\t\tdelSize := s.cstatus.delSize(i)\r\n . . 558:\r\n . . 559:\t\tl := s.levels[i]\r\n . . 560:\t\tsz := l.getTotalSize() - delSize\r\n . 113121 561:\t\taddPriority(i, float64(sz)/float64(t.targetSz[i]))\r\n . . 562:\t}\r\n . . 563:\ty.AssertTrue(len(prios) == len(s.levels))\r\n . . 564:\r\n . . 565:\t// The following code is borrowed from PebbleDB and results in healthier LSM tree structure.\r\n . . 566:\t// If Li-1 has score > 1.0, then we'll divide Li-1 score by Li. If Li score is >= 1.0, then Li-1\r\n . . 567:\t// score is reduced, which means we'll prioritize the compaction of lower levels (L5, L4 and so\r\n . . 568:\t// on) over the higher levels (L0, L1 and so on). On the other hand, if Li score is < 1.0, then\r\n . . 569:\t// we'll increase the priority of Li-1.\r\n . . 570:\t// Overall what this means is, if the bottom level is already overflowing, then de-prioritize\r\n . . 571:\t// compaction of the above level. If the bottom level is not full, then increase the priority of\r\n . . 572:\t// above level.\r\n . . 573:\tvar prevLevel int\r\n . . 574:\tfor level := t.baseLevel; level < len(s.levels); level++ {\r\n . . 575:\t\tif prios[prevLevel].adjusted >= 1 {\r\n . . 576:\t\t\t// Avoid absurdly large scores by placing a floor on the score that we'll\r\n . . 577:\t\t\t// adjust a level by. The value of 0.01 was chosen somewhat arbitrarily\r\n . . 578:\t\t\tconst minScore = 0.01\r\n . . 579:\t\t\tif prios[level].score >= minScore {\r\n . . 580:\t\t\t\tprios[prevLevel].adjusted /= prios[level].adjusted\r\n . . 581:\t\t\t} else {\r\n . . 582:\t\t\t\tprios[prevLevel].adjusted /= minScore\r\n . . 583:\t\t\t}\r\n . . 584:\t\t}\r\n . . 585:\t\tprevLevel = level\r\n . . 586:\t}\r\n . . 587:\r\n . . 588:\t// Pick all the levels whose original score is >= 1.0, irrespective of their adjusted score.\r\n . . 589:\t// We'll still sort them by their adjusted score below. Having both these scores allows us to\r\n . . 590:\t// make better decisions about compacting L0. If we see a score >= 1.0, we can do L0->L0\r\n . . 591:\t// compactions. If the adjusted score >= 1.0, then we can do L0->Lbase compactions.\r\n . . 592:\tout := prios[:0]\r\n . . 593:\tfor _, p := range prios[:len(prios)-1] {\r\n . . 594:\t\tif p.score >= 1.0 {\r\n . . 595:\t\t\tout = append(out, p)\r\n . . 596:\t\t}\r\n . . 597:\t}\r\n . . 598:\tprios = out\r\n . . 599:\r\n . . 600:\t// Sort by the adjusted score.\r\n 43691 43691 601:\tsort.Slice(prios, func(i, j int) bool {\r\n . . 602:\t\treturn prios[i].adjusted > prios[j].adjusted\r\n . . 603:\t})\r\n . . 604:\treturn prios\r\n . . 605:}\r\n . . 606:\r\nROUTINE ======================== github.com/dgraph-io/badger/v4.(*levelsController).pickCompactLevels.func1 in /Users/deff/go/pkg/mod/github.com/dgraph-io/badger/v4@v4.2.0/levels.go\r\n 155255 155255 (flat, cum) 57.88% of Total\r\n . . 541:\taddPriority := func(level int, score float64) {\r\n . . 542:\t\tpri := compactionPriority{\r\n . . 543:\t\t\tlevel: level,\r\n . . 544:\t\t\tscore: score,\r\n . . 545:\t\t\tadjusted: score,\r\n . . 546:\t\t\tt: t,\r\n . . 547:\t\t}\r\n 155255 155255 548:\t\tprios = append(prios, pri)\r\n . . 549:\t}\r\n . . 550:\r\n . . 551:\t// Add L0 priority based on the number of tables.\r\n . . 552:\taddPriority(0, float64(s.levels[0].numTables())/float64(s.kv.opt.NumLevelZeroTables))\r\n . . 553:\r\n```\r\n\r\n \r\n\r\n## Solution\r\nI suggest two optimizations:\r\n1. Pre-allocate `prios` capacity according to numbers of `s.levels`\r\n2. Reuse `prios` memory in compaction process, thanks to one-threaded\r\nlogic of compactor\r\n\r\nResults after optimization (10 min run of reproducer):\r\n```\r\n(pprof) top\r\nShowing nodes accounting for 165466, 99.84% of 165735 total\r\nDropped 27 nodes (cum <= 828)\r\nShowing top 10 nodes out of 48\r\n flat flat% sum% cum cum%\r\n 40962 24.72% 24.72% 40962 24.72% github.com/dgraph-io/badger/v4.(*levelsController).levelTargets\r\n 32768 19.77% 44.49% 32768 19.77% github.com/dgraph-io/badger/v4/skl.(*Arena).putNode\r\n 32768 19.77% 64.26% 32768 19.77% github.com/dgraph-io/badger/v4/y.KeyWithTs (inline)\r\n 21845 13.18% 77.44% 62807 37.90% github.com/dgraph-io/badger/v4.(*levelsController).pickCompactLevels\r\n 21845 13.18% 90.62% 21845 13.18% github.com/dgraph-io/badger/v4.(*logFile).encodeEntry\r\n 8192 4.94% 95.56% 8192 4.94% github.com/dgraph-io/badger/v4/table.(*Builder).addHelper\r\n 4681 2.82% 98.39% 4681 2.82% regexp/syntax.(*Regexp).Simplify\r\n 2341 1.41% 99.80% 2341 1.41% runtime/pprof.allFrames\r\n 64 0.039% 99.84% 32832 19.81% github.com/dgraph-io/badger/v4.(*Txn).commitAndSend\r\n 0 0% 99.84% 32832 19.81% github.com/dgraph-io/badger/v4.(*DB).Update\r\n(pprof) sample_index=alloc_space\r\n(pprof) top\r\nShowing nodes accounting for 180.47MB, 97.79% of 184.54MB total\r\nDropped 22 nodes (cum <= 0.92MB)\r\nShowing top 10 nodes out of 53\r\n flat flat% sum% cum cum%\r\n 166.41MB 90.17% 90.17% 166.41MB 90.17% github.com/dgraph-io/badger/v4/skl.newArena\r\n 4MB 2.17% 92.34% 4MB 2.17% github.com/dgraph-io/ristretto/z.Calloc\r\n 3.01MB 1.63% 93.97% 3.01MB 1.63% github.com/google/flatbuffers/go.NewBuilder\r\n 2.50MB 1.35% 95.32% 2.50MB 1.35% github.com/dgraph-io/badger/v4.(*levelsController).levelTargets\r\n 1.76MB 0.96% 96.28% 2.97MB 1.61% compress/flate.NewWriter (inline)\r\n 1.16MB 0.63% 96.91% 1.16MB 0.63% github.com/dgraph-io/ristretto/z.(*Bloom).Size\r\n 0.64MB 0.34% 97.25% 1.20MB 0.65% compress/flate.(*compressor).init\r\n 0.50MB 0.27% 97.52% 1MB 0.54% github.com/dgraph-io/badger/v4.(*Txn).commitAndSend\r\n 0.50MB 0.27% 97.79% 3MB 1.63% github.com/dgraph-io/badger/v4.(*levelsController).pickCompactLevels\r\n 0 0% 97.79% 2.97MB 1.61% compress/gzip.(*Writer).Write\r\n```\r\n\r\nAnd inside pickCompactLevels:\r\n```\r\nROUTINE ======================== github.com/dgraph-io/badger/v4.(*levelsController).pickCompactLevels in /Users/deff/dev/work/badger/levels.go\r\n 21845 62807 (flat, cum) 37.90% of Total\r\n . . 544:func (s *levelsController) pickCompactLevels(prios []compactionPriority) []compactionPriority {\r\n . 40962 545:\tt := s.levelTargets()\r\n . . 546:\taddPriority := func(level int, score float64) {\r\n . . 547:\t\tpri := compactionPriority{\r\n . . 548:\t\t\tlevel: level,\r\n . . 549:\t\t\tscore: score,\r\n . . 550:\t\t\tadjusted: score,\r\n . . 551:\t\t\tt: t,\r\n . . 552:\t\t}\r\n . . 553:\t\tprios = append(prios, pri)\r\n . . 554:\t}\r\n . . 555:\r\n . . 556:\tif cap(prios) < len(s.levels) {\r\n . . 557:\t\tprios = make([]compactionPriority, 0, len(s.levels))\r\n . . 558:\t}\r\n . . 559:\tprios = prios[:0]\r\n . . 560:\r\n . . 561:\t// Add L0 priority based on the number of tables.\r\n . . 562:\taddPriority(0, float64(s.levels[0].numTables())/float64(s.kv.opt.NumLevelZeroTables))\r\n . . 563:\r\n . . 564:\t// All other levels use size to calculate priority.\r\n . . 565:\tfor i := 1; i < len(s.levels); i++ {\r\n . . 566:\t\t// Don't consider those tables that are already being compacted right now.\r\n . . 567:\t\tdelSize := s.cstatus.delSize(i)\r\n . . 568:\r\n . . 569:\t\tl := s.levels[i]\r\n . . 570:\t\tsz := l.getTotalSize() - delSize\r\n . . 571:\t\taddPriority(i, float64(sz)/float64(t.targetSz[i]))\r\n . . 572:\t}\r\n . . 573:\ty.AssertTrue(len(prios) == len(s.levels))\r\n . . 574:\r\n . . 575:\t// The following code is borrowed from PebbleDB and results in healthier LSM tree structure.\r\n . . 576:\t// If Li-1 has score > 1.0, then we'll divide Li-1 score by Li. If Li score is >= 1.0, then Li-1\r\n . . 577:\t// score is reduced, which means we'll prioritize the compaction of lower levels (L5, L4 and so\r\n . . 578:\t// on) over the higher levels (L0, L1 and so on). On the other hand, if Li score is < 1.0, then\r\n . . 579:\t// we'll increase the priority of Li-1.\r\n . . 580:\t// Overall what this means is, if the bottom level is already overflowing, then de-prioritize\r\n . . 581:\t// compaction of the above level. If the bottom level is not full, then increase the priority of\r\n . . 582:\t// above level.\r\n . . 583:\tvar prevLevel int\r\n . . 584:\tfor level := t.baseLevel; level < len(s.levels); level++ {\r\n . . 585:\t\tif prios[prevLevel].adjusted >= 1 {\r\n . . 586:\t\t\t// Avoid absurdly large scores by placing a floor on the score that we'll\r\n . . 587:\t\t\t// adjust a level by. The value of 0.01 was chosen somewhat arbitrarily\r\n . . 588:\t\t\tconst minScore = 0.01\r\n . . 589:\t\t\tif prios[level].score >= minScore {\r\n . . 590:\t\t\t\tprios[prevLevel].adjusted /= prios[level].adjusted\r\n . . 591:\t\t\t} else {\r\n . . 592:\t\t\t\tprios[prevLevel].adjusted /= minScore\r\n . . 593:\t\t\t}\r\n . . 594:\t\t}\r\n . . 595:\t\tprevLevel = level\r\n . . 596:\t}\r\n . . 597:\r\n . . 598:\t// Pick all the levels whose original score is >= 1.0, irrespective of their adjusted score.\r\n . . 599:\t// We'll still sort them by their adjusted score below. Having both these scores allows us to\r\n . . 600:\t// make better decisions about compacting L0. If we see a score >= 1.0, we can do L0->L0\r\n . . 601:\t// compactions. If the adjusted score >= 1.0, then we can do L0->Lbase compactions.\r\n . . 602:\tout := prios[:0]\r\n . . 603:\tfor _, p := range prios[:len(prios)-1] {\r\n . . 604:\t\tif p.score >= 1.0 {\r\n . . 605:\t\t\tout = append(out, p)\r\n . . 606:\t\t}\r\n . . 607:\t}\r\n . . 608:\tprios = out\r\n . . 609:\r\n . . 610:\t// Sort by the adjusted score.\r\n 21845 21845 611:\tsort.Slice(prios, func(i, j int) bool {\r\n . . 612:\t\treturn prios[i].adjusted > prios[j].adjusted\r\n . . 613:\t})\r\n . . 614:\treturn prios\r\n . . 615:}\r\n . . 616:\r\n```\r\n \r\n\r\n## Profile from real project\r\nBoth profiles are measured after 30 minutes from application start\r\n\r\n### Before optimization:\r\n```\r\n(pprof) top\r\nShowing nodes accounting for 621.02MB, 85.32% of 727.90MB total\r\nDropped 550 nodes (cum <= 3.64MB)\r\nShowing top 10 nodes out of 146\r\n flat flat% sum% cum cum%\r\n 380.72MB 52.30% 52.30% 380.72MB 52.30% github.com/dgraph-io/badger/v3.(*levelsController).pickCompactLevels.func1\r\n 104.01MB 14.29% 66.59% 104.01MB 14.29% github.com/dgraph-io/badger/v3/skl.newArena\r\n 33.27MB 4.57% 71.16% 33.27MB 4.57% github.com/dgraph-io/ristretto.newCmRow\r\n 27.05MB 3.72% 74.88% 27.05MB 3.72% github.com/dgraph-io/badger/v3/y.SafeCopy\r\n 23.50MB 3.23% 78.11% 23.50MB 3.23% github.com/dgraph-io/badger/v3.(*levelsController).levelTargets\r\n 18.31MB 2.52% 80.62% 18.31MB 2.52% github.com/dgraph-io/ristretto/z.(*Bloom).Size\r\n 18.02MB 2.48% 83.10% 18.02MB 2.48% github.com/dgraph-io/badger/v3/y.(*Slice).Resize\r\n 8MB 1.10% 84.20% 412.23MB 56.63% github.com/dgraph-io/badger/v3.(*levelsController).pickCompactLevels\r\n 4.12MB 0.57% 84.77% 8.13MB 1.12% github.com/blevesearch/vellum.(*FSTIterator).next\r\n 4MB 0.55% 85.32% 4MB 0.55% github.com/blevesearch/vellum.(*decoderV1).stateAt\r\n```\r\n### After optimization:\r\n```\r\nType: alloc_space\r\nTime: Sep 11, 2023 at 5:50pm (+05)\r\nEntering interactive mode (type \"help\" for commands, \"o\" for options)\r\n(pprof) top\r\nShowing nodes accounting for 262.17MB, 66.88% of 391.99MB total\r\nDropped 453 nodes (cum <= 1.96MB)\r\nShowing top 10 nodes out of 290\r\n flat flat% sum% cum cum%\r\n 104.01MB 26.53% 26.53% 104.01MB 26.53% github.com/dgraph-io/badger/v3/skl.newArena\r\n 33.91MB 8.65% 35.18% 33.91MB 8.65% github.com/dgraph-io/ristretto.newCmRow\r\n 28.50MB 7.27% 42.45% 28.50MB 7.27% github.com/dgraph-io/badger/v3.(*levelsController).levelTargets\r\n 26.52MB 6.77% 49.22% 26.52MB 6.77% github.com/dgraph-io/badger/v3/y.(*Slice).Resize\r\n 25.03MB 6.38% 55.61% 25.03MB 6.38% github.com/dgraph-io/badger/v3/y.SafeCopy\r\n 17.16MB 4.38% 59.98% 17.16MB 4.38% github.com/dgraph-io/ristretto/z.(*Bloom).Size\r\n 7.12MB 1.82% 61.80% 9.12MB 2.33% github.com/anyproto/go-chash.(*cHash).addMembers\r\n 6.72MB 1.72% 63.51% 12.22MB 3.12% github.com/blevesearch/vellum.(*FSTIterator).next\r\n 6.71MB 1.71% 65.22% 6.71MB 1.71% bytes.growSlice\r\n 6.50MB 1.66% 66.88% 6.50MB 1.66% github.com/blevesearch/vellum.(*builderNodePool).Get\r\n```\r\n# Reproducer\r\n```\r\npackage main\r\n\r\nimport (\r\n\t\"fmt\"\r\n\t\"os\"\r\n\t\"github.com/dgraph-io/badger/v4\"\r\n\t_ \"net/http/pprof\"\r\n\t\"net/http\"\r\n\t\"log\"\r\n)\r\n\r\nfunc generateItems(db *badger.DB, n int) error {\r\n\treturn db.Update(func(txn *badger.Txn) error {\r\n\t\tfor i := 0; i < n; i++ {\r\n\t\t\terr := txn.Set([]byte(fmt.Sprintf(\"key-%d\", i)), []byte(fmt.Sprintf(\"value-%d\", i)))\r\n\t\t\tif err != nil {\r\n\t\t\t\treturn err\r\n\t\t\t}\r\n\t\t}\r\n\t\treturn nil\r\n\t})\r\n}\r\n\r\nfunc run() error {\r\n\tforever := make(chan struct{})\r\n\r\n\tdb, err := badger.Open(badger.DefaultOptions(\"./tmp\"))\r\n\tif err != nil {\r\n\t\treturn fmt.Errorf(\"open badger: %w\", err)\r\n\t}\r\n\terr = generateItems(db, 1000)\r\n\tif err != nil {\r\n\t\treturn fmt.Errorf(\"generate items: %w\", err)\r\n\t}\r\n\r\n\tgo func() {\r\n\t\tlog.Println(http.ListenAndServe(\"localhost:9000\", nil))\r\n\t}()\r\n\r\n\t<-forever\r\n\treturn nil\r\n}\r\n\r\nfunc main() {\r\n\terr := run()\r\n\tif err != nil {\r\n\t\tfmt.Fprintln(os.Stderr, err)\r\n\t\tos.Exit(1)\r\n\t}\r\n}\r\n```","shortMessageHtmlLink":"optimize allocations by using pre allocated buffer for priorities (#2006"}},{"before":"1493d7f87211fd060b1c82d9a2f6274474db6e96","after":null,"ref":"refs/heads/dependabot/go_modules/golang.org/x/net-0.17.0","pushedAt":"2023-10-13T07:37:57.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"mangalaman93","name":"Aman Mangal","path":"/mangalaman93","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/2542340?s=80&v=4"}},{"before":"1741e474446b4eb053b1a959ea8d31d9d5cf6948","after":"b84bc01e234c7b9fd8ea1ddcef633786b2239489","ref":"refs/heads/main","pushedAt":"2023-10-13T07:37:56.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"mangalaman93","name":"Aman Mangal","path":"/mangalaman93","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/2542340?s=80&v=4"},"commit":{"message":"chore(deps): bump golang.org/x/net from 0.7.0 to 0.17.0 (#2017)","shortMessageHtmlLink":"chore(deps): bump golang.org/x/net from 0.7.0 to 0.17.0 (#2017)"}},{"before":"27724c4dbe8eeef057407cf418ca833cdf7f40d8","after":"1741e474446b4eb053b1a959ea8d31d9d5cf6948","ref":"refs/heads/main","pushedAt":"2023-10-13T07:37:15.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"mangalaman93","name":"Aman Mangal","path":"/mangalaman93","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/2542340?s=80&v=4"},"commit":{"message":"fix resource consumption on empty write transaction (#2018)\n\n## Problem\r\nIf a write transaction is opened without doing any updates, e.g. \r\n```\r\ntxn := db.NewTransaction(true)\r\ntxn.Commit()\r\n```\r\nthe read mark is never marked done, because `txn.Discard` has not been\r\ncalled and neither has the commit logic because of the early return in\r\n`Commit()`:\r\n```\r\nif len(txn.pendingWrites) == 0 {\r\n return nil // Nothing to do.\r\n}\r\n```\r\nThis causes unbounded storage growth until the service is restarted. The\r\nwatermark process in `y/watermark.go` never receives a mark for that\r\ntransaction, and so never updates the `DoneUntil`. This value is used in\r\n`levels.go` for compaction to determine the `discardTS` value, and since\r\nit never updates, compaction can never occur.\r\n\r\n## Solution\r\nThe solution here is to call `txn.Discard` in the case when there are no\r\nwrites on the transaction. This marks the read as done and allows the\r\nwatermark process to properly update.","shortMessageHtmlLink":"fix resource consumption on empty write transaction (#2018)"}},{"before":null,"after":"1493d7f87211fd060b1c82d9a2f6274474db6e96","ref":"refs/heads/dependabot/go_modules/golang.org/x/net-0.17.0","pushedAt":"2023-10-11T23:35:31.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"dependabot[bot]","name":null,"path":"/apps/dependabot","primaryAvatarUrl":"https://avatars.githubusercontent.com/in/29110?s=80&v=4"},"commit":{"message":"chore(deps): bump golang.org/x/net from 0.7.0 to 0.17.0\n\nBumps [golang.org/x/net](https://github.com/golang/net) from 0.7.0 to 0.17.0.\n- [Commits](https://github.com/golang/net/compare/v0.7.0...v0.17.0)\n\n---\nupdated-dependencies:\n- dependency-name: golang.org/x/net\n dependency-type: direct:production\n...\n\nSigned-off-by: dependabot[bot] ","shortMessageHtmlLink":"chore(deps): bump golang.org/x/net from 0.7.0 to 0.17.0"}},{"before":"a5635e18d3232a468928f565c667783aa7a011ba","after":null,"ref":"refs/heads/joshua/ghost-jobs","pushedAt":"2023-09-29T21:35:17.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"joshua-goldstein","name":"Joshua Goldstein","path":"/joshua-goldstein","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/92491720?s=80&v=4"}}],"hasNextPage":true,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"djE6ks8AAAAEQQXHGQA","startCursor":null,"endCursor":null}},"title":"Activity · dgraph-io/badger"}