Prometheus database corruption v.2.9.2 to 2.39.1 - Version Compatability

[cwelford]

We recently done an upgrade and found to have corruption in the blocks

prometheus[905]: ts=2022-11-01T15:20:48.467Z caller=main.go:1119 level=error err="opening storage failed: reloadBlocks: 6 errors: corrupted block 01GGQXGB6514QBTRHS8MQAYMZ3: mmap files: mmap, size 211755776: cannot allocate memory; corrupted block 01GGE8GMG1HH1DAHECPFMWP96R: mmap files: mmap, size 221582703: cannot allocate memory; corrupted block 01GGG6A6NHPWYTEBYMBQ39JCHN: mmap files: mmap, size 211094197: cannot allocate memory; corrupted block 01GGJ43PPC7JE4TW2VBS96EBS5: mmap files: mmap, size 196406599: cannot allocate memory; corrupted block 01GGM1X7FMHRJQRYQ4F5ZWSPJW: mmap files: mmap, size 186025539: cannot allocate memory; corrupted block 01GGNZPSPVZD5DAJYN32AG0BMM: mmap files: mmap, size 193833735: cannot allocate memory"

Is there a specific upgrade path to keep the database from corruption? IE 2.9.2 to 2.13 to 2.39?

[bboreham]

The error "cannot allocate memory" is coming from your OS; possibly adding more memory would help.
Although Prometheus says "corrupted block", this is not necessarily true. It really means "block I was unable to open".

The upgrade from 2.9 to 2.39 should work; no incompatible changes have been made.