6.0.100 (Corax) Killed by OOM, "corrupted" 50% of databases #18291
Comments
|
What is the version of RavenDB server? |
|
Sorry was in the title 6.0.100 |
|
It looks very much like there are missing files there. In short, we are running checkpoint at various time (last sync tx in this case) being 191. In this case, the last sync transaction (191) doesn't match the latest journal version Can you check if there is an older journal file? |
|
For me it looks more like we read incorrect / old info from one of (The relevant code: https://github.com/ravendb/ravendb/blob/v6.0/src/Raven.Pal/src/posix/writefileheader.c) |
|
The issue is that we |
|
/Databases/CUSTOMERNAME1/Journals]# ls I got 20 databases,
P.s. I'm almost certain that the faulty indexes weren't running when the OOM happened. They were changed during the uptime of the database. Did I mention that this is on a ARM64 server. |
|
ARM64 shouldn't matter. |
|
This is very suspicious. The way that we work, we flip between writing to |
P.s. the last time I synced was the 29th of November. Back then this database had 50.000 docs, now it has 100.000 docs, so it grew quite a bit. |
|
What do you mean synced, the last time you wrote to the database? If that is the case, Nov make sense then So both of them written at roughly the same time. That indicates (properly) that there was a checkpoint at that time, but can't tell more without knowing exact (minute granularity isn't sufficient). Basically, I'm highly suspicious that the order of writes as reflected on the volume after connecting to the new node isn't respected. |
|
Synced: Files: I don't get the "new node", it is all on a single server with local storage. |
|
Can you send the |
|
|
Okay, this is weird. I looked into this, and it doesn't make sense. The rest of the data there is nonsense as well. What is the value on The issue is that this indicates some problem in the files, which then moves the problem to the invariant violations that we already suspect. For reference, here is the expected layout of the The first 8 bytes should be: |
|
|
P.s. as a little note, xxd (vim) shows 1500 ef0d, not 0015 0def. So I don't know which one does little-endian or big-endian. If I order your input the other way around I will get Which equals your 0xB16BAADC0DEF0015 |
|
Can you send both of those files as binary (zip, etc)? I think that roundtripping through |
|
Archive.zip |
|
Thanks, that was very helpful. The files are now workable. Can you send us the journal files 9 and 10 for analysis (note, they contain some user data). |
|
You should have received it on your support mailbox. |
|
Thanks, that was really interesting. The rest of the file is set to zeroes. On journal 10, the first transaction is 237 all the way to 276, byte ranges 0 .. 26,300,416. This looks very much like the journal 9 file was zeroed somehow. |
|
Ok strange, did you notice that this happened on several databases, and that either indexes, raven.voron or the transaction files are corrupt. Could this be ZFS related? Should I try to replicate (since this happened on multiple databases it might not be hard to reproduce)? |
|
If you can replicate, that would be amazing yes. I'm leaning more toward the k8s setup |
|
k3s should be using local path, will read into this, but I guess it shouldn't be more as a volume mount in docker (which should directly write without any virtual layers). The storage is just local SSD storage with default ZFS configuration. |
|
If everything is running on a single machine, then yes, that is really something that we would love to reproduce. |
|
Another issue, this is ARM64, and we are tracking a separate problem that may be related to bad JIT. |
So I was syncing to a new server and indexing and it got killed by OOM, which is not why I started this case. Half of my databases now have:
Voron.Exceptions.InvalidJournalException: The last synced transaction id was 191 (in journal: 9) but the first transaction being read in the recovery process is 237 in journal /ravendb/Databases/CUSTOMERNAME/Journals/0000000000000000010.journal
Or on multiple indexes:
---> Voron.Exceptions.InvalidJournalException: Failed to open a storage at /ravendb/Databases/CUSTOMERNAME/Indexes/ArticleAll due to invalid or missing journal files. In order to load the storage successfully we need all journals to be not corrupted. The recommended approach is to reset the index in order to recover from this error. Alternatively you can temporarily start the server in dangerous mode so it will ignore invalid journals on startup:
Raven.Server --Storage.Dangerous.IgnoreInvalidJournalErrors=true
This switch is meant to be use only for recovery purposes. Please make sure that you won't use it after you manage to recover your data. Eventually you should reset the index anyway to be sure you won't experience any invalid data errors in the future.
Error details: The last synced transaction id was 23 (in journal: 2) but the first transaction being read in the recovery process is 117 in journal /ravendb/Databases/CUSTOMERNAME/Indexes/ArticleAll/Journals/0000000000000000003.journal (transaction has valid hash). Tx diff is: 94. Some journals might be missing. Debug details - file header Version: 23, HeaderRevision: 10, TransactionId: 23, LastPageNumber: 3313, RootPageNumber: 0, CurrentJournal: 3, LastSyncedJournal: 2, LastSyncedTransactionId: 2, Flags: None. Journal details: CurrentJournal - 3, LastSyncedJournal - 2, LastSyncedTransactionId - 23, Flags - None
It has 2 other databases running on k3s, which don't have dataloss, and as far as I can see it isn't a hardware failure.
Maybe someone can shed some light on this.
Things I checked:
-- However, it is using local-path provisioner (not local, but the rancher / k3s default local-path), as far as I can tell it just maps a folder to the container/pod.
NAME PROPERTY VALUE SOURCE
poolname sync standard default
The text was updated successfully, but these errors were encountered: