Inconsistent data between nodes in Raven Cloud prod cluster

[JudahGabriel]

Hi folks!

I have a RavenDB cloud cluster with 3 nodes and I'm seeing some data inconsistencies between the nodes.

On Friday, January 5th, I added some new documents and updated some others via my .NET app. Everything seemed to be working fine and the new data showed up in my app over next 3 days.

But this morning (Tuesday, January 9th), I opened my app again and discovered that it was missing data I had added on Friday. Also the documents I modified on Friday now appeared unmodified.

Looking at Raven Studio, I see nodes B and C have the correct data: both the new documents and the modified ones. But node A has neither new documents nor modified docs. Several hours have passed now and Raven doesn't seem to be rectifying the differences. Here are the change vectors for a doc I modified on Friday:

Looking at Raven Studio alerts, I see this notification, maybe it's related?

Raven Studio shows no conflicts. Raven version 6.0.2.

I'd be glad to give any additional information to the Raven team to help diagnose this, including access to my Raven Cloud cluster if needed.

[ayende]

Can you create a support ticket from within the cloud portal?
That looks like the database is rebuilding entirely. Are you using high-IO instances?

[ayende]

Note that the node C should not be visible for the application, since it was removed from the topology for not catching up

[JudahGabriel]

Are you using high-IO instances?

I don't think so.

Can you create a support ticket from within the cloud portal?

Will do, thanks.

[ayende]

High IO would be a PN instance, so not, that isn't what is happening.