3 node cluster - not behaving as expected after 1 node is offline

[db-wally007]

Hello,
I started experimenting with Grafana Mimir for our team with 3 node cluster running in "monolithic" mode on RHEL9.3 VMs (with HAProxy in front) and S3 Netapp backend.

I thought everything was running great on default settings, until I brought 1 node down for 4 hours to deploy it as RHEL9.4.

The upgraded node was redeployed and joined the cluster just as I would expect - however I soon realized that the some panels in Grafana had over 10 hour gaps in data - seemingly at random. Every time I refreshed the dashboard, different panel had missing 10 hours of metrics.

It looked like when the node joined the cluster, whatever metrics were not yet in long term storage (S3 bucket) was shown only from the 2 nodes that had the data locally while the newly deployed node obviously had no local data.

Since it was late at night, I gave up after about an hour and went to bed. Now 14 hours later, at here are no gaps.

Is this expected ?
Is there anyway to adjust this or prevent this behavior ? (as mentioned I'm using default config)
I'm thinking if I would have re-deployed 2 nodes (66%) , then 2/3rds of the metrics would have had gaps ?