When I restart any Receive, the entire Seek cluster is unavailable, and the reboot can only be restored after the local data is fully understood

[zyz-1998]

I have a three-node Retain cluster with --receive.replication-factor of 2 and when I reboot one of them, it appears
receive-writer tenant=default-tenant msg="Error on ingesting samples that are too old or are too far into the future issue
Error fetching server time detected 30 secods time difference between your browser and the server thanos relies on accurate time and time drift might cause unexpected query result

[zyz-1998]

TSDB_PATH="./receiveData"
GRPC_ADDRESS="0.0.0.0:10912"
HTTP_ADDRESS="0.0.0.0:10911"
REPLICATION_FACTOR=2
LABEL_REPLICA="receive_replica="0""
LABEL_CLUSTER="receive_cluster="pre""
LOCAL_ENDPOINT="10.8.3.39:10912"
HASHRING_FILE="./hashring.json"
REMOTE_WRITE_ADDRESS="0.0.0.0:10918"
OBJSTORE_CONFIG_FILE="bucket_config.yml"

thanos receive
--tsdb.path=$TSDB_PATH
--grpc-address=$GRPC_ADDRESS
--http-address=$HTTP_ADDRESS
--receive.replication-factor=$REPLICATION_FACTOR
--label="$LABEL_REPLICA"
--label="$LABEL_CLUSTER"
--receive.local-endpoint="$LOCAL_ENDPOINT"
--receive.hashrings-file=$HASHRING_FILE
--remote-write.address=$REMOTE_WRITE_ADDRESS
--objstore.config-file=$OBJSTORE_CONFIG_FILE

--- hashring.json
[
{
"endpoints":[
"10.8.3.39:10912",
"10.8.3.40:10912",
"10.8.3.41:10912"
]
}
]

When I restart any Receive, the entire Seek cluster is unavailable, and the reboot can only be restored after the local data is fully understood

[GiedriusS]

Please see the discussion here. We either need to forbid replication factor 2 because it doesn't allow for any downtime or you need to switch to 3. Or replication factor 2 must be made a special case in the formula 😄