raftstore: gc abnormal snapshots and destroy peer if failed to apply snapshots.

[LykxSassinator]

What is changed and how it works?

Issue Number: Close #15292

What's Changed:

Previously, there were pending tasks to address the scenario where TiKV would panic if applying snapshots failed due to abnormal conditions such as IO errors or unexpected issues.

This pull request resolves the issue by introducing additional traits tombstone: bool to SnapshotApplied, indicating whether the failure occurred due to abnormal snapshots.
Additionally, the abnormal peer will send ExtraMessageType::MsgGcPeerRequest to related leader of this region, trigger a new ConfChange with RemoveNode to gc the associated peer. Finally, this peer will be destroyed later to ensure the cluster will add one new peer by sending a fresh snapshot to the affected node.

Replace `SnapshotApplied` with `SnapshotApplied { peer_id: u64, tombstone: bool}`. And if `tombstone` == true, the
relative peer will be automatically GCed.

Related changes

• PR to update pingcap/docs/pingcap/docs-cn:
• Need to cherry-pick to the release branch

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Release note

None.

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has not been approved.

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[ti-chi-bot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[glorv]

I have 2 questions about this PR:

1. Is it possible to directly retry applying the snapshot if the failure is caused by unexpected(maybe some IO layer) error?
2. Why need to tombstone the peer after applying snapshot failed? Why not just switch the peer status to normal and so the leader should start a new snapshot automatically.

[LykxSassinator]

I have 2 questions about this PR:

1. Is it possible to directly retry applying the snapshot if the failure is caused by unexpected(maybe some IO layer) error?
2. Why need to tombstone the peer after applying snapshot failed? Why not just switch the peer status to normal and so the leader should start a new snapshot automatically.

1. Nope, it's caused by loading abnormal blocks. According to the logs in the issue tikv panic repeatedly after this tikv recover from io hang #15292 and tikv panic repeatedly with “\"[region 16697056] 19604003 applying snapshot failed\"” after down this tikv for 20mins and recover #16958, it shows that the given snapshot has abnormal file blocks, causing the failure of the applying.
2. For safety. Tombstone the peer and destroy it will assure that this TiKV node do not have any other remains data / meta data about this peer.

[LykxSassinator]

By the way, as for the point 2, I agree what u mentioned before. But for safety, this pr takes current implementation.

Why not just switch the peer status to normal and so the leader should start a new snapshot automatically.

I'll have a try for the point 2 u mentioned and do some extra tests on it to find the more appropriate way.

[overvenus]

I don't think we can directly mark it as Tombstone or Normal, because both options violate the current raft state machine protocol.

• Tombstone indicates that the peer has been fully removed, including its Raft membership and data. However, in this case, the peer remains a valid member.
• Normal means that the peer has all data up to its last commit index. But, in this case, it does not, as its last commit index has been updated to the snapshot index.

---

There are two ways to fix the panic:

1. Have PD remove this peer via a confchange, which I believe is the simplest solution.
2. Introduce a new RPC to instruct the leader to resend the snapshot, which may change lots of code.

[glorv]

Introduce a new RPC to instruct the leader to resend the snapshot, which may change lots of code.

Why need this extra RPC? At the leader side, it will switch the peer's state to normal after finishing send the snapshot. At the follower side, when apply snapshot failed, it is also doable to restore the raft state to its previous state before this snapshot. Thus, If I understand correctly, the leader should start another snapshot without any extra operation?

[overvenus]

At the follower side, when apply snapshot failed, it is also doable to restore the raft state to its previous state before this snapshot.

Do you mean persisting the previous state so it can be restored even after restarting TiKV?

That's doable (without introducing a new RPC), but it does add extra complexity to raftstore (and we'll need to review every code path related to snapshot handling).

[glorv]

Do you mean persisting the previous state so it can be restored even after restarting TiKV?

Yes. But I think just update the peer's state to its previous status should be simpler than tombstone or let pd remove this peer and later add it back.

[LykxSassinator]

Do you mean persisting the previous state so it can be restored even after restarting TiKV?

Yes. But I think just update the peer's state to its previous status should be simpler than tombstone or let pd remove this peer and later add it back.

Correct me if I'm wrong, but I think that removing the peer and making PD add one fresh peer sounds valid ? As the peer, which panics due to loading the abnormal snapshot, has already advanced its persisted_index and committed_index, rolling back the persisted state of this peer to a previous one will violate rules of raft protocol. Right ?

[glorv]

Correct me if I'm wrong, but I think that removing the peer and making PD add one fresh peer sounds valid ? As the peer, which panics due to loading the abnormal snapshot, has already advanced its persisted_index and committed_index, rolling back the persisted state of this peer to a previous one will violate rules of raft protocol. Right ?

You are right, the raft state is reset to the snapshot's state when receives the snapshot msg, so it not easy to revert to the previous state. And makes the raft state revert may cause other side-effects.

Thus, if it's doable, I prefer the current idea to just destroy the current peer and let newer raft message to trigger create it automatically.

[glorv]

@LykxSassinator from

tikv/components/raftstore/src/store/fsm/store.rs

Lines 2114 to 2130 in 06eed73

	if let Some(local_peer_id) = find_peer(region, self.ctx.store_id()).map(|r| r.get_id()) {
	if to_peer_id <= local_peer_id {
	self.ctx
	.raft_metrics
	.message_dropped
	.region_tombstone_peer
	.inc();
	info!(
	"tombstone peer receives a stale message, local_peer_id >= to_peer_id in msg";
	"region_id" => region_id,
	"local_peer_id" => local_peer_id,
	"to_peer_id" => to_peer_id,
	"msg_type" => ?msg_type
	);
	return Ok(CheckMsgStatus::DropMsg);
	}
	}

It seems after tombstoning the peer, the region will drop all following raft messages, thus the peer won't be recovered automatically.

[LykxSassinator]

@LykxSassinator from

tikv/components/raftstore/src/store/fsm/store.rs

Lines 2114 to 2130 in 06eed73

	if let Some(local_peer_id) = find_peer(region, self.ctx.store_id()).map(|r| r.get_id()) {
	if to_peer_id <= local_peer_id {
	self.ctx
	.raft_metrics
	.message_dropped
	.region_tombstone_peer
	.inc();
	info!(
	"tombstone peer receives a stale message, local_peer_id >= to_peer_id in msg";
	"region_id" => region_id,
	"local_peer_id" => local_peer_id,
	"to_peer_id" => to_peer_id,
	"msg_type" => ?msg_type
	);
	return Ok(CheckMsgStatus::DropMsg);
	}
	}

It seems after tombstoning the peer, the region will drop all following raft messages, thus the peer won't be recovered automatically.

Yes, thank you for the reminder. It seems that we may need to utilize ConfChange for this task. I will investigate further, and this pull request will be put on hold until the necessary changes are implemented and ready for review.

[LykxSassinator]

/cc @glorv @overvenus PTAL, now this pr is ready.

[Connor1996]

Why not handle it when JOB_STATUS_FAILED in check_applying_snap

[LykxSassinator]

IMO, check_applying_snap should be responsible solely for checking and returning the latest applying status, which is updated by the RegionRunner thread. The RegionRunner thread is the first to obtain the applying status.

[glorv]

What if send_tombstone_peer_msg failed or the message is dropped due to epoch change or leadership change. If current peer is not destroyed directly, we need some mechanism to mark that it is abnormal and should not be used for read/write anymore.

[LykxSassinator]

Make senses. The absence of messages can result in abnormal peer residue on this node.

Let me dive deeper in it.

[ti-chi-bot]

@LykxSassinator: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-unit-test	b34a49f	link	true	/test pull-unit-test

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[hbisheng]

Out of curiosity and just trying to learn, what are the typical scenarios where applying snapshots may fail? In those cases, would it help if we retry a few times? If we destroy the peer and add a new one, it's possible that the new peer may hit the same issue again, right?

[LykxSassinator]

Out of curiosity and just trying to learn, what are the typical scenarios where applying snapshots may fail? In those cases, would it help if we retry a few times? If we destroy the peer and add a new one, it's possible that the new peer may hit the same issue again, right?

Yep. One typical case is that loading snapshots encounters IO errors. In this case, the issue can be attributed to physical errors on the disk or system errors, causing the TiKV panic.
And as for whether the retrying is success or not, it depends on the root cause of the IO error. If it's due to physical errors on the disk, retries may not always be successful. But if it's a system error or bugs, retries can have a higher chance of success.