RFC:TiKV RawKV MVCC GC #90

ystaticy · 2022-03-22T06:08:13Z

Summary

Move TiKV MVCC GC worker from TiDB into a group of independent GC worker node role and implement a new GC process in TiKV for RawKV.

Motivation

GC worker is an important component for TiKV that deletes outdated MVCC data so as to not explode the storage. But currently, the GC worker is implemented in TiDB, which makes TiKV not usable without TiDB.And current GC process is just for transaction of TiDB,it's not usable for RawKV.

Signed-off-by: ystaticy <y_static_y@sina.com>

andylokandy · 2022-03-22T06:22:22Z

text/0090-tikv-gc.md

+
+## Detailed design
+For support TiKV cluster deploy without TiDB nodes.
+1. Add a new node role instead of GC worker in TiDB nodes.


We need to decide exactly which role before merging the RFC.

andylokandy · 2022-03-22T06:23:31Z

text/0090-tikv-gc.md

+For support TiKV cluster deploy without TiDB nodes.
+1. Add a new node role instead of GC worker in TiDB nodes.
+   The code of new GC worker,It will be added into [tikv/migration](https://github.com/tikv/migration)
+2. And for API V2, we need add new CompactionFilter which is named RawGCcompactionFilter, and add a new GCTask type implementation.


There is already a CompactionFilter for RawKV. Are we going to add a new one or just modify it?

TTLCompactionFilter is just for APIV1TTL,In API V2 ,There are several differences with APIV1TTL.I think add a new CompactionFilter is better, @pingyu What do you think?

andylokandy · 2022-03-22T06:24:26Z

text/0090-tikv-gc.md

+1. Add a new node role instead of GC worker in TiDB nodes.
+   The code of new GC worker,It will be added into [tikv/migration](https://github.com/tikv/migration)
+2. And for API V2, we need add new CompactionFilter which is named RawGCcompactionFilter, and add a new GCTask type implementation.
+3. GC conditions in RawGCcompactionFilter is:  (ts < GCSafePoint) && ( ttl-expired || deleted-mark || not the newest version ).  


Need to clarify the GCSafePoint. For example (1) Do RawKV, TiDB, and TxnKV share the same GCSafePoint? (2) Who will push GCSafePoint forward?

We may want to remove a key that ts > GCSafePoint but ttl-expired = true. @pingyu What do you think?

We should keep such entries. Expired entries are just the same as logical deleted. Miss to capture these entries will cause downstream to return an earlier version, and being inconsistent to upstream.

@pingyu So there will be this case: the CDC may sync a key that is already expired with downstream, right?

(2) Who will push GCSafePoint forward?
the new node role will instead of GC Worker in TiDB, It will push GCSafePoint forward.

@pingyu So there will be this case: the CDC may sync a key that is already expired with downstream, right?

Yes.

BusyJay · 2022-03-22T12:59:23Z

The RFC is quite simple yet. For example, how is the new node designed? How it triggers GC? How does GC work in details? What's the triggering frequency?

nolouch · 2022-03-23T02:14:49Z

text/0090-tikv-gc.md

+## Background
+According to the documentation for the current GC worker in a TiDB cluster, the GC process is as follows:
+
+In TiDB GC worker leader:


Currently, TiDB GC worker is not standardized use client-go. I think there are some problems that also need to be solved. details: https://docs.google.com/document/d/1jA3lK9QbYlwsvn67wGsSuusD1Dzx7ANq_vya384RBIg/edit#

I have read this document and discussed with @MyonKeminta .For implement this proposal, it need a clearer design and more development.
But due to scheduling reasons, we hope to make changes to GC as small as possible to support TiKV API V2.
So I submit this RFC.

ystaticy · 2022-03-23T07:00:29Z

@BusyJay OK,I'll add more details for this RFC

Signed-off-by: ystaticy <y_static_y@sina.com>

pingyu · 2022-03-30T05:11:59Z

text/0090-tikv-gc.md

+
+
+## Summary
+Move TiKV MVCC GC worker from TiDB into a group of independent GC worker node role and implement a new GC process in TiKV for RawKV.


As the subject of this RFC is for RawKV, I think we are not discussing about "move GC worker from TiDB".

I suggest the main contents to be:

Design of TiKV procedures for RawKV GC.

PD meta data & interfaces.

The new standalone GC worker component.

Discuss whether or how TiDB can migrate to this design.

"node role" seems to be not easy to understand.
Suggest to use "component", as TiUP using this word. See tiup.io.

pingyu · 2022-03-30T07:48:12Z

text/0090-tikv-gc.md

+## Summary
+Move TiKV MVCC GC worker from TiDB into a group of independent GC worker node role and implement a new GC process in TiKV for RawKV.
+
+## Motivation


The description of GC worker would be better to place in "Background" section.
I think the motivation includes:

We change RawKV encoding to MVCC, so the GC is necessary.

No GC for TxnKV scenario when TiDB is not deployed.

The GC safe point & Txn safe point is easy to be misunderstand.

pingyu · 2022-03-30T07:53:35Z

text/0090-tikv-gc.md

+Move TiKV MVCC GC worker from TiDB into a group of independent GC worker node role and implement a new GC process in TiKV for RawKV.
+
+## Motivation
+1.GC worker is an important component for TiKV that deletes outdated MVCC data so as to not explode the storage. But currently, the GC worker is implemented in TiDB, which makes TiKV not usable without TiDB.And current GC process is just for transaction of TiDB,it's not usable for RawKV.  


Suggested change

1.GC worker is an important component for TiKV that deletes outdated MVCC data so as to not explode the storage. But currently, the GC worker is implemented in TiDB, which makes TiKV not usable without TiDB.And current GC process is just for transaction of TiDB,it's not usable for RawKV.

1. GC worker is an important component for TiKV that deletes outdated MVCC data so as to not explode the storage. But currently, the GC worker is implemented in TiDB, which makes TiKV not usable without TiDB. And current GC process is just for transaction of TiDB, it's not usable for RawKV.

pingyu · 2022-03-30T07:55:36Z

text/0090-tikv-gc.md

+
+In TiDB GC worker leader:
+1. Regularly calculates a new timestamp called "GC safe point"(The default interval is 10min), and push the safe point to PD.
+2. Get the minimal Service safe point  among all services from the response of step 2, which is GC safe point .


Suggested change

2. Get the minimal Service safe point among all services from the response of step 2, which is GC safe point .

2. Get the minimal Service safe point among all services from the response of step 1, which is GC safe point .

pingyu · 2022-03-30T07:58:39Z

text/0090-tikv-gc.md

+1. Regularly calculates a new timestamp called "GC safe point"(The default interval is 10min), and push the safe point to PD.
+2. Get the minimal Service safe point  among all services from the response of step 2, which is GC safe point .
+3. Txn GC process: resolve locks and record delete ranges information.
+


Refer to RunDistributedGCJob, where are two more steps:

Save Txn safe point.

Upload GC safe point to PD.

pingyu · 2022-03-30T08:10:18Z

text/0090-tikv-gc.md

+For support RawKV GC in TiKV cluster deploy without TiDB nodes.
+1. Add a new node role instead of GC worker in TiDB nodes.
+- Why we choose to create a new node role:
+  - IF we add GC Worker in PD: It will cause the problem of client-go circular dependency.


I think the main reason is that PD is designed for scheduling & meta storage. Acting as GC worker will bring inappropriate feature and lower availability of PD.

pingyu · 2022-03-30T08:25:36Z

text/0090-tikv-gc.md

+  - Due to TiDB, TxnKV and RawKV are allowed to coexist. Because the data of the three scenarios are independent, Because the data of the three scenarios are independent, separate safepoints are used in the GC, which helps to reduce the interference between businesses and speed up the GC.
+  - If multi tenancy is supported in the future, 'service group' can also support it.
+  - Need to design new interfaces for update service safepoint with 'service group'.
+  - Add UpdateServiceGCSafepointByServiceGroup and getGCSafepointByServiceGroup to standardize the API.


Please also specify the arguments and returns, as well as the main logics of the interface.

The naming is a little too long. "UpdateServiceGCSafepointByGroup" may be better.

How to allocate $service_group_id ?

What's the path of GC safepoint ?

Will there be new interface & ETCD path for Txn safepoint ? TxnKV will need a Txn safepoint.

pingyu · 2022-03-30T08:33:09Z

text/0090-tikv-gc.md

+1. Get GC safe point from PD regularly.
+2. Deletion will be triggered in CompactionFilter and GcTask thread;
+
+## New GC worker architecture


Maybe it is not a new architecture. We are utilizing the same architecture but another implementation.

pingyu · 2022-03-30T08:34:30Z

text/0090-tikv-gc.md

+## New GC worker architecture
+In a TiKV cluster without TiDB nodes , there are a few different points as follows:
+1. We need to move GC worker into another node role.
+2. For [API V2](https://github.com/tikv/rfcs/blob/master/text/0069-api-v2.md) .It need gc the earlier version in default cf. But Txn GC worker process will be triggered by WriteCompactionFilter of write cf.


Suggested change

2. For [API V2](https://github.com/tikv/rfcs/blob/master/text/0069-api-v2.md) .It need gc the earlier version in default cf. But Txn GC worker process will be triggered by WriteCompactionFilter of write cf.

2. For [API V2](https://github.com/tikv/rfcs/blob/master/text/0069-api-v2.md), it need gc the earlier version in default cf. But Txn GC worker process will be triggered by WriteCompactionFilter of write cf.

Signed-off-by: ystaticy <y_static_y@sina.com>

AmoebaProtozoa · 2022-04-18T09:00:30Z

text/0090-tikv-gc.md

+         2. used to get GC safepoint for TiKV:
+            1. interface:
+               ```shell  
+               func (s *GrpcServer) GetAllServiceGroupGcSafePoint(ctx context.Context, request *pdpb.GetServiceGroupServiceGcSafeRequest)  (*pdpb.UpdateServiceGCSafePointResponse, error)


Should Gc here be capitalized to stay consistent with other interfaces?

Signed-off-by: ystaticy <y_static_y@sina.com>

sunxiaoguang · 2022-06-03T15:26:26Z

text/0090-tikv-gc.md

+Move TiKV MVCC GC worker from TiDB into a group of independent GC worker component and implement a new GC process in TiKV for RawKV.
+
+## Motivation
+1. GC worker is an important component for TiKV that deletes outdated MVCC data so as to not explode the storage. But currently, the GC worker is implemented in TiDB, which makes TiKV not usable without TiDB.And current GC process is just for transaction of TiDB,it's not usable for RawKV.  


Suggested change

1. GC worker is an important component for TiKV that deletes outdated MVCC data so as to not explode the storage. But currently, the GC worker is implemented in TiDB, which makes TiKV not usable without TiDB.And current GC process is just for transaction of TiDB,it's not usable for RawKV.

1. GC worker is an important component for TiKV that deletes outdated MVCC data so as to not explode the storage. But currently, the GC worker is implemented in TiDB, which makes TiKV not usable without TiDB. And current GC process is just for transaction of TiDB, it's not usable for RawKV.

sunxiaoguang · 2022-06-03T15:27:56Z

text/0090-tikv-gc.md

+
+In TiDB GC worker leader:
+1. Regularly calculates a new timestamp called "GC safe point"(The default interval is 10min), and push the safe point to PD.
+2. Get the minimal Service safe point among all services from the response of step 1, which is GC safe point .


Suggested change

2. Get the minimal Service safe point among all services from the response of step 1, which is GC safe point .

2. Get the minimal service safe point among all services from the response of step 1, which is GC safe point .

sunxiaoguang · 2022-06-03T15:29:55Z

text/0090-tikv-gc.md

+2. Deletion will be triggered in CompactionFilter and GcTask thread;
+
+## New GC worker implementation
+In a TiKV cluster without TiDB nodes , there are a few different points as follows:


Suggested change

In a TiKV cluster without TiDB nodes , there are a few different points as follows:

In a TiKV cluster without TiDB nodes, there are a few different points as follows:

sunxiaoguang · 2022-06-03T15:30:55Z

text/0090-tikv-gc.md

+## New GC worker implementation
+In a TiKV cluster without TiDB nodes , there are a few different points as follows:
+1. We need to move GC worker into another component.
+2. For [API V2](https://github.com/tikv/rfcs/blob/master/text/0069-api-v2.md) , it need gc the earlier version in default cf. But Txn GC worker process will be triggered by WriteCompactionFilter of write cf.


Suggested change

2. For [API V2](https://github.com/tikv/rfcs/blob/master/text/0069-api-v2.md) , it need gc the earlier version in default cf. But Txn GC worker process will be triggered by WriteCompactionFilter of write cf.

2. For [API V2](https://github.com/tikv/rfcs/blob/master/text/0069-api-v2.md), it needs gc the earlier version in default cf. But Txn GC worker process will be triggered by WriteCompactionFilter of write cf.

sunxiaoguang · 2022-06-04T03:48:18Z

text/0090-tikv-gc.md

+                        RequestHeader header = 1;
+                        bytes service_group_id = 2;
+                        bytes service_id = 3;
+                        int64 TTL = 4;


TTL doesn't follow field name standard. If ttl generates strange field accessor, we can change it to xxx_ttl or maybe time_to_live. They both look better and consistent.

sunxiaoguang · 2022-06-04T03:57:40Z

text/0090-tikv-gc.md

+2. Changes on PD:
+   1. A new concept is 'service group':  
+        - Due to TiDB, TxnKV and RawKV are allowed to coexist. Because the data of the three scenarios are independent, Because the data of the three scenarios are independent, separate safepoints are used in the GC, which helps to reduce the interference between businesses and speed up the GC.
+        - If multi tenancy is supported in the future, 'service group' can also support it.


Where do we put start and end key for a service group?

ystaticy added 2 commits March 22, 2022 14:09

RFC:TiKV RawKV MVCC GC

ed3ba92

Signed-off-by: ystaticy <y_static_y@sina.com>

RFC:TiKV RawKV MVCC GC

04c39e3

Signed-off-by: ystaticy <y_static_y@sina.com>

ystaticy force-pushed the rfc_tikv_gc branch from df6fd1c to 04c39e3 Compare March 22, 2022 06:18

andylokandy reviewed Mar 22, 2022

View reviewed changes

nolouch reviewed Mar 23, 2022

View reviewed changes

ystaticy added 2 commits March 29, 2022 16:40

RFC:TiKV RawKV MVCC GC. add design details and flow chart

66a5cb8

Signed-off-by: ystaticy <y_static_y@sina.com>

RFC:TiKV RawKV MVCC GC. add some details.

92b6b54

Signed-off-by: ystaticy <y_static_y@sina.com>

pingyu mentioned this pull request Mar 29, 2022

[to #48] modify TiDB CDC to TiKV CDC tikv/migration#74

Merged

pingyu reviewed Mar 30, 2022

View reviewed changes

ystaticy added 2 commits April 7, 2022 08:57

RFC:TiKV RawKV MVCC GC. add PD interface and pb design

2464d0c

Signed-off-by: ystaticy <y_static_y@sina.com>

feature RawKV GC,modify pd interface design

2186c22

Signed-off-by: ystaticy <y_static_y@sina.com>

AmoebaProtozoa reviewed Apr 18, 2022

View reviewed changes

ystaticy added 4 commits April 19, 2022 11:41

feature RawKV GC,modify interface response

15573ff

Signed-off-by: ystaticy <y_static_y@sina.com>

feature RawKV GC,modify interface response.

a86c475

Signed-off-by: ystaticy <y_static_y@sina.com>

feature RawKV GC,modify interface Safepoint to SafePoint.

552fbd6

Signed-off-by: ystaticy <y_static_y@sina.com>

feature RawKV GC,modify interface name in desc

03e416f

Signed-off-by: ystaticy <y_static_y@sina.com>

This was referenced Apr 25, 2022

pd: RawKV GC SafePoint API pingcap/kvproto#908

Merged

Add PD APIs for RawKV GC tikv/pd#4865

Closed

server, endpoint, client: Add PD APIs for RawKV GC Worker tikv/pd#4866

Closed

ystaticy added 5 commits April 28, 2022 10:09

RFC:TiKV RawKV MVCC GC.update pd interface

dcf16a3

Signed-off-by: ystaticy <y_static_y@sina.com>

RFC:TiKV RawKV MVCC GC.update etcd path

89194ba

Signed-off-by: ystaticy <y_static_y@sina.com>

RFC:TiKV RawKV MVCC GC.update compactionFilter name

3489396

Signed-off-by: ystaticy <y_static_y@sina.com>

feature RawKV GC.modify pb info

b211769

Signed-off-by: ystaticy <y_static_y@sina.com>

Merge branch 'master' into rfc_tikv_gc

27c71e8

pingyu mentioned this pull request May 11, 2022

storage: Add API for RawKV GC tikv/pd#4918

Closed

This was referenced May 12, 2022

storage: Add APIs for RawKV GC tikv/pd#4937

Merged

client: Add APIs for RawKV GC tikv/pd#4943

Closed

storage: Use etcd lease to manage safe point expiration tikv/pd#4984

Closed

sunxiaoguang reviewed Jun 3, 2022

View reviewed changes

sunxiaoguang reviewed Jun 4, 2022

View reviewed changes

AmoebaProtozoa mentioned this pull request Jun 9, 2022

server: Add APIs for RawKV GC tikv/pd#5138

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RFC:TiKV RawKV MVCC GC #90

RFC:TiKV RawKV MVCC GC #90

ystaticy commented Mar 22, 2022

andylokandy Mar 22, 2022

andylokandy Mar 22, 2022

ystaticy Mar 23, 2022

andylokandy Mar 22, 2022 •

edited

andylokandy Mar 22, 2022

pingyu Mar 22, 2022

andylokandy Mar 22, 2022

ystaticy Mar 23, 2022

pingyu Mar 25, 2022

BusyJay commented Mar 22, 2022

nolouch Mar 23, 2022 •

edited

ystaticy Mar 23, 2022

ystaticy commented Mar 23, 2022

pingyu Mar 30, 2022

pingyu Mar 30, 2022

pingyu Mar 30, 2022

pingyu Mar 30, 2022

pingyu Mar 30, 2022

pingyu Mar 30, 2022

pingyu Mar 30, 2022

pingyu Mar 30, 2022

pingyu Mar 30, 2022

pingyu Mar 30, 2022

AmoebaProtozoa Apr 18, 2022

sunxiaoguang Jun 3, 2022

sunxiaoguang Jun 3, 2022

sunxiaoguang Jun 3, 2022

sunxiaoguang Jun 3, 2022

sunxiaoguang Jun 4, 2022

sunxiaoguang Jun 4, 2022



		## Summary
		Move TiKV MVCC GC worker from TiDB into a group of independent GC worker node role and implement a new GC process in TiKV for RawKV.

	1.GC worker is an important component for TiKV that deletes outdated MVCC data so as to not explode the storage. But currently, the GC worker is implemented in TiDB, which makes TiKV not usable without TiDB.And current GC process is just for transaction of TiDB,it's not usable for RawKV.
	1. GC worker is an important component for TiKV that deletes outdated MVCC data so as to not explode the storage. But currently, the GC worker is implemented in TiDB, which makes TiKV not usable without TiDB. And current GC process is just for transaction of TiDB, it's not usable for RawKV.

	2. Get the minimal Service safe point among all services from the response of step 2, which is GC safe point .
	2. Get the minimal Service safe point among all services from the response of step 1, which is GC safe point .

	2. For [API V2](https://github.com/tikv/rfcs/blob/master/text/0069-api-v2.md) .It need gc the earlier version in default cf. But Txn GC worker process will be triggered by WriteCompactionFilter of write cf.
	2. For [API V2](https://github.com/tikv/rfcs/blob/master/text/0069-api-v2.md), it need gc the earlier version in default cf. But Txn GC worker process will be triggered by WriteCompactionFilter of write cf.

	In a TiKV cluster without TiDB nodes , there are a few different points as follows:
	In a TiKV cluster without TiDB nodes, there are a few different points as follows:

RFC:TiKV RawKV MVCC GC #90

Are you sure you want to change the base?

RFC:TiKV RawKV MVCC GC #90

Conversation

ystaticy commented Mar 22, 2022

Summary

Motivation

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andylokandy Mar 22, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BusyJay commented Mar 22, 2022

nolouch Mar 23, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ystaticy commented Mar 23, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andylokandy Mar 22, 2022 •

edited

nolouch Mar 23, 2022 •

edited