Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enhance heartbeat processing capabilities #7897

Open
7 of 8 tasks
nolouch opened this issue Mar 7, 2024 · 0 comments
Open
7 of 8 tasks

Enhance heartbeat processing capabilities #7897

nolouch opened this issue Mar 7, 2024 · 0 comments
Labels
type/development The issue belongs to a development tasks

Comments

@nolouch
Copy link
Contributor

nolouch commented Mar 7, 2024 •
edited

Development Task

Sigle node processing capacity

Improve the diagnosis process and implement hierarchical processing for Heartbeat logic

Optimize Lock and Reduce lock holding time

Improve the memory alloc

Quantify

  • Benchmark

Scalability

  • [TBD] Proposal
    • only metadata(lookup) part, statics part not consider
@nolouch nolouch added the type/development The issue belongs to a development tasks label Mar 7, 2024
@nolouch nolouch mentioned this issue Mar 7, 2024
Closed
This was referenced Apr 3, 2024
Merged
Merged
ti-chi-bot bot added a commit that referenced this issue Apr 8, 2024
@nolouch @ti-chi-bot
pkg/schedule: optimize the lock usage of operator controller (#8032)
949d602 
ref #7897

pkg/schedule: optimize the lock usage of the operator controller
- use sync.Map for operators, which is friendly for check operators on the heartbeat hot path
- reduce the lock hold time by splitting the locks.

Signed-off-by: nolouch <nolouch@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ti-chi-bot bot added a commit that referenced this issue Apr 9, 2024
@nolouch @ti-chi-bot
pkg/ratelimit: introduce an executor that can run with a limiter (#8024)
35e8e95 
ref #7897

pkg/ratelimit: Introduce an executor that can run with a limiter
- async runner can run the task with the limiter, the task will parallel run.
- sync runner used to keep the behavior like before

Signed-off-by: nolouch <nolouch@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
This was referenced Apr 10, 2024
Merged
Merged
Merged
ti-chi-bot bot added a commit that referenced this issue Apr 11, 2024
@nolouch @ti-chi-bot
pkg/schedule: put merge operators together (#8050)
96a69fc 
ref #7897, close #8049

pkg/schedule: put merge operators together to maintain atomicity

Signed-off-by: nolouch <nolouch@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@ti-chi-bot ti-chi-bot mentioned this issue Apr 15, 2024
Merged
ti-chi-bot bot pushed a commit that referenced this issue Apr 15, 2024
@ti-chi-bot @nolouch
pkg/schedule: put merge operators together (#8050) (#8063)
3ec92bd 
ref #7897, close #8049

pkg/schedule: put merge operators together to maintain atomicity

Signed-off-by: nolouch <nolouch@gmail.com>

Co-authored-by: nolouch <nolouch@gmail.com>
ti-chi-bot bot pushed a commit that referenced this issue Apr 15, 2024
@nolouch
*: optimize heartbeat process with concurrent runner - part 1 (#8053)
0214778 
ref #7897

*: Optimize the heartbeat process with the concurrent runner 
- add a switch about the runner
- move some tasks out, not block the heartbeat process

Signed-off-by: nolouch <nolouch@gmail.com>
@rleungx rleungx mentioned this issue Apr 22, 2024
Open
ti-chi-bot bot added a commit that referenced this issue Apr 24, 2024
@nolouch @ti-chi-bot
*: optimize heartbeat process with concurrent runner - part 2 (#8052)
141186e 
ref #7897

Optimize heartbeat process
- Split the statics updates on the subtree

Signed-off-by: nolouch <nolouch@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@nolouch nolouch mentioned this issue Apr 28, 2024
Merged
@rleungx rleungx pinned this issue Apr 29, 2024
@rleungx rleungx mentioned this issue Apr 29, 2024
Merged
@nolouch nolouch mentioned this issue Apr 29, 2024
Merged
nolouch added a commit to nolouch/pd that referenced this issue Apr 29, 2024
@nolouch @ti-chi-bot
*: optimize heartbeat process with concurrent runner - part 2 (tikv#8052
cd0e1bf 
)

ref tikv#7897

Optimize heartbeat process
- Split the statics updates on the subtree

Signed-off-by: nolouch <nolouch@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ti-chi-bot bot added a commit that referenced this issue Apr 30, 2024
@rleungx @ti-chi-bot
pkg: fix RegionStatsNeedUpdate condition (#8133)
8ee18a4 
ref #7897

Signed-off-by: Ryan Leung <rleungx@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@nolouch nolouch mentioned this issue Apr 30, 2024
Merged
ti-chi-bot bot added a commit that referenced this issue May 6, 2024
@nolouch @ti-chi-bot
core: check stats healthy by reference (#8132)
f83feba 
ref #7897, close #8131

Signed-off-by: nolouch <nolouch@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ti-chi-bot bot added a commit that referenced this issue May 7, 2024
@rleungx @ti-chi-bot
*: split into multiple runner for heartbeat (#8130)
a3c5950 
ref #7897

Signed-off-by: Ryan Leung <rleungx@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@rleungx rleungx mentioned this issue May 9, 2024
Merged
ti-chi-bot bot pushed a commit that referenced this issue May 10, 2024
@nolouch
*: add pending task metrics in runner and add templ (#8138)
78cb081 
ref #7897

Signed-off-by: nolouch <nolouch@gmail.com>
This was referenced May 10, 2024
Open
Merged
ti-chi-bot bot added a commit that referenced this issue May 13, 2024
@rleungx @ti-chi-bot
*: remove task name opt (#8168)
2278609 
ref #7897

Signed-off-by: Ryan Leung <rleungx@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@rleungx rleungx mentioned this issue May 15, 2024
Merged
This was referenced May 16, 2024
Merged
Merged
Merged
ti-chi-bot bot pushed a commit that referenced this issue May 16, 2024
@rleungx
pkg: improve get forwarded host (#8182)
7e85817 
ref #7897

Signed-off-by: Ryan Leung <rleungx@gmail.com>
@rleungx rleungx mentioned this issue May 16, 2024
Merged
ti-chi-bot bot added a commit that referenced this issue May 16, 2024
@rleungx @ti-chi-bot
pkg: improve region path (#8180)
59e29cc 
ref #7897

Signed-off-by: Ryan Leung <rleungx@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@JmPotato JmPotato mentioned this issue May 16, 2024
Merged
ti-chi-bot bot added a commit that referenced this issue May 17, 2024
@rleungx @ti-chi-bot
pkg: improve region statistics observe (#8181)
79083e9 
ref #7897

Signed-off-by: Ryan Leung <rleungx@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ti-chi-bot bot added a commit that referenced this issue May 17, 2024
@rleungx @ti-chi-bot
core: use sync pool for tracer (#8183)
644e904 
ref #7897

Signed-off-by: Ryan Leung <rleungx@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@nolouch nolouch mentioned this issue May 17, 2024
Merged
ti-chi-bot bot added a commit that referenced this issue May 17, 2024
@JmPotato @ti-chi-bot @HuSharp
core/region: use a separate tree to check for overlaps in the subtree (…
36a1fa2 
…#8185)

ref #7897, close #8136

Previously, removing overlapped regions in subtrees required finding overlaps for each subtree individually.
This PR introduces a separate tree specifically for checking overlaps in the subtrees, thereby reducing lookup times and improving performance.

Signed-off-by: JmPotato <ghzpotato@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Co-authored-by: Hu# <jinhao.hu@pingcap.com>
@rleungx rleungx mentioned this issue May 17, 2024
Merged
ti-chi-bot bot pushed a commit that referenced this issue May 18, 2024
@nolouch
core/region: fix sub tree may not consist with root tree (#8187)
6ded6f8 
ref #7897

core/region: fix the sub tree may not consist of root tree

Signed-off-by: nolouch <nolouch@gmail.com>
@JmPotato JmPotato mentioned this issue May 20, 2024
Merged
ti-chi-bot bot pushed a commit that referenced this issue May 21, 2024
@JmPotato
core/region: fix the bug causing incorrect reference count after a ca…
19eda09 
…che reset (#8201)

ref #7897

#8132 introduced a count reference to ensure the consistency between
the root tree and subtrees, however, the reference count was not handled correctly during the cache
reset process, which may lead to incorrect reference counts in later heartbeat processing.

Signed-off-by: JmPotato <ghzpotato@gmail.com>
@JmPotato JmPotato mentioned this issue May 21, 2024
Merged
ti-chi-bot bot pushed a commit that referenced this issue May 21, 2024
@rleungx
pkg: reduce the allocation of observe (#8188)
7731ee4 
ref #7897

Signed-off-by: Ryan Leung <rleungx@gmail.com>
ti-chi-bot bot pushed a commit that referenced this issue May 21, 2024
@rleungx
*: use a separate runner for updating subtree (#8158)
58e7580 
ref #7897

Signed-off-by: Ryan Leung <rleungx@gmail.com>
@nolouch nolouch mentioned this issue May 22, 2024
Merged
ti-chi-bot bot pushed a commit that referenced this issue May 22, 2024
@nolouch
pkg/statistics: reduce the memory alloc for hot statistics (#8208)
43e9492 
ref #7897, close #8207

pkg/statistics: reduce the memory alloc for hot statistics

Signed-off-by: nolouch <nolouch@gmail.com>
@nolouch nolouch mentioned this issue May 22, 2024
Merged
ti-chi-bot bot pushed a commit that referenced this issue May 22, 2024
@nolouch
metrics: update grafana template to add heartbeat latency overview (#…
b871b57 
…8209)

ref #7897

metrics: update grafana template to add heartbeat latency overview

Signed-off-by: nolouch <nolouch@gmail.com>
ti-chi-bot bot added a commit that referenced this issue May 22, 2024
@JmPotato @ti-chi-bot
core/region: optimize the efficiency of random regions selecting (#8205)
0056569 
ref #7897

Optimize the efficiency of random regions selecting.

Signed-off-by: JmPotato <ghzpotato@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@nolouch nolouch mentioned this issue May 22, 2024
Merged
ti-chi-bot bot added a commit that referenced this issue May 23, 2024
@nolouch @ti-chi-bot
metrics: fix the duplicate avg metrics (#8210)
6b8d227 
ref #7897

metrics: fix the duplicate avg metrics

Signed-off-by: nolouch <nolouch@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@JmPotato JmPotato mentioned this issue May 23, 2024
Merged
okJiang pushed a commit to okJiang/pd that referenced this issue May 23, 2024
@nolouch @ti-chi-bot @okJiang
metrics: fix the duplicate avg metrics (tikv#8210)
ec6e1b5 
ref tikv#7897

metrics: fix the duplicate avg metrics

Signed-off-by: nolouch <nolouch@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: okJiang <819421878@qq.com>
okJiang pushed a commit to okJiang/pd that referenced this issue May 23, 2024
@nolouch @ti-chi-bot @okJiang
metrics: fix the duplicate avg metrics (tikv#8210)
27a74a5 
ref tikv#7897

metrics: fix the duplicate avg metrics

Signed-off-by: nolouch <nolouch@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Signed-off-by: okJiang <819421878@qq.com>
ti-chi-bot bot pushed a commit that referenced this issue May 23, 2024
@JmPotato
core/region: optimize the logic of randomly selecting regions within …
4cd42b3 
…a single range (#8211)

ref #7897

In the case of only one key range, follow the fast path logic to avoid unnecessary random and loop operations.

Signed-off-by: JmPotato <ghzpotato@gmail.com>
@rleungx rleungx mentioned this issue May 24, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/development The issue belongs to a development tasks
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@nolouch