Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Track memory usage in pessimistic locking phase #51217

Open
ekexium opened this issue Feb 21, 2024 · 5 comments · May be fixed by tikv/client-go#1301 or #52739
Open

Track memory usage in pessimistic locking phase #51217

ekexium opened this issue Feb 21, 2024 · 5 comments · May be fixed by tikv/client-go#1301 or #52739
Assignees
Labels
good first issue Denotes an issue ready for a new contributor, according to the "help wanted" guidelines. type/enhancement

Comments

@ekexium
Copy link
Contributor

ekexium commented Feb 21, 2024

Enhancement

part of #49643
image
The memory consumed in InitCheckExistence in the graph is not tracked by the memory tracker.

@ekexium ekexium added type/enhancement good first issue Denotes an issue ready for a new contributor, according to the "help wanted" guidelines. labels Feb 21, 2024
@RemHero
Copy link

RemHero commented Apr 11, 2024

I'm interested in this. Can you assign it to me? Could you please give me some tips?

@ekexium
Copy link
Contributor Author

ekexium commented Apr 11, 2024

Hi @RemHero, you can start from learning how other trackers track the memory usage of data structures, and apply the same way to this one. You can find the memory tracker defined in pkg/util/memory/tracker.go

@RemHero
Copy link

RemHero commented Apr 12, 2024

The InitCheckExistence is a method in tikv/client-go; refer to InitCheckExistence.
There seems to be no way to get the memory consumption inside InitCheckExistence with Tracker? Could you please give me another hint? @ekexium Thanks

@ekexium
Copy link
Contributor Author

ekexium commented Apr 15, 2024

@RemHero

  1. You need to figure out how to estimate the memory footprint of this part, as accurate as possible. mem_aware_map.go might help
  2. You need to record the memory usage via a tracker. It involves both TiDB repo and the client-go repo. You need to define a new tracker type, initialize it, and pass it to the function appropriately. One way worth consideration is passing a onMemChange closure to the structure.

@RemHero
Copy link

RemHero commented Apr 19, 2024

@ekexium
Hello! I have submitted client-go PR and TiDB PR. It should be noted that the structure of LockCtx in client-go is changed, but the client-go version referenced in TiDB is v2. Even if PR in client-go is merged, it cannot make TiDB perceive the structural change of LockCtx.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Denotes an issue ready for a new contributor, according to the "help wanted" guidelines. type/enhancement
Projects
None yet
2 participants