Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can we only focus the data changes of the tables we care about? #808

Open
elliotchenzichang opened this issue Jul 23, 2023 · 1 comment

Comments

@elliotchenzichang
Copy link

elliotchenzichang commented Jul 23, 2023

BackGround
Our usage scenario is that there are about a dozen tables in our database, but we only pay attention to five changes. The amount of changes in a table other than the following five tables far exceeds the five we are concerned about. Therefore, the memory consumption of our service is very high, but the CPU utilization is very low. In addition, the synchronization lag of these five tables has also become significantly larger.

Thoughts
I feel that it should be caused by the continuous creation of buffer and decoding operations when receiving changes to that table. Do we have a mechanism to judge that the received event is not what we care about, so we don't continue to decode it and don't transmit it to the user? If not, do you think it would make sense if it was implemented as a feature? I can do it if I can. grateful!

@lance6716
Copy link
Collaborator

lance6716 commented Jul 24, 2023

Please help translate your issue to English so more users will understand it.


As a user of this library, we also meet this problem before, and we propose this solution

#737

And this is the change at applition side https://github.com/pingcap/tiflow/blob/02c6aa9d2d8c256d9177db55f0a3b3138e7ec761/dm/syncer/util.go#L168-L213

Welcome to discuss to provide a more elegant API 😄

@elliotchenzichang elliotchenzichang changed the title 可以配置只接受指定几张table的变更事件吗? Can we only focus the data changes of the tables we care about? Jul 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants