Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Enhancement]: Trigger compaction by size #32328

Open
1 task done
tedxu opened this issue Apr 16, 2024 · 1 comment
Open
1 task done

[Enhancement]: Trigger compaction by size #32328

tedxu opened this issue Apr 16, 2024 · 1 comment
Labels
kind/enhancement Issues or changes related to enhancement stale indicates no udpates for 30 days

Comments

@tedxu
Copy link
Collaborator

tedxu commented Apr 16, 2024

Is there an existing issue for this?

  • I have searched the existing issues

What would you like to be added?

I would like to change the compaction trigger size based, instead of estimation based.

Why is this needed?

Compaction is currently triggered by size estimated indirectly from row number, instead of segment size itself.

That kind of estimation is not accurate, especially in those cases that variable length columns are involved. If there is an underestimate, the actual size of the compact segment may exceed expectation and will never be fixed.

Anything else?

No response
@tedxu tedxu added the kind/enhancement Issues or changes related to enhancement label Apr 16, 2024
@tedxu tedxu mentioned this issue Apr 16, 2024
Merged
sre-ci-robot pushed a commit that referenced this issue Apr 23, 2024
@tedxu
enhance: trigger compaction by size (#32326)
fef7812 
See #32328

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
XuanYang-cn pushed a commit to XuanYang-cn/milvus that referenced this issue May 8, 2024
@tedxu @XuanYang-cn
enhance: [2.3]trigger compaction by size and fix (milvus-io#32326)
ba2ea28 
fix: Compaction trigger choose 2 same segments (milvus-io#32800)

DataNode would stuck at compactor try to lock the
same segmentID

See milvus-io#32328, milvus-io#32765

Co-Authored-by: Ted Xu <ted.xu@zilliz.com>
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
XuanYang-cn pushed a commit to XuanYang-cn/milvus that referenced this issue May 8, 2024
@tedxu @XuanYang-cn
enhance: [2.4]trigger compaction by size and fix (milvus-io#32326)
af6811f 
fix: Compaction trigger choose 2 same segments (milvus-io#32800)

DataNode would stuck at compactor try to lock the
same segmentID

See milvus-io#32328, milvus-io#32765

Co-Authored-by: Ted Xu <ted.xu@zilliz.com>
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
@XuanYang-cn XuanYang-cn mentioned this issue May 8, 2024
Merged
sre-ci-robot pushed a commit that referenced this issue May 9, 2024
@XuanYang-cn @tedxu
enhance: [2.4]trigger compaction by size and fix (#32326) (#32846)
c0d9efa 
fix: Compaction trigger choose 2 same segments (#32800)

DataNode would stuck at compactor try to lock the
same segmentID

See: #32328, #32765
pr: #32800 #32326

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: Ted Xu <ted.xu@zilliz.com>
@stale Stale
Copy link

stale bot commented May 18, 2024

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

@stale stale bot added the stale indicates no udpates for 30 days label May 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/enhancement Issues or changes related to enhancement stale indicates no udpates for 30 days
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@tedxu