[Enhancement]: Stop generate compaction plan when channel checkpoint lag is too big

[congqixia]

Is there an existing issue for this?

• I have searched the existing issues

What would you like to be added?

Stop generate compaction plan when channel checkpoint lag is too big

Why is this needed?

In 2.3, sync and compaction is mutually exclusive. When datanode is trying to catch up dml channel from position behind now a lot, compaction execution will block this procedure hand make is harder to catch up.
In 2.4(master), the segment view is not complete when checkpoint lag is large since there may be still some segment need to be flushed after.

It's always a better choice to wait channel checkpoint is up-to-date when trying to generate compaction plan.

Anything else?

No response

[xiaofan-luan]

What will happen in milvus 3.0?

1. flush is handled by lognode, there is no reason for compaction to understand about checkpoint?
2. flush and compaction will not depend on each other.
3. whether compaction should triggered could be purely depend on compaction service busy or not and data stats.

on 2.3, easier could be datanode directly reject compaction if it is still catch up checkpoints? @congqixia

[congqixia]

@xiaofan-luan
IMHO, for 3.0 implementation the compaction shall be scheduled the the meta holder, lognode/datacoord
And the executor shall be dumb worker and has no knowledge about channel checkpoint.
This enhancement has one benefit or takeaway for new architecture that maybe it's a better choice not to trigger compaction for each new flush segment since the data view might be changed soon.

Since datanode is dumb worker role in this case, maybe it's better to let Datacoord choose whether the compaction shall be executed or not?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

[xiaofan-luan]

Anything we need to improve on this issue?

[xiaofan-luan]

Maybe we should simply avoid pick segment to compact if they their checkpoint fall behind channel checkpoint?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.