Fix endless chunk rewriting when db is idle - improvement #4000
Conversation
| int adjustedChunksFillRate = 100 - (100 - chunksFillRate) / 2; | ||
| int fillRateToCompare = isIdle() ? chunksFillRate : adjustedChunksFillRate; | ||
| int chunksFillRate = getChunksFillRate(); | ||
| if (lastHousekeepingHasRewrittenChunks && chunksFillRate == previousChunksFillRate) { |
There was a problem hiding this comment.
It looks like this condition may yield true only by a coincident, because if last housekeeping has rewritten chunks, chunks fill rate is likely to change, even without any user activity
There was a problem hiding this comment.
To be honest, It is based on the observations I have made on my database where practically the chunkFillRate was not moving during the housekeeping (quickly stayed in my case at 78%): Only the rewriteable chunk fill rate was incrementally increasing (up to 100%) until the retention time of the first rewritten chunks was hit, driving back the rewritable chunk fill rate to the starting point and restarting the useless housekeeping cycle.
The condition has also an equality ( and not a <=) because I needed a way to detect when to "resume" the housekeeping if some user operations were made (at least enough operations to have an impact on the chunkFillRate) and without using isIdle.
So the logic is really:
- The housekeeping has just rewritten some chunk with no impact on the chunkFillRate ? -> Yes, then stop performing the housekeeping
- When to resume the housekeeping? -> Whenever there was enough user operations to change the chunkFillRate (by at least a point)
But you are right, I guess the fact that the chunkFillRate is not "moved" by the housekeeping in my tests might very well be specific to my database(s). And depending on the database, we could probably have a situation where chunkFillRate is also somehow fluctuating (78, 79, 77, 78, 79, ...) after each housekeeping, blowing up the stopping condition.
I have seen you have made another proposal (in #4006). Thanks a lot for that. I will thoroughly test it and will come back to you as soon as I can.
…-db-is-idle' into fix-endless-chunk-rewriting-when-db-is-idle
|
Close/Declining this as @andreitokar PR has been merged |
Hello,
I would like to officially propose this change in the MVStore housekeeping and would be happy to get any feedback.
This is based on the comment I have made here: #3909 (comment)
Thank you in advance for your feedback.