Multiple AOF files with temporary prefix #11408
Replies: 2 comments 12 replies
-
|
You can probably delete them (leftover from a failed attempt for a rewrite) Can you please share a portion of your logs, and mention which redis version you are using? |
Beta Was this translation helpful? Give feedback.
-
|
@oranagra Please let me know if I can help with more information. |
Beta Was this translation helpful? Give feedback.
-
|
question is if the multitude of temporary files is caused by running out of disk space, or is it the other way around. in any case, this whole mechanism changed greatly in redis 7.0, so there's a good chance the problem is already resolved. @chenyang8094 can you please review the code (7.0, and if you have time 6.2 too) and make sure there are no flows that lead to leftover temp files on the disk? |
Beta Was this translation helpful? Give feedback.
-
|
It is the other way round. After we deleted all the temporary prefixed files, the data volume had available space upto 2.1G .The AOF file was close to 5.9G . However ,we saw that the AOF file spiked to 7.6G in the next 4 hours. Wanted help to know what potentially can contribute to the increase in the AOF file size? |
Beta Was this translation helpful? Give feedback.
-
|
i'm not sure i follow your last comment, please be more explicit, did the multiple failures cause the disk space issue or did the disk space issue caused the failures. AOF file size is controlled by the |
Beta Was this translation helpful? Give feedback.
-
|
the looking at that log file how ever, some things are unclear to me.
|
Beta Was this translation helpful? Give feedback.
-
|
The logs shared are for the master pod , fetched the same from Kibana. Sharing the below logs which brought down the whole cluster on 18th October ( after deleting the temporary files) and made a huge AOF file, the disk went out of space . The base AOF before deleting the temporary file was 3.9 G. Oct 18, 2022 @ 19:49:54.294 redis-service-master-0 1:X 18 Oct 2022 19:49:54.294 # +odown master mymaster ip3 6379 #quorum 3/2 |
Beta Was this translation helpful? Give feedback.
-
|
[this log is upside down (took me a while to figure it out)] i do see the RDB child printing i'm still not certain what causes the temporary AOF files, but i'd advise to upgrade to a more recent version, in which the problem is likely to be solved, and if not, we can invest a greater effort to fix it. |
Beta Was this translation helpful? Give feedback.
-
|
@oranagra - Thank you , we wlll plan to update the version soon. However to close on the thread ,I understand with the data that we shared there are unknowns , but as we know there was a huge spike in the base AOF file , could that be related to
Any pointers/thoughts which might have been related to the spike of AOF file will be highly helpful for us to take care of failure in the interim. |
Beta Was this translation helpful? Give feedback.
-
|
@chenyang8094 - Can you please help with the above. |
Beta Was this translation helpful? Give feedback.
-
Hi Team ,
We are seeing a lot of temporary prefixed AOF files in the volume path ( used kubernetes operator to build the redis cluster) . Is this because of the retry process for AOFRW ?
Any guidance will be of great help.
Beta Was this translation helpful? Give feedback.
All reactions