Buffer files remain after restart of fluentd

[chrko]

Describe the bug

After restart of fluentd, buffer files from previous runs are not being resumed and instead there for infinite time period. Example directory listing:

./worker0:
total 34M
drwxr-xr-x.  2 fluentd fluentd  12K Feb 19 13:47 .
drwxr-xr-x. 10 fluentd fluentd  162 Oct 27 16:27 ..
-rw-r--r--.  1 fluentd fluentd  476 Oct 27 16:27 buffer.b608b37e77dd4e1fc58407ccb1cfc2e6b.log
-rw-r--r--.  1 fluentd fluentd   90 Oct 27 16:27 buffer.b608b37e77dd4e1fc58407ccb1cfc2e6b.log.meta
-rw-r--r--.  1 fluentd fluentd 2.2K Nov 16 09:59 buffer.b60a413bab1e6ff1ac03ef16196cf381a.log
-rw-r--r--.  1 fluentd fluentd  133 Nov 16 09:59 buffer.b60a413bab1e6ff1ac03ef16196cf381a.log.meta
-rw-r--r--.  1 fluentd fluentd 1.1M Nov 16 09:59 buffer.b60a413bc1fdef1b4e58ac8bd4c6d8283.log
-rw-r--r--.  1 fluentd fluentd  121 Nov 16 09:59 buffer.b60a413bc1fdef1b4e58ac8bd4c6d8283.log.meta
-rw-r--r--.  1 fluentd fluentd 889K Nov 16 09:59 buffer.b60a413bc4d133e2441053fc6a4228f56.log
-rw-r--r--.  1 fluentd fluentd  131 Nov 16 09:59 buffer.b60a413bc4d133e2441053fc6a4228f56.log.meta
-rw-r--r--.  1 fluentd fluentd  32K Nov 16 09:59 buffer.b60a413bca34208585420e32d683c586f.log
-rw-r--r--.  1 fluentd fluentd  118 Nov 16 09:59 buffer.b60a413bca34208585420e32d683c586f.log.meta
-rw-r--r--.  1 fluentd fluentd  26K Nov 16 09:59 buffer.b60a413bd94483eb093e93ce74b023225.log
-rw-r--r--.  1 fluentd fluentd  118 Nov 16 09:59 buffer.b60a413bd94483eb093e93ce74b023225.log.meta
-rw-r--r--.  1 fluentd fluentd  78K Nov 16 09:59 buffer.b60a413bd9ddb8843d3da4863f8bcda3a.log
-rw-r--r--.  1 fluentd fluentd  118 Nov 16 09:59 buffer.b60a413bd9ddb8843d3da4863f8bcda3a.log.meta
-rw-r--r--.  1 fluentd fluentd  103 Nov 16 09:59 buffer.b60a413bdb43bef11c6e52eb0484253d0.log
-rw-r--r--.  1 fluentd fluentd   90 Nov 16 09:59 buffer.b60a413bdb43bef11c6e52eb0484253d0.log.meta
-rw-r--r--.  1 fluentd fluentd 4.2K Dec 14 10:18 buffer.b60c74c30550cdd1da9858df612edee45.log
-rw-r--r--.  1 fluentd fluentd  127 Dec 14 10:18 buffer.b60c74c30550cdd1da9858df612edee45.log.meta
-rw-r--r--.  1 fluentd fluentd 286K Dec 14 10:18 buffer.b60c74c3069c6a20aeecf40ccf3808336.log
-rw-r--r--.  1 fluentd fluentd  124 Dec 14 10:18 buffer.b60c74c3069c6a20aeecf40ccf3808336.log.meta
-rw-r--r--.  1 fluentd fluentd 2.6M Dec 14 10:18 buffer.b60c74c30b21db4a7b1fe4c178d7b8100.log
-rw-r--r--.  1 fluentd fluentd  120 Dec 14 10:18 buffer.b60c74c30b21db4a7b1fe4c178d7b8100.log.meta
-rw-r--r--.  1 fluentd fluentd 4.2M Dec 14 10:18 buffer.b60c74c3171230ed3e87786db6538818c.log
-rw-r--r--.  1 fluentd fluentd  125 Dec 14 10:18 buffer.b60c74c3171230ed3e87786db6538818c.log.meta
-rw-r--r--.  1 fluentd fluentd  90K Dec 14 10:18 buffer.b60c74c31781754ef9ad1afc97f5ce3df.log
-rw-r--r--.  1 fluentd fluentd  118 Dec 14 10:18 buffer.b60c74c31781754ef9ad1afc97f5ce3df.log.meta
-rw-r--r--.  1 fluentd fluentd  34K Dec 14 10:18 buffer.b60c74c31833e84db28e0bc51ba1f6745.log
-rw-r--r--.  1 fluentd fluentd  118 Dec 14 10:18 buffer.b60c74c31833e84db28e0bc51ba1f6745.log.meta
-rw-r--r--.  1 fluentd fluentd 744K Dec 14 10:18 buffer.b60c74c31ce490e051a60e9cb972de292.log
-rw-r--r--.  1 fluentd fluentd  114 Dec 14 10:18 buffer.b60c74c31ce490e051a60e9cb972de292.log.meta
-rw-r--r--.  1 fluentd fluentd 8.3K Dec 14 10:18 buffer.b60c74c31d3812faf86032ccdb07e7620.log
-rw-r--r--.  1 fluentd fluentd  116 Dec 14 10:18 buffer.b60c74c31d3812faf86032ccdb07e7620.log.meta
-rw-r--r--.  1 fluentd fluentd  21K Dec 14 10:18 buffer.b60c74c31dbc3cbb4d40025345d1967f7.log
-rw-r--r--.  1 fluentd fluentd  131 Dec 14 10:18 buffer.b60c74c31dbc3cbb4d40025345d1967f7.log.meta
-rw-r--r--.  1 fluentd fluentd 4.9M Dec 14 10:18 buffer.b60c74c321696a035494b9c54aa21fb2d.log
-rw-r--r--.  1 fluentd fluentd  109 Dec 14 10:18 buffer.b60c74c321696a035494b9c54aa21fb2d.log.meta
-rw-r--r--.  1 fluentd fluentd 869K Dec 14 10:18 buffer.b60c74c32d89b248652d05ccba9373ea5.log
-rw-r--r--.  1 fluentd fluentd  131 Dec 14 10:18 buffer.b60c74c32d89b248652d05ccba9373ea5.log.meta
-rw-r--r--.  1 fluentd fluentd  20K Dec 14 10:18 buffer.b60c74c3353d65e06377ccd170a7f2c7c.log
-rw-r--r--.  1 fluentd fluentd  109 Dec 14 10:18 buffer.b60c74c3353d65e06377ccd170a7f2c7c.log.meta
-rw-r--r--.  1 fluentd fluentd  25K Dec 14 10:18 buffer.b60c74c3355a31db7637adceaced6902f.log
-rw-r--r--.  1 fluentd fluentd  118 Dec 14 10:18 buffer.b60c74c3355a31db7637adceaced6902f.log.meta
-rw-r--r--.  1 fluentd fluentd 178K Dec 14 10:18 buffer.b60c74c3363f550f1abf5b55c16dfa83d.log
-rw-r--r--.  1 fluentd fluentd  120 Dec 14 10:18 buffer.b60c74c3363f550f1abf5b55c16dfa83d.log.meta
-rw-r--r--.  1 fluentd fluentd  103 Dec 14 10:18 buffer.b60c74c33cc77f3b0ab9c630051d6c135.log
-rw-r--r--.  1 fluentd fluentd   90 Dec 14 10:18 buffer.b60c74c33cc77f3b0ab9c630051d6c135.log.meta
-rw-r--r--.  1 fluentd fluentd  12K Jan 25 10:14 buffer.b60fc19cefe62096a56aaeeabc1911f43.log
-rw-r--r--.  1 fluentd fluentd  118 Jan 25 10:14 buffer.b60fc19cefe62096a56aaeeabc1911f43.log.meta
-rw-r--r--.  1 fluentd fluentd  25K Jan 25 10:14 buffer.b60fc19cf2d1dad727afa41516e2b3747.log
-rw-r--r--.  1 fluentd fluentd  128 Jan 25 10:14 buffer.b60fc19cf2d1dad727afa41516e2b3747.log.meta
-rw-r--r--.  1 fluentd fluentd  16K Jan 25 10:14 buffer.b60fc19cf5e20b1eee39ca3df5517a0e6.log
-rw-r--r--.  1 fluentd fluentd  122 Jan 25 10:14 buffer.b60fc19cf5e20b1eee39ca3df5517a0e6.log.meta
-rw-r--r--.  1 fluentd fluentd 117K Jan 25 10:14 buffer.b60fc19cf870d603641f9c74e00340bb4.log
-rw-r--r--.  1 fluentd fluentd  119 Jan 25 10:14 buffer.b60fc19cf870d603641f9c74e00340bb4.log.meta
-rw-r--r--.  1 fluentd fluentd 2.2K Jan 25 10:14 buffer.b60fc19cfa34e6c388506984cd53edc09.log
-rw-r--r--.  1 fluentd fluentd  132 Jan 25 10:14 buffer.b60fc19cfa34e6c388506984cd53edc09.log.meta
-rw-r--r--.  1 fluentd fluentd 106K Jan 25 10:15 buffer.b60fc19cfe8183b250ed3eb79034f9f02.log
-rw-r--r--.  1 fluentd fluentd  126 Jan 25 10:15 buffer.b60fc19cfe8183b250ed3eb79034f9f02.log.meta
-rw-r--r--.  1 fluentd fluentd  26K Jan 25 10:14 buffer.b60fc19d040a527c2b271d6095ff57238.log
-rw-r--r--.  1 fluentd fluentd  121 Jan 25 10:14 buffer.b60fc19d040a527c2b271d6095ff57238.log.meta
-rw-r--r--.  1 fluentd fluentd 1.6M Jan 25 10:14 buffer.b60fc19d09592747c7ab09fe02c04d9a8.log
-rw-r--r--.  1 fluentd fluentd  120 Jan 25 10:14 buffer.b60fc19d09592747c7ab09fe02c04d9a8.log.meta
-rw-r--r--.  1 fluentd fluentd  48K Jan 25 10:14 buffer.b60fc19d0e9937bc1de2fff416816ef77.log
-rw-r--r--.  1 fluentd fluentd  119 Jan 25 10:14 buffer.b60fc19d0e9937bc1de2fff416816ef77.log.meta
-rw-r--r--.  1 fluentd fluentd 3.2K Jan 25 10:14 buffer.b60fc19d110c935ca649f97fc7f974694.log
-rw-r--r--.  1 fluentd fluentd  132 Jan 25 10:14 buffer.b60fc19d110c935ca649f97fc7f974694.log.meta
-rw-r--r--.  1 fluentd fluentd 2.0M Jan 25 10:14 buffer.b60fc19d188e8f60e0a750d03a1e2e8d3.log
-rw-r--r--.  1 fluentd fluentd  125 Jan 25 10:14 buffer.b60fc19d188e8f60e0a750d03a1e2e8d3.log.meta
-rw-r--r--.  1 fluentd fluentd 2.0M Jan 25 10:14 buffer.b60fc19d1ef253a1024f18a9c6c53d4b0.log
-rw-r--r--.  1 fluentd fluentd  115 Jan 25 10:14 buffer.b60fc19d1ef253a1024f18a9c6c53d4b0.log.meta
-rw-r--r--.  1 fluentd fluentd  19K Jan 25 10:15 buffer.b60fc19d27f9a02d881910af0d1f755f1.log
-rw-r--r--.  1 fluentd fluentd  128 Jan 25 10:15 buffer.b60fc19d27f9a02d881910af0d1f755f1.log.meta
-rw-r--r--.  1 fluentd fluentd 121K Jan 25 10:15 buffer.b60fc19d2a1a62a15559597cc37dc8499.log
-rw-r--r--.  1 fluentd fluentd  106 Jan 25 10:15 buffer.b60fc19d2a1a62a15559597cc37dc8499.log.meta
-rw-r--r--.  1 fluentd fluentd 406K Jan 25 10:15 buffer.b60fc19d3156fab5d19afeb4c5df734da.log
-rw-r--r--.  1 fluentd fluentd  119 Jan 25 10:15 buffer.b60fc19d3156fab5d19afeb4c5df734da.log.meta
-rw-r--r--.  1 fluentd fluentd  103 Jan 25 10:15 buffer.b60fc19d315cdba7dc53b0321f8d50316.log
-rw-r--r--.  1 fluentd fluentd   90 Jan 25 10:15 buffer.b60fc19d315cdba7dc53b0321f8d50316.log.meta
-rw-r--r--.  1 fluentd fluentd 1.9K Jan 25 10:15 buffer.b60fc19d3308dc7a5058877703749a7fb.log
-rw-r--r--.  1 fluentd fluentd  127 Jan 25 10:15 buffer.b60fc19d3308dc7a5058877703749a7fb.log.meta
-rw-r--r--.  1 fluentd fluentd 152K Feb 15 10:19 buffer.b61168217bcbdb52182b29e5569cb9f5a.log
-rw-r--r--.  1 fluentd fluentd  119 Feb 15 10:19 buffer.b61168217bcbdb52182b29e5569cb9f5a.log.meta
-rw-r--r--.  1 fluentd fluentd  71K Feb 15 10:19 buffer.b61168218b52aff43df34d0df772b1388.log
-rw-r--r--.  1 fluentd fluentd  125 Feb 15 10:19 buffer.b61168218b52aff43df34d0df772b1388.log.meta
-rw-r--r--.  1 fluentd fluentd 2.8K Feb 15 10:19 buffer.b611682194441d76f04a17fdebf4a46e0.log
-rw-r--r--.  1 fluentd fluentd  135 Feb 15 10:19 buffer.b611682194441d76f04a17fdebf4a46e0.log.meta
-rw-r--r--.  1 fluentd fluentd 2.0M Feb 15 10:19 buffer.b61168219777e36babccff1907bbaa4ff.log
-rw-r--r--.  1 fluentd fluentd  109 Feb 15 10:19 buffer.b61168219777e36babccff1907bbaa4ff.log.meta
-rw-r--r--.  1 fluentd fluentd  75K Feb 15 10:19 buffer.b611682197c957955e42dd112e1a7cd96.log
-rw-r--r--.  1 fluentd fluentd  121 Feb 15 10:19 buffer.b611682197c957955e42dd112e1a7cd96.log.meta
-rw-r--r--.  1 fluentd fluentd 6.9K Feb 15 10:19 buffer.b61168219ae10ae4d0503d610c012d1b8.log
-rw-r--r--.  1 fluentd fluentd  131 Feb 15 10:19 buffer.b61168219ae10ae4d0503d610c012d1b8.log.meta
-rw-r--r--.  1 fluentd fluentd 1.2M Feb 15 10:19 buffer.b61168219f74f74d9593432f18e490ffb.log
-rw-r--r--.  1 fluentd fluentd  121 Feb 15 10:19 buffer.b61168219f74f74d9593432f18e490ffb.log.meta
-rw-r--r--.  1 fluentd fluentd 1.1K Feb 15 10:19 buffer.b6116821a44887a60dbcc41135e60565a.log
-rw-r--r--.  1 fluentd fluentd  113 Feb 15 10:19 buffer.b6116821a44887a60dbcc41135e60565a.log.meta
-rw-r--r--.  1 fluentd fluentd  44K Feb 15 10:19 buffer.b6116821abbc8c2f0ca48f7a7b8004d91.log
-rw-r--r--.  1 fluentd fluentd  122 Feb 15 10:19 buffer.b6116821abbc8c2f0ca48f7a7b8004d91.log.meta
-rw-r--r--.  1 fluentd fluentd 4.2K Feb 15 10:19 buffer.b6116821af5cc12a2295b992ed290b9a8.log
-rw-r--r--.  1 fluentd fluentd  127 Feb 15 10:19 buffer.b6116821af5cc12a2295b992ed290b9a8.log.meta
-rw-r--r--.  1 fluentd fluentd 125K Feb 15 10:20 buffer.b6116821b7da2e66ac59741b3b634200a.log
-rw-r--r--.  1 fluentd fluentd  106 Feb 15 10:20 buffer.b6116821b7da2e66ac59741b3b634200a.log.meta
-rw-r--r--.  1 fluentd fluentd 681K Feb 15 10:20 buffer.b6116821bf11c966eb10525beabc490d9.log
-rw-r--r--.  1 fluentd fluentd  131 Feb 15 10:20 buffer.b6116821bf11c966eb10525beabc490d9.log.meta
-rw-r--r--.  1 fluentd fluentd  103 Feb 15 10:20 buffer.b6116821bf5a6e50960d1e38addcb204d.log
-rw-r--r--.  1 fluentd fluentd   90 Feb 15 10:20 buffer.b6116821bf5a6e50960d1e38addcb204d.log.meta
-rw-r--r--.  1 fluentd fluentd  56K Feb 19 13:40 buffer.b611bb65a7c7b3c33be397b1855790414.log
-rw-r--r--.  1 fluentd fluentd  125 Feb 19 13:40 buffer.b611bb65a7c7b3c33be397b1855790414.log.meta
-rw-r--r--.  1 fluentd fluentd 126K Feb 19 13:40 buffer.b611bb65a9e744758d50a2e31fdd999db.log
-rw-r--r--.  1 fluentd fluentd  117 Feb 19 13:40 buffer.b611bb65a9e744758d50a2e31fdd999db.log.meta
-rw-r--r--.  1 fluentd fluentd  26K Feb 19 13:40 buffer.b611bb65ad19c96246b3e4b9cece1656c.log
-rw-r--r--.  1 fluentd fluentd  122 Feb 19 13:40 buffer.b611bb65ad19c96246b3e4b9cece1656c.log.meta
-rw-r--r--.  1 fluentd fluentd 942K Feb 19 13:40 buffer.b611bb65b732ab9124e7a4473fd200a8a.log
-rw-r--r--.  1 fluentd fluentd  121 Feb 19 13:40 buffer.b611bb65b732ab9124e7a4473fd200a8a.log.meta
-rw-r--r--.  1 fluentd fluentd 3.1M Feb 19 13:40 buffer.b611bb65c330a659948963b0c347028e5.log
-rw-r--r--.  1 fluentd fluentd  121 Feb 19 13:40 buffer.b611bb65c330a659948963b0c347028e5.log.meta
-rw-r--r--.  1 fluentd fluentd 105K Feb 19 13:40 buffer.b611bb65c3e396ec10b95cf905f58379e.log
-rw-r--r--.  1 fluentd fluentd  119 Feb 19 13:40 buffer.b611bb65c3e396ec10b95cf905f58379e.log.meta
-rw-r--r--.  1 fluentd fluentd 2.7K Feb 19 13:40 buffer.b611bb65c7ae2c47cdab5660771fab267.log
-rw-r--r--.  1 fluentd fluentd  133 Feb 19 13:40 buffer.b611bb65c7ae2c47cdab5660771fab267.log.meta
-rw-r--r--.  1 fluentd fluentd  50K Feb 19 13:40 buffer.b611bb65c92a90733913bdedeb967d70a.log
-rw-r--r--.  1 fluentd fluentd  106 Feb 19 13:40 buffer.b611bb65c92a90733913bdedeb967d70a.log.meta
-rw-r--r--.  1 fluentd fluentd 218K Feb 19 13:40 buffer.b611bb65d2b9fb9d5db3354398357d1b3.log
-rw-r--r--.  1 fluentd fluentd  119 Feb 19 13:40 buffer.b611bb65d2b9fb9d5db3354398357d1b3.log.meta
-rw-r--r--.  1 fluentd fluentd 1.3M Feb 19 13:40 buffer.b611bb65e0dc387e75349d529d6de4e78.log
-rw-r--r--.  1 fluentd fluentd  120 Feb 19 13:40 buffer.b611bb65e0dc387e75349d529d6de4e78.log.meta
-rw-r--r--.  1 fluentd fluentd  103 Feb 19 13:40 buffer.b611bb65e0e94baf6c91e6721c73e38ed.log
-rw-r--r--.  1 fluentd fluentd   90 Feb 19 13:40 buffer.b611bb65e0e94baf6c91e6721c73e38ed.log.meta
-rw-r--r--.  1 fluentd fluentd  25K Feb 19 13:40 buffer.b611bb65e10c5ed7aaaa4d9d20668435a.log
-rw-r--r--.  1 fluentd fluentd  118 Feb 19 13:40 buffer.b611bb65e10c5ed7aaaa4d9d20668435a.log.meta
-rw-r--r--.  1 fluentd fluentd 2.8K Feb 19 13:47 buffer.b611bb7f01abd9a6013d906fef94313fc.log
-rw-r--r--.  1 fluentd fluentd  127 Feb 19 13:47 buffer.b611bb7f01abd9a6013d906fef94313fc.log.meta
-rw-r--r--.  1 fluentd fluentd  716 Feb 19 13:47 buffer.b611bb7f025c0ba31b72c0ec65b9c3133.log
-rw-r--r--.  1 fluentd fluentd  116 Feb 19 13:47 buffer.b611bb7f025c0ba31b72c0ec65b9c3133.log.meta
-rw-r--r--.  1 fluentd fluentd 815K Feb 15 10:19 buffer.q6116821704a9b78d6d68e8af0d128480.log
-rw-r--r--.  1 fluentd fluentd  114 Feb 15 10:20 buffer.q6116821704a9b78d6d68e8af0d128480.log.meta
-rw-r--r--.  1 fluentd fluentd 4.6K Feb 15 10:19 buffer.q6116821766b5d914ad9169573628d406.log
-rw-r--r--.  1 fluentd fluentd  133 Feb 15 10:20 buffer.q6116821766b5d914ad9169573628d406.log.meta

To Reproduce

During a continuous stream of events (source doesn't matter), restart fluentd:

systemctl restart fluentd.service

Afterwards check the contents of the directory containing the buffer files.

Expected behavior

No files with modification time older than ~5 seconds (the flush interval) / ~1 minute (default resolution of directory listing) should be present in the buffer directories.

Your Environment

Fluentd version: fluent-package 5.0.1 fluentd 1.16.2 (d5685ada81ac89a35a79965f1e94bbe5952a5d3a)

Operating system:
NAME="Red Hat Enterprise Linux"
VERSION="8.9 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.9"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.9 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.9
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.9"

Kernel version: 4.18.0-513.11.1.el8_9.x86_64

Your Configuration

<system>
  workers 8
  root_dir /application/diagnostics/fluentd
</system>

<source>
  @type forward
</source>

### PROMETHEUS PLUGIN CONFIG

<filter *.**>
  @type prometheus
  <metric>
    name fluentd_input_status_num_records_total
    type counter
    desc The total number of incoming records
    <labels>
      tag ${tag}
      hostname ${hostname}
    </labels>
  </metric>
</filter>


# Everything will be saved according to its tag
<match dhc.**>
  @type copy
  @id buffer_dhc
  <store>
    @type prometheus
    <metric>
      name fluentd_output_status_num_records_total
      type counter
      desc The total number of outgoing records
      <labels>
        tag ${tag}
        hostname ${hostname}
      </labels>
    </metric>
  </store>
  <store>
   @type file
   path /application/diagnostics/${tag[1]}/${tag[3]}/${tag[1]}-%Y-%m-%d
   append true
   <buffer tag,time>
     chunk_limit_size 10MB
     total_limit_size 4096MB
     flush_mode interval
     flush_interval 5s
     flush_thread_count 8
     timekey 1d
   </buffer>
  </store>
</match>

<match *.**>
  @type copy
  @id buffer_non-dhc
  <store>
    @type prometheus
    <metric>
      name fluentd_output_status_num_records_total
      type counter
      desc The total number of outgoing records
      <labels>
        tag ${tag}
        hostname ${hostname}
      </labels>
    </metric>
  </store>
  <store>
   @type file
   path /application/diagnostics/${tag[0]}/${tag[2]}/${tag[3]}-%Y-%m-%d
   append true
   <buffer tag,time>
     chunk_limit_size 10MB
     total_limit_size 4096MB
     flush_mode interval
     flush_interval 5s
     flush_thread_count 8
     timekey 1d
   </buffer>
  </store>
</match>

### PROMETHEUS-PLUGIN OUTPUT
<source>
  @type prometheus
  bind 0.0.0.0
  port 9101
  metrics_path /metrics
  aggregated_metrics_path /aggregated_metrics
</source>
<source>
  @type prometheus_output_monitor
  interval 30
</source>

Your Error Log

No errors being present.

Additional context

No response

[daipom]

./worker0

Where is this directory?

[chrko]

./worker0

Where is this directory?

The buffer file directory structure is located in /application/diagnostics/${tag[0]}/${tag[2]}/${tag[3]}-%Y-%m-%d. This folder contains the folders worker[0-7].
The same pattern is present for the second path template /application/diagnostics/${tag[1]}/${tag[3]}/${tag[1]}-%Y-%m-%d.

[daipom]

Thanks for your report.
I found a bug.
When the buffer path includes [ or ], the resuming process can not find the remaining files.
This is caused by the lack of escapes.

fluentd/lib/fluent/plugin/buf_file.rb

Lines 139 to 146 in 2b4ca5d

	def resume
	stage = {}
	queue = []
	exist_broken_file = false
	patterns = [@path]
	patterns.unshift @additional_resume_path if @additional_resume_path
	Dir.glob(escaped_patterns(patterns)) do |path|

fluentd/lib/fluent/plugin/buf_file.rb

Lines 234 to 239 in 2b4ca5d

	def escaped_patterns(patterns)
	patterns.map { |pattern|
	# '{' '}' are special character in Dir.glob
	pattern.gsub(/[\{\}]/) { |c| "\\#{c}" }
	}
	end

[daipom]

You can avoid this problem by improving the setting.
Currently, you don't set path of buffer.
So, out_file sets the buffer path automatically to out_file's path.
You can set a more simple and safe path for buffer.

[chrko]

@daipom This sounds promising. I'll check out, but unfortunately we couldn't put the story on the topic into the current sprint 😞 I'll check in 2 weeks again!