Found *.meta and *.log files at "/var/log/fluentd-buffers/kubernetes.system.buffer/" causing Old indices rediscover at OpenSearch

[amansharma846]

What is a problem?

Background: 
We are using fluentd and OpenSearch installed as a part of k8s setup.

Problem Statement:
- We have a cluster installed that includes fluentd and OpenSearch but we were not using or monitoring the logging stack for quite a long period. After that, found that the cluster was in a bad state, so we have redeployed the fluentd and OpenSearch. 
- After the redeployment, we have seen that the old indexes are getting created on OpenSearch and after deleting them using the DELETE command at OpenSearch Dev tools, they are re-appearing.
- Post analysis, we have found that the *.meta and *.log files of previous dates(months ago when the system was not in use) are present at "/var/log/fluentd-buffers/kubernetes.system.buffer/".
- So we suspect those files and delete them and old indices are not getting recreated and the system is running fine.

Note: After the redeployment, we have not seen any issue with the new data till now i.e *.log and *.meta files are getting deleted.

Queries:
- In what scenarios, the *.meta and *.log files are not getting deleted from the fluentd buffer?
- Is there any specific configuration that drive the deletion of *.meta and *.log files?

Describe the configuration of Fluentd

USER-SUPPLIED VALUES:
USER-SUPPLIED VALUES: null
fileConfigs:
  01_sources.conf: |-
    <source>
      @type tail
      @id in_tail_container_logs
      @label @KUBERNETES
      @log_level info
      path /var/log/containers/*.log
      pos_file /var/log/fluentd-containers.log.pos
      tag kubernetes.*
      read_from_head true
      <parse>
        @type multi_format
        <pattern>
          format json
          time_key time
          time_type string
          time_format "%Y-%m-%dT%H:%M:%S.%NZ"
          keep_time_key false
        </pattern>
        <pattern>
          format regexp
          expression /^(?<time>.+)(?<stream>stdout|stderr)( (.))? (?<log>.*)$/
          time_format '%Y-%m-%dT%H:%M:%S.%NZ'
          keep_time_key false
        </pattern>
      </parse>
      emit_unmatched_lines true
    </source>
    <source>
      @type prometheus
      bind 0.0.0.0
      port 24231
      metrics_path /metrics
    </source>
    <source>
      @type prometheus_output_monitor
      interval 10
     <labels>
       hostname ${hostname}
     </labels>
    </source>
  02_filters.conf: |-
    <label @KUBERNETES>
      <match kubernetes.var.log.containers.fluentd**>
        @type relabel
        @label @FLUENT_LOG
      </match>

      <filter kubernetes.**>
        @type kubernetes_metadata
        @id filter_kube_metadata
        skip_labels true
        skip_container_metadata false
        skip_namespace_metadata true
        skip_master_url true
      </filter>

      <filter kubernetes.**>
        @type record_transformer
        enable_ruby
        <record>
          log_size ${record.to_json.length}
        </record>
      </filter>

    <filter kubernetes.**>
      @id filter_parser
      @type parser
      key_name log
      reserve_time true
      reserve_data true
      remove_key_name_field true
      <parse>
        @type multi_format
        <pattern>
          format json
        </pattern>
        <pattern>
          format none
        </pattern>
      </parse>
    </filter>

      <match **>
        @type relabel
        @label @DISPATCH
      </match>
    </label>
  03_dispatch.conf: |-
    <label @DISPATCH>
      <filter **>
        @type prometheus
        <metric>
          name fluentd_input_status_num_records_total
          type counter
          desc The total number of incoming records
          <labels>
            tag ${tag}
            hostname ${hostname}
          </labels>
        </metric>
        <metric>
          name fluentd_input_byte_received_total
          type counter
          desc The total number of input bytes received
          key log_size
          <labels>
            tag ${tag}
            hostname ${hostname}
          </labels>
        </metric>
      </filter>
      <match **>
        @type relabel
        @label @OUTPUT
      </match>
    </label>
  04_outputs.conf: |-
    <label @OUTPUT>
      <match **>
        @type opensearch
        @log_level info
        host <hostname>
        port 9200
        suppress_type_name true
        logstash_format true
        logstash_dateformat '%Y.%m.%d'
        reload_connections false
        reload_on_failure false
        request_timeout 600s
        reconnect_on_error true
        scheme http
        ssl_verify true
        ssl_version TLSv1_2
        <buffer>
          @type "file"
          path "/var/log/fluentd-buffers/kubernetes.system.buffer"
          flush_mode "interval"
          retry_type "exponential_backoff"
          flush_thread_count "64"
          flush_interval "5s"
          retry_forever "true"
          retry_max_interval "60"
          chunk_limit_size "128M"
          total_limit_size "10GB"
          queue_limit_length "4"
          overflow_action "block"
        </buffer>
      </match>
    </label>
image:
  repository: <repo:port>/docker.io/fluent/fluentd-kubernetes-daemonset
  tag: v1.16.2-debian-opensearch-1.0
livenessProbe:
  initialDelaySeconds: 60
  timeoutSeconds: 10
readinessProbe:
  initialDelaySeconds: 60
  timeoutSeconds: 10
resources:
  limits:
    cpu: 100m
    memory: 1Gi
  requests:
    cpu: 100m
    memory: 1Gi
variant: opensearch

Describe the logs of Fluentd

No response

Environment

- Fluentd version: v1.16.2-debian-opensearch-1.0
- TD Agent version: 
- Fluent Package version: 
- Docker image (tag): <repo:port>/docker.io/fluent/fluentd-kubernetes-daemonset:v1.16.2-debian-opensearch-1.0
- Operating system: Oracle linux 9
- Kernel version: 5.15.0-203.146.5.1.el9uek.x86_64

[daipom]

• In what scenarios, the *.meta and *.log files are not getting deleted from the fluentd buffer?

There are 2 scenarios:

1. When stopping Fluentd, some buffer files remain and will be flushed after restarting Fluentd.
2. When some anomaly occurs.

• Is there any specific configuration that drive the deletion of *.meta and *.log files?

You can use flush_at_shutdown setting to flush all buffers before Fluentd stops.

[amansharma846]

Hi @daipom

1. I remember in my scenario, Fluentd was not working for 3 to 4 days but when it started, the buffer files were still there and not flushed.
2. Do we have any existing reported references to refer so that can get help in understanding?
3. Thanks for sharing the setting

What if the Fluentd crashes, what happens to the buffer files?

Thanks in advance

[amansharma846]

Hi @daipom .. a gentle reminder !!