[Q&A] DiskPressure due to 80 GB /home/fluentd/core. files

[Alisher-Nabiev]

What is a problem?

hi everyone,

i got on my eks fluentd that sents log to elastiserach 7.17.

after upgrade my cluster from 1.24 to 1.25 my nodes got DiskPressure due to 80 GB /home/fluentd/core.  files.

when the fluentd pod fills up the space of the node its evicted and the node in DiskPressure stat for 3-4 min after that its gone and fluentd pod runs again.

Describe the configuration of Fluentd

# AUTOMATICALLY GENERATED
# DO NOT EDIT THIS FILE DIRECTLY, USE /templates/conf/fluent.conf.erb   

@include "#{ENV['FLUENTD_SYSTEMD_CONF'] || 'systemd'}.conf"
@include "#{ENV['FLUENTD_PROMETHEUS_CONF'] || 'prometheus'}.conf"       
@include kubernetes.conf
@include conf.d/*.conf

<filter kubernetes.var.log.containers.**.log>
  @type parser
  key_name log
  reserve_data true
  reserve_time true
  hash_value_field ******-fields
  remove_key_name_field false
  replace_invalid_sequence false
  emit_invalid_record_to_error true
  <parse>
    @type json
  </parse>
</filter>

<filter kubernetes.var.log.containers.****.log kubernetes.var.log.containers.**.log kubernetes.var.log.containers.**.log kubernetes.var.log.containers.**.log>
  @type parser
  key_name log
  reserve_data true
  reserve_time true
  hash_value_field fields
  remove_key_name_field false
  replace_invalid_sequence false
  emit_invalid_record_to_error true
  <parse>
    @type json
  </parse>
</filter>

<label @ERROR>
  <match **>
    @type null
  </match>
</label>

<match kubernetes.var.log.containers.**.log kubernetes.var.log.containers**.log kubernetes.var.log.containers.**.log kubernetes.var.log.containers.**.log>
  @type elasticsearch
  @id out_es
  @log_level debug 
  include_tag_key true
  hosts "#{ENV['FLUENT_ELASTICSEARCH_HOST']}"
  port "#{ENV['FLUENT_ELASTICSEARCH_PORT']}"
  path "#{ENV['FLUENT_ELASTICSEARCH_PATH']}"
  scheme "#{ENV['FLUENT_ELASTICSEARCH_SCHEME'] || 'http'}"
  ssl_verify "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERIFY'] || 'true'}"     
  ssl_version "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERSION'] || 'TLSv1_2'}"
  user "#{ENV['FLUENT_ELASTICSEARCH_USER'] || use_default}"
  password "#{ENV['FLUENT_ELASTICSEARCH_PASSWORD'] || use_default}"
  reload_connections "#{ENV['FLUENT_ELASTICSEARCH_RELOAD_CONNECTIONS'] || 'false'}"
  reconnect_on_error "#{ENV['FLUENT_ELASTICSEARCH_RECONNECT_ON_ERROR'] || 'true'}"
  reload_on_failure "#{ENV['FLUENT_ELASTICSEARCH_RELOAD_ON_FAILURE'] || 'true'}"
  log_es_400_reason "#{ENV['FLUENT_ELASTICSEARCH_LOG_ES_400_REASON'] || 'false'}"
  logstash_prefix "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_PREFIX'] || 'logstash'}"
  logstash_dateformat "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_DATEFORMAT'] || '%Y.%m.%d'}"
  logstash_format "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_FORMAT'] || 'true'}"
  index_name "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_INDEX_NAME'] || 'logstash'}"
  target_index_key "#{ENV['FLUENT_ELASTICSEARCH_TARGET_INDEX_KEY'] || use_nil}"
  type_name "#{ENV['FLUENT_ELASTICSEARCH_LOGSTASH_TYPE_NAME'] || 'fluentd'}"
  include_timestamp "#{ENV['FLUENT_ELASTICSEARCH_INCLUDE_TIMESTAMP'] || 'false'}"
  template_name "#{ENV['FLUENT_ELASTICSEARCH_TEMPLATE_NAME'] || use_nil}"
  template_file "#{ENV['FLUENT_ELASTICSEARCH_TEMPLATE_FILE'] || use_nil}"
  template_overwrite "#{ENV['FLUENT_ELASTICSEARCH_TEMPLATE_OVERWRITE'] || use_default}"
  sniffer_class_name "#{ENV['FLUENT_SNIFFER_CLASS_NAME'] || 'Fluent::Plugin::ElasticsearchSimpleSniffer'}"
  request_timeout "#{ENV['FLUENT_ELASTICSEARCH_REQUEST_TIMEOUT'] || '5s'}"
  suppress_type_name "#{ENV['FLUENT_ELASTICSEARCH_SUPPRESS_TYPE_NAME'] || 'true'}"
  enable_ilm "#{ENV['FLUENT_ELASTICSEARCH_ENABLE_ILM'] || 'false'}"
  ilm_policy_id "#{ENV['FLUENT_ELASTICSEARCH_ILM_POLICY_ID'] || use_default}"
  ilm_policy "#{ENV['FLUENT_ELASTICSEARCH_ILM_POLICY'] || use_default}"
  ilm_policy_overwrite "#{ENV['FLUENT_ELASTICSEARCH_ILM_POLICY_OVERWRITE'] || 'false'}"
  <buffer>
    flush_thread_count "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_FLUSH_THREAD_COUNT'] || '8'}"
    flush_interval "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_FLUSH_INTERVAL'] || '5s'}"
    chunk_limit_size "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_CHUNK_LIMIT_SIZE'] || '10M'}"
    queue_limit_length "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_QUEUE_LIMIT_LENGTH'] || '64'}"
    retry_max_interval "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_RETRY_MAX_INTERVAL'] || '30'}"
    retry_forever true
  </buffer>
</match>

Describe the logs of Fluentd

fluentd.log

Environment

- Fluentd version: fluentd 1.16.3
- Fluent Package version: Debian GNU/Linux 12 (bookworm)
- Docker image (tag): fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch
- Kernel version: 5.10.209-198.812.amzn2.x86_64

[anumantha-sw]

We are also facing the same disk pressure issue in the EKS cluster. Attaching relevant logs from the fluentd containers and environment details.

df -lh
Filesystem Size Used Avail Use% Mounted on
overlay 128G 105G 24G 82% /
tmpfs 64M 0 64M 0% /dev
tmpfs 16G 0 16G 0% /sys/fs/cgroup
....
....

core files from the container

302M core.9443
326M core.9454
333M core.9506
317M core.9611
209M core.9634
325M core.9768
158M core.9823
302M core.9853
222M core.9904
pwd
/home/fluent

EKS Version: 1.25
Fluentd Version: v1.12

[Alisher-Nabiev]

@anumantha-sw
I resolved the issue by downgrading the version from 1.16 to 1.15

Docker image (tag): fluent/fluentd-kubernetes-daemonset:v1.15-debian-elasticsearch7

[daipom]

Is it possible to identify the files occupying disk space? Buffer files, etc.