I want to add the rollover index in fluentd config with index template #4472
Answered
by
ankit-groverDevops
ankit-groverDevops
asked this question in
Q&A
-
What is a problem?I want to add the index template while creating the index in elasticsearch using fluentd config with rollover on it ilm policy enabled as mentioned name Describe the configuration of Fluentd <source>
@type tail
@id tail_resumeparserLogs
tag resumeparser.*
path "/var/log/containers/rchilli-resumeparser-*.log"
pos_file "/var/log/fluentd-resumeparser.log.pos"
exclude_path ["/var/log/containers/fluentd-*.log"]
refresh_interval 30
read_from_head true
<parse>
@type none
</parse>
</source>
<match resumeparser**>
@type elasticsearch
@log_level debug
host "#{ENV['FLUENT_ELASTICSEARCH_HOST'] || 'elastic-cluster-es-http.elastic-system.svc.cluster.local'}"
port "#{ENV['FLUENT_ELASTICSEARCH_PORT'] || '9200'}"
scheme "#{ENV['FLUENT_ELASTICSEARCH_SCHEME'] || 'http'}"
ssl_verify "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERIFY'] || 'false'}"
user "#{ENV['FLUENT_ELASTICSEARCH_USER']}"
password "#{ENV['FLUENT_ELASTICSEARCH_PASSWORD']}"
reload_connections "#{ENV['FLUENT_ELASTICSEARCH_RELOAD_CONNECTIONS'] || 'true'}"
include_tag_key true
include_timestamp true
time_key time
index_name resumeparser
rollover_index true
application_name ${tag}
index_separator "."
index_date_pattern "now/d"
enable_ilm true
ilm_policy_id resumetest
ilm_policy {}
template_name fluentd
template_file /fluentd/etc/fluent-template.json
<buffer>
flush_thread_count "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_FLUSH_THREAD_COUNT'] || '8'}"
flush_interval "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_FLUSH_INTERVAL'] || '5s'}"
chunk_limit_size "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_CHUNK_LIMIT_SIZE'] || '2M'}"
queue_limit_length "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_QUEUE_LIMIT_LENGTH'] || '32'}"
retry_max_interval "#{ENV['FLUENT_ELASTICSEARCH_BUFFER_RETRY_MAX_INTERVAL'] || '30'}"
retry_forever true
</buffer>
</match>template.json {
"template": {
"settings": {
"index": {
"lifecycle": {
"name": "resumetest",
"rollover_alias": "resumeparser"
},
"number_of_shards": "1",
"number_of_replicas": "1"
}
},
"mappings": {
"properties": {
"timestamp": {
"type": "date"
}
}
},
"aliases": {
"resumeparser": {
"is_write_index": true
}
}
},
"index_patterns": [
"resumeparser*"
]
}ILM policy {
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {
"rollover": {
"max_age": "1m" // just for testing purpose
},
"set_priority": {
"priority": 100
}
}
},
"warm": {
"min_age": "8d",
"actions": {
"set_priority": {
"priority": 50
}
}
},
"cold": {
"min_age": "25d",
"actions": {
"set_priority": {
"priority": 0
}
}
},
"delete": {
"min_age": "30d",
"actions": {
"delete": {
"delete_searchable_snapshot": true
}
}
}
}
}
}Describe the logs of FluentdEnvironment- Fluentd version: starting fluentd-1.11.5 pid=7 ruby="2.6.6"
- TD Agent version:
- Fluent Package version:
- Docker image (tag): custom image with all major plugins
- Operating system: k8s
- Kernel version: |
Beta Was this translation helpful? Give feedback.
Answered by
ankit-groverDevops
Apr 29, 2024
Replies: 2 comments 2 replies
-
|
Your help is greatly appreciated please provide the solution |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
I got the config and it works for me Thankyou. |
Beta Was this translation helpful? Give feedback.
2 replies
-
|
I don't see any answers anywhere....link? |
Beta Was this translation helpful? Give feedback.
-
|
Sorry, I'm not familiar with Elasticsearch. But, there are many warning logs. |
Beta Was this translation helpful? Give feedback.
Answer selected by
ankit-groverDevops
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I got the config and it works for me Thankyou.