Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to use templates option in latest version of td-agent along with opensearch plugin. #91

Open
vikranth06 opened this issue Jan 16, 2023 · 0 comments

Comments

@vikranth06
Copy link

vikranth06 commented Jan 16, 2023

Steps to replicate

We are using the latest version of td-agent along with Opensearch from AWS but as soon as we put templates section in the config file following error message is thrown "2023-01-16 13:47:39 +0000 [error]: #0 fluent/log.rb:372:error: unexpected error error_class=ArgumentError error="expected both :access_key_id and :secret_access_key options""

Expected Behavior or What you need to ask

As per the documentation templates is supported. Can the below config file be reviewed and let us know if there is a mistake?
...

Config file

<match **.*******.UPLOAD>
  @type copy
  <store>
    @type opensearch
    index_name xx-xxxx-xxxxx-data
    type_name "_doc"
    logstash_format true
    logstash_prefix xx-xxxx-xxxxx-data
    logstash_dateformat %Y.%m
    template_overwrite true
    templates {"xxxxxxxxxxxxxx_template": "/opt/aws/custom/xxxxx_index_template.json"}
    include_tag_key false
    buffer_type file
    buffer_chunk_limit 2m
    buffer_queue_limit 1000
    buffer_path /var/log/td-agent/uploadreportsupesbuffer
    disable_retry_limit true
    flush_interval 60s
    retry_wait 10s
    id_key MessageId
    reload_connections false
    remove_keys MessageId
    <endpoint>
      url xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
      region us-east-1
      access_key_id xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
      secret_access_key xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
	  </endpoint>
  </store>
</match>

Using Fluentd and OpenSearch plugin versions

  • OS version - Ubuntu 22.04
  • Bare Metal or within Docker or Kubernetes or others?
  • Fluentd v1.0 or later
    • td-agent 4.4.2 fluentd 1.15.3
  • OpenSearch plugin version
    • paste boot log of fluentd or td-agent
    • paste result of fluent-gem list, td-agent-gem list or your Gemfile.lock
  • OpenSearch version (optional)
  • OpenSearch template(s) (optional)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant