Data stream ingestion throws 400 - Rejected by OpenSearch

[casabre]

(check apply)

• read the contribution guideline
• (optional) already reported 3rd party upstream repository or mailing list if you use k8s addon or helm charts.

Steps to replicate

<source>
  @type forward
</source>

<match **>
  @type opensearch
  host os-http
  port 9200
  scheme http
  user "#{ENV['OS_USERNAME']}"
  password "#{ENV['OS_PASSWORD']}"
  index_name ${tag}
  include_timestamp true
  <buffer tag, time>
    @type file
    path /tmp/log/fluent/buffer_${tag}
    timekey 3600
    flush_mode interval
    flush_interval 5
  </buffer>
</match>

<match my_datastream**>
  @type opensearch_data_stream
  data_stream_name my_datastream
  host os-http
  port 9200
  scheme http
  user "#{ENV['OS_USERNAME']}"
  password "#{ENV['OS_PASSWORD']}"
  include_timestamp true
  <buffer tag, time>
    @type file
    path /tmp/log/fluent/buffer
    timekey 3600
    flush_mode interval
    flush_interval 5
  </buffer>
</match>

Data stream shows following error when ingesting at Opensearch.

2022-07-01 09:14:21 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::OpenSearchErrorHandler::OpenSearchError error="400 - Rejected by OpenSearch" location=nil tag="my_datastream" time=2022-07-01 09:04:38.007341687 +0000 record={"cpu_p"=>43.5, "user_p"=>40.0, "system_p"=>3.5000000000000004, "cpu0.p_cpu"=>53.0, "cpu0.p_user"=>50.0, "cpu0.p_system"=>3.0, "cpu1.p_cpu"=>35.0, "cpu1.p_user"=>30.0, "cpu1.p_system"=>5.0}

Expected Behavior or What you need to ask

How should the data stream setup look like in order to ingest data successfully? Is this more a Opensearch configuration problem or related to the plugin?

The first non-datastream part creates and ingests data the right way.

Using Fluentd and OpenSearch plugin versions

• OS version
  Docker image: docker.io/bitnami/fluentd:1.15.0-debian-11-r1

• Helm chart
  https://artifacthub.io/packages/helm/bitnami/fluentd

• Bare Metal or within Docker or Kubernetes or others?
  Openshift 4.x

• Fluentd v1.0 or later
  fluentd 1.15.0

• OpenSearch plugin version
  *** LOCAL GEMS ***

abbrev (default: 0.1.0)
activesupport (7.0.3)
addressable (2.8.0)
aws-eventstream (1.2.0)
aws-partitions (1.601.0)
aws-sdk-core (3.131.2)
aws-sdk-kms (1.57.0)
aws-sdk-s3 (1.114.0)
aws-sdk-sqs (1.51.1)
aws-sigv4 (1.5.0)
base64 (default: 0.1.1)
benchmark (default: 0.2.0)
bigdecimal (default: 3.1.1)
bundler (2.3.16, 2.3.13)
cgi (default: 0.3.1)
concurrent-ruby (1.1.10)
cool.io (1.7.1)
csv (default: 3.2.2)
date (default: 3.2.2)
debug (1.4.0)
delegate (default: 0.2.0)
did_you_mean (default: 1.6.1)
digest (default: 3.1.0)
digest-crc (0.6.4)
domain_name (0.5.20190701)
drb (default: 2.1.0)
elastic-transport (8.0.1)
elasticsearch (8.3.0)
elasticsearch-api (8.3.0)
elasticsearch-xpack (7.17.1)
english (default: 0.7.1)
erb (default: 2.2.3)
error_highlight (default: 0.3.0)
etc (default: 1.3.0)
excon (0.92.3)
faraday (1.10.0)
faraday-em_http (1.0.0)
faraday-em_synchrony (1.0.0)
faraday-excon (1.1.0)
faraday-httpclient (1.0.1)
faraday-multipart (1.0.4)
faraday-net_http (1.0.1)
faraday-net_http_persistent (1.2.0)
faraday-patron (1.0.0)
faraday-rack (1.0.0)
faraday-retry (1.0.3)
faraday_middleware-aws-sigv4 (0.6.1)
fcntl (default: 1.0.1)
ffi (1.15.5)
ffi-compiler (1.0.1)
fiddle (default: 1.1.0)
fileutils (default: 1.6.0)
find (default: 0.1.1)
fluent-config-regexp-type (1.0.0)
fluent-plugin-concat (2.5.0)
fluent-plugin-detect-exceptions (0.0.14)
fluent-plugin-elasticsearch (5.2.3)
fluent-plugin-grafana-loki (1.2.18)
fluent-plugin-kafka (0.17.5)
fluent-plugin-kubernetes_metadata_filter (2.11.1)
fluent-plugin-multi-format-parser (1.0.0)
fluent-plugin-opensearch (1.0.7)
fluent-plugin-prometheus (2.0.3)
fluent-plugin-record-modifier (2.1.0)
fluent-plugin-rewrite-tag-filter (2.4.0)
fluent-plugin-s3 (1.7.0)
fluent-plugin-systemd (1.0.5)
fluentd (1.15.0, 1.14.6)
forwardable (default: 1.3.2)
getoptlong (default: 0.1.1)
http (4.4.1)
http-accept (1.7.0)
http-cookie (1.0.5)
http-form_data (2.3.0)
http-parser (1.2.3)
http_parser.rb (0.8.0)
i18n (1.10.0)
io-console (default: 0.5.11)
io-nonblock (default: 0.1.0)
io-wait (default: 0.2.1)
ipaddr (default: 1.2.4)
irb (default: 1.4.1)
jmespath (1.6.1)
json (default: 2.6.1, 2.1.0)
jsonpath (1.1.2)
kubeclient (4.9.3)
logger (default: 1.5.0)
lru_redux (1.1.0)
ltsv (0.1.2)
matrix (0.4.2)
mime-types (3.4.1)
mime-types-data (3.2022.0105)
minitest (5.16.1, 5.15.0)
msgpack (1.5.2)
multi_json (1.15.0)
multipart-post (2.2.3)
mutex_m (default: 0.1.1)
net-ftp (0.1.3)
net-http (default: 0.2.0)
net-imap (0.2.3)
net-pop (0.1.1)
net-protocol (default: 0.1.2)
net-smtp (0.3.1)
netrc (0.11.0)
nkf (default: 0.1.1)
observer (default: 0.1.1)
oj (3.3.10)
open-uri (default: 0.2.0)
open3 (default: 0.1.1)
opensearch-api (2.0.2)
opensearch-ruby (2.0.2)
opensearch-transport (2.0.0)
openssl (default: 3.0.0)
optparse (default: 0.2.0)
ostruct (default: 0.5.2)
pathname (default: 0.2.0)
power_assert (2.0.1)
pp (default: 0.3.0)
prettyprint (default: 0.1.1)
prime (0.1.2)
prometheus-client (4.0.0)
pstore (default: 0.1.1)
psych (default: 4.0.3)
public_suffix (4.0.7)
racc (default: 1.6.0)
rake (13.0.6)
rbs (2.1.0)
rdoc (default: 6.4.0)
readline (default: 0.0.3)
readline-ext (default: 0.1.4)
recursive-open-struct (1.1.3)
reline (default: 0.3.0)
resolv (default: 0.2.1)
resolv-replace (default: 0.1.0)
rest-client (2.1.0)
rexml (3.2.5)
rinda (default: 0.1.1)
rss (0.2.9)
ruby-kafka (1.5.0)
ruby2_keywords (0.0.5)
rubygems-update (3.3.13)
securerandom (default: 0.1.1)
serverengine (2.3.0)
set (default: 1.0.2)
shellwords (default: 0.1.0)
sigdump (0.2.4)
singleton (default: 0.1.1)
stringio (default: 3.0.1)
strptime (0.2.5)
strscan (default: 3.0.1)
syslog (default: 0.1.0)
systemd-journal (1.4.2)
tempfile (default: 0.1.2)
test-unit (3.5.3)
time (default: 0.2.0)
timeout (default: 0.2.0)
tmpdir (default: 0.1.2)
tsort (default: 0.1.0)
typeprof (0.21.2)
tzinfo (2.0.4)
tzinfo-data (1.2022.1)
un (default: 0.2.0)
unf (0.1.4)
unf_ext (0.0.8.2)
uri (default: 0.11.0)
weakref (default: 0.1.1)
webrick (1.7.0)
yajl-ruby (1.4.3)
yaml (default: 0.2.0)
zlib (default: 2.1.1)

• OpenSearch version
  1.3.0

• OpenSearch template(s) (optional)

[toby181]

Did you already try with enabling this parameter: https://github.com/fluent/fluent-plugin-opensearch#log_os_400_reason?
os_400 is in my cases a mapping conflict between the data type that is send and was OS expects.

[casabre]

@toby181 thanks for the hint 😄. I will check that log_os_400_reason flag. It could be actually the the mapping because I didn't set it for the trial run.

[leowinterde]

Relates #82