IndexFieldTypePollerPeriodical throws occasionally ResponseException

[user29835461]

Expected Behavior

No errors in the logs.

Current Behavior

This error shows occasionally in the logs. It refers to the Graylog Events index set.

2024-05-02T15:18:01.048+03:00 ERROR [IndexFieldTypePollerPeriodical] Uncaught exception in Periodical
org.graylog.shaded.opensearch2.org.opensearch.OpenSearchException: Couldn't get index list for index set <5d89b76d0a25285d26afd396>
        at org.graylog.storage.opensearch2.OpenSearchClient.exceptionFrom(OpenSearchClient.java:182) ~[?:?]
        at org.graylog.storage.opensearch2.OpenSearchClient.execute(OpenSearchClient.java:133) ~[?:?]
        at org.graylog.storage.opensearch2.cat.CatApi.perform(CatApi.java:105) ~[?:?]
        at org.graylog.storage.opensearch2.cat.CatApi.requestIndices(CatApi.java:101) ~[?:?]
        at org.graylog.storage.opensearch2.cat.CatApi.indices(CatApi.java:76) ~[?:?]
        at org.graylog.storage.opensearch2.cat.CatApi.indices(CatApi.java:71) ~[?:?]
        at org.graylog.storage.opensearch2.IndicesAdapterOS2.indices(IndicesAdapterOS2.java:414) ~[?:?]
        at org.graylog2.indexer.indices.Indices.getIndices(Indices.java:283) ~[graylog.jar:?]
        at org.graylog2.indexer.fieldtypes.IndexFieldTypePoller.poll(IndexFieldTypePoller.java:68) ~[graylog.jar:?]
        at org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical.lambda$refreshFieldTypes$2(IndexFieldTypePollerPeriodical.java:155) ~[graylog.jar:?]
        at java.lang.Iterable.forEach(Unknown Source) ~[?:?]
        at org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical.refreshFieldTypes(IndexFieldTypePollerPeriodical.java:144) ~[graylog.jar:?]
        at org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical.doRun(IndexFieldTypePollerPeriodical.java:131) ~[graylog.jar:?]
        at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:99) [graylog.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?]
        at java.util.concurrent.FutureTask.runAndReset(Unknown Source) [?:?]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
        at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: org.graylog.shaded.opensearch2.org.opensearch.client.ResponseException: method [GET], host [https://data1:9200], URI [/_cat/indices/gl-events_*?format=json&h=index%2Cstatus&s=index%2Cstatus&expand_wildcards=all], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Values less than -1 bytes are not supported: -1328b"}],"type":"illegal_argument_exception","reason":"Values less than -1 bytes are not supported: -1328b"},"status":400}
        at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient.convertResponse(RestClient.java:375) ~[?:?]
        at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient.performRequest(RestClient.java:345) ~[?:?]
        at org.graylog.shaded.opensearch2.org.opensearch.client.RestClient.performRequest(RestClient.java:320) ~[?:?]
        at org.graylog.storage.opensearch2.cat.CatApi.lambda$perform$4(CatApi.java:108) ~[?:?]
        at org.graylog.storage.opensearch2.OpenSearchClient.execute(OpenSearchClient.java:131) ~[?:?]
        ... 18 more

Possible Solution

No idea. The storage driver for Opensearch does something wrong?

Steps to Reproduce (for bugs)

It happens on its own.

The API from Opensearch seems to give proper information:

$ curl -k -X GET "https://admin:passwowrd@data1:9200/_cat/indices/gl-events_*?format=json&h=index%2Cstatus&s=index%2Cstatus&expand_wildcards=all&pretty"
[
  {
    "index" : "gl-events_10",
    "status" : "open"
  },
  {
    "index" : "gl-events_11",
    "status" : "open"
  },
  {
    "index" : "gl-events_12",
    "status" : "open"
  },
  {
    "index" : "gl-events_3",
    "status" : "open"
  },
  {
    "index" : "gl-events_4",
    "status" : "open"
  },
  {
    "index" : "gl-events_5",
    "status" : "open"
  },
  {
    "index" : "gl-events_6",
    "status" : "open"
  },
  {
    "index" : "gl-events_7",
    "status" : "open"
  },
  {
    "index" : "gl-events_8",
    "status" : "open"
  },
  {
    "index" : "gl-events_9",
    "status" : "open"
  }
]

Context

I have no idea what this might cause.

Your Environment

• Graylog Version: 5.2.6
• Java Version: Bundled 17.x
• OpenSearch Version: 2.13
• MongoDB Version: 5.0.26
• Operating System: RHEL8
• Browser version: n/a

[dennisoelkers]

Hey @user29835461!

Thanks for reporting this and providing helpful information! My current impression is that it might be related to OpenSearch 2.13 not being supported completely by the 2.9 client library included in 5.2.6. 6.0 will include a more recent client library that could potentially fix this issue. In the meantime I will check if I can reproduce the issue locally.