json.error name collision between UI and Storage/Access

[terrywbrady]

No description provided.

[terrywbrady]

OpenSearch is rejecting records containing error messages due to a datatype conflict

Logstash error

[WARN ][logstash.outputs.opensearch][opensearch-uc3][891ea1d6949ab94e687c284e33501033384175ae642113b316c5b44ea6fdd65e] Could not index event to OpenSearch. ..., "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [json.error] tried to parse field [error] as object, but found a concrete value"}}}}

Storage treats json.error as a hash.

    "json": {
      "log.logger": "org.cdlib.mrt.s3.aws.AWSS3Cloud",
      "ecs.version": "1.2.0",
      "log": {
        "origin": {
          "file": {
            "name": "AWSS3Cloud.java",
            "line": 522
          },
          "function": "putObject"
        }
      },
      "service.node.name": "service",
      "message": "putObject ex:org.cdlib.mrt.s3.aws.AWSS3Cloud-putObject:442 REMOTE_IO_SERVICE_EXCEPTION[Attempt to read or write to remote IO service fails] AWS service exception:java.lang.InterruptedException - ...,
      "error.type": "org.cdlib.mrt.utility.TException$REMOTE_IO_SERVICE_EXCEPTION",
      "@timestamp": "2024-05-15T11:17:55.647Z",
      "process.thread.name": "pool-30826-thread-1",
      "log.level": "ERROR",
      "error.stack_trace": "org.cdlib.mrt.s3.aws.AWSS3Cloud-putObject:442 REMOTE_IO_SERVICE_EXCEPTION[Attempt to read or write to remote IO service fails] AWS service exception:java.lang.InterruptedException - ...\n\tat org.cdlib.mrt.s3.aws.AWSS3Cloud.putObject(AWSS3Cloud.java:442)\n\tat org.cdlib.mrt.s3.aws.AWSS3Cloud.putObject(AWSS3Cloud.java:538)\n\tat org.cdlib.mrt.cloud.object.StateHandler.add(StateHandler.java:310)\n\tat org.cdlib.mrt.cloud.object.StateHandler.process(StateHandler.java:133)\n\tat org.cdlib.mrt.s3.aws.AWSS3Cloud.getState(AWSS3Cloud.java:1646)\n\tat org.cdlib.mrt.s3.service.NodeIOStatus$Task.run(NodeIOStatus.java:222)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\n",
      "service.name": "service",
      "event.dataset": "tomcat"
    },

UI treats json.error as a string

{"method":"GET","path":"/m/ark%3A%2F13030%2Fqt0r21m045","format":"*/*","controller":"ObjectController","action":"index","status":500,"error":"MerrittRetryMixin::RetryException: undefined method `first' for nil:NilClass","allocations":13099,"duration":3009.13,"view":0.0,"db":0.24,"time":680350.212300001,"@timestamp":"2024-05-15T19:31:25.524Z","@version":"1","message":"[500] GET /m/ark%3A%2F13030%2Fqt0r21m045 (ObjectController#index)"}

[elopatin-uc3]

Rather than implementing a service fix, we're planning on approaching this via two separate OpenSearch indices.