Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs)

Engineering effort wise, what is the least effort? I'm thinking of starting with 1 and then follow up with 3. Or if 3 is as simple as one, directly jump to 3.

One thing we have to consider is also how much load we put on the cluster. I assume for the default of 24h it should be doable, but for longer periods, it could take a long time and make Elasticsearch read all the data.

Thanks for your input @ruflin, I have now another question: is there a way to move those indices that don't support the aggregation to a state where they support it?

is there a way to move those indices that don't support the aggregation to a state where they support it?

For data streams, my understanding is as soon as a rollover is triggered and the new index is created, it will be available on all the new data. @salvatore-campagna Can you confirm?

For data streams, my understanding is as soon as a rollover is triggered and the new index is created

We could guide users to do this, so next time they can benefit fully from the aggregation

@mdbirnstiehl could you help us here with some communications?

1. The request might take longer due to datastreams X, Y, Z not supporting _ignored aggregation
2. In order to support the aggregation for future data, and consequently improving performance in the page, we recommend users to perform manual rollovers to the aforementioned datastreams

@mdbirnstiehl could you help us here with some communications?

1. The request might take longer due to datastreams X, Y, Z not supporting _ignored aggregation
2. In order to support the aggregation for future data, and consequently improving performance in the page, we recommend users to perform manual rollovers to the aforementioned datastreams

How about:

Your request may take longer to complete due to datastreams X, Y, Z not supporting _ignored aggregation. Manually [rollover](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-rollover-index.html) these indices to prevent future delays.

My concern is whether or not "supporting ignored aggregation" is meaningful to users, but I'm not sure of a more meaningful way to put that without making the message too long.

Just my 2c here but I'd go with option 1 and warn users about partial results and encourage them to roll over the data stream. I'd rather not give users a foot-gun that can affect their cluster's performance. If we choose to still do a fallback query for indices that don't support aggregations for _ignored, I'd implement it as an opt-in, i.e. the user needs to click a button to trigger the slow aggregation.

Hey Felix, initially I though we would need to change the query to something like

"aggs": {
    "degraded": {
      "terms": {
        "field": "_ignored"
      }
    }
  }

With that type of query we will get failures and partial results, the thing is that query will only tell us how many documents we have containing a value of an specific _ignored property
a bucket will look like this in the response

{
   "key": {
     "dataset": "synth",
     "namespace": "default"
   },
   "doc_count": 34260,
   "degraded": {
     "doc_count_error_upper_bound": 0,
     "sum_other_doc_count": 0,
     "buckets": [
       {
         "key": "cloud.availability_zone",
         "doc_count": 3460
       },
       {
         "key": "log.level",
         "doc_count": 260
       }
     ]
   }
}

The problem with that type of response is that we are not able to tell how many documents in that dataset actually have _ignored properties, a document could have only cloud.availability_zone or only log.level or maybe cloud.availability_zone and log.level ignored.

Maybe I'm missing something but still I feel we would need to make use of exists as in the original query at least at this stage.

I think I've found a simple solution to get the ratio of degraded documents only for indices that have doc_values for the _ignored field. Instead of doing an exists query, we could use a missing aggregation like so:

{
  "size": 0,
  "aggs": {
    "non_degraded": {
      "missing": {
        "field": "_ignored"
      }
    }
  }
}

Based on some quick testing, the aggregation fails on shards where the field doesn't have doc_values but succeeds on other shards.

PUT /test-missing-agg
{
  "mappings": {
    "properties": {
      "ignored":{
        "type": "keyword",
        "doc_values": false,
        "store": true
      }
    }
  }
}

PUT /test-missing-agg2
{
  "mappings": {
    "properties": {
      "ignored":{
        "type": "keyword"
      }
    }
  }
}


POST test-missing-agg/_doc?refresh
Authorization: Basic {{username}} {{password}}
Content-Type: application/json
{
  "foo": "bar",
  "ignored": ["foo", "bar"]
}


POST test-missing-agg2/_doc?refresh
Authorization: Basic {{username}} {{password}}
Content-Type: application/json
{
  "foo": "bar",
  "ignored": ["foo", "bar"]
}

POST /test-missing-agg/_doc?refresh
{
  "foo": "bar"
}


POST /test-missing-agg2/_doc?refresh
{
  "foo": "bar"
}

POST /test-missing-agg/_search
{
  "size": 0,
  "aggs": {
    "non_degraded": {
      "missing": {
        "field": "ignored"
      }
    }
  }
}


### response

{
  "took": 6,
  "timed_out": false,
  "_shards": {
    "total": 2,
    "successful": 1,
    "skipped": 0,
    "failed": 1,
    "failures": [
      {
        "shard": 0,
        "index": "test-missing-agg",
        "node": "0G1taQ10S6KjBFKEnwdOkA",
        "reason": {
          "type": "illegal_argument_exception",
          "reason": "Can't load fielddata on [ignored] because fielddata is unsupported on fields of type [keyword]. Use doc values instead."
        }
      }
    ]
  },
  "hits": {
    "total": {
      "value": 2,
      "relation": "eq"
    },
    "max_score": null,
    "hits": []
  },
  "aggregations": {
    "non_degraded": {
      "doc_count": 1
    }
  }
}

Thanks @felixbarny, I did some quick testings in edge-lite.
The following tables will contain the queries used, the amount of time (ms) that took every run (from 1st to 10th), and lastly the avg of time took in ms

For 5d, including non aggregatable indices

*missing is not returning complete data, instead it returns failures for the indices not supporting the _ignored aggregation

For latest 24h, all indices supporting the aggregation

Avg in performance is very similar, do you think for higher amount of data we could conclude that using missing will be more beneficial?

I was also trying to split the query because I think that will speed things up for most of the cases (I expect total of documents with ignored property to be a reduced number compared to the rest) if we could request the queries in parallel

##### Number of documents with ignored property per dataset
GET /logs-*/_search
{
  "size": 0,
  "query": {
    "bool": {
      "filter": [
          {"range":
            {
              "@timestamp":{
                "gte": "now-24h",
                "lte": "now",
                "format":"epoch_millis"
              }
            }
          },
          {"term":{"data_stream.type":"logs"}
        }
      ],
      "must": {
        "exists": {
          "field": "_ignored"
        }
      }
    }
  },
  "aggs": {
    "datasets":{
      "composite": {
        "size":10000,
        "sources":[
          {"dataset":{"terms":{"field":"data_stream.dataset"}}},
          {"namespace":{"terms":{"field":"data_stream.namespace"}}}
        ]
      }
    }
  }
}

##### Total documents per dataset
GET /logs-*/_search
{
  "size": 0,
  "query": {
    "bool": {
      "filter": [
          {"range":
            {
              "@timestamp":{
                "gte": "now-24h",
                "lte": "now",
                "format":"epoch_millis"
              }
            }
          },
          {"term":{"data_stream.type":"logs"}
        }
      ]
    }
  },
  "aggs": {
    "datasets":{
      "composite": {
        "size":10000,
        "sources":[
          {"dataset":{"terms":{"field":"data_stream.dataset"}}},
          {"namespace":{"terms":{"field":"data_stream.namespace"}}}
        ]
      }
    }
  }
}

Doing that I got the following numbers

For 5d, including non aggregatable indices

For latest 24h, all indices supporting the aggregation

If we could do that, parallelise, we might lowering the time taken by elastic search to the ~half because the critical path will be on the longest query. wdyt?

I think we should run the query only for indices that have doc values on _ignored, discovering them using the field caps api. For other indices we can have the slow option to be triggered explicitly by the user.

@salvatore-campagna What you propose above adds a lot of complexity on the UI side for a "temporary" problem for our users. We opted to go for a banner on the top.