Plan for elasticsearch_dsl update for ES 8

[obulat]

We have migrated from ES 7 to ES 8 recently. While everything seems to be working now (after some wrinkles of migration were fixed), we have one dependency that does not support ES 8: elasticsearch_dsl.

We use elasticsearch_dsl to create the search queries using high-level methods. This is an official library from Elasticsearch, but it wasn't updated in a long time, and it does not yet support ES 8: When will there be a release for 8.0.0?
#1569. There are comments in this thread that the library works with ES 7, it's still not safe to keep using the non-supported version.

There are several links from other libraries on how they approached updating to ES 8 with the unsupported dependency.
One of them has decided to refactor the code to use a more low-level elasticsearch library to create the queries.

Possible improvements if manually writing the queries

I've looked into the queries that we currently create, and it seems that our queries are much more nested than they need to be.
The snippet below contains the queries that we currently create using elasticsearch_dsl:

Very nested query

{ "from": 0, "highlight": { "fields": { "description": {}, "tags.name": {}, "title": {} }, "order": "score" }, "query": { "bool": { "must": [ { "bool": { "must": [ { "bool": { "filter": [ { "bool": { "should": [ { "terms": { "extension": [ "png" ] } } ] } }, { "bool": { "should": [ { "terms": { "source": [ "" ] } } ] } }, { "bool": { "must_not": [ { "term": { "mature": "True" } } ] } }, { "bool": { "must_not": [ { "terms": { "provider": [] } } ] } } ], "must": [ { "simple_query_string": { "default_operator": "AND", "fields": [ "tags.name", "title", "description" ], "query": "cat" } } ] } } ], "should": [ { "simple_query_string": { "boost": 10000, "fields": [ "title" ], "query": "cat" } } ] } } ], "should": [ { "rank_feature": { "boost": 10000, "field": "standardized_popularity"}}]}} }

We can simplify this query by removing unnecessary nested bool queries and merging the filter conditions into the top level "filter" clause to reduce the complexity. If we moved to creating the queries manually, I think it would be easier to make it less nested and possibly more efficient.

Additional information

I discovered this problem when I was researching how to create the ES query for Additional search views.

[AetherUnbound]

FWIW it looks like there may be a version with support for ES8 out! elastic/elasticsearch-dsl-py#1569 (comment)

[obulat]

Yes! And we've migrated to it in #3029. Sorry for not closing the discussion in time.