[Proposal] : Filtering system

Overview

The goal of this improvement is to improve MapStore filter format to make it:

more extensible
more consistent
easy to use in other contexts

Proposed By

Lorenzo Natali

Assigned to Release

The proposal is for 2023.01.00 .

State

Motivation

The current filter format is not extensible and not easy to read. It is bind to the current implementation of the query panel and it is not easy to use it in other contexts.

Proposal

State of the art

The current implementation of the filter format is based on the following assumptions:

the current filter format is actually not properly documented. It has this shape.

"layerFilter": {
    "searchUrl": null,
    "featureTypeConfigUrl": null,
    "showGeneratedFilter": false,
    "attributePanelExpanded": true,
    "spatialPanelExpanded": true,
    "crossLayerExpanded": true,
    "showDetailsPanel": false,
    "groupLevels": 5,
    "useMapProjection": false,
    "toolbarEnabled": true,
    "groupFields": [
        // ..
    ],
    "maxFeaturesWPS": 5,
    "filterFields": [
        // ...
    ],
    "spatialField": {
        // ...
    },
    "simpleFilterFields": [],
    "crossLayerFilter": {
      // ...
    },
    "autocompleteEnabled": true
},

As you can see this format is strongly shaped on the Query Panel implementation. Here the limitations:

attributePanelExpanded, spatialPanelExpanded, crossLayerExpanded, showDetailsPanel, groupLevels, useMapProjection, toolbarEnabled, maxFeaturesWPS, simpleFilterFields, autocompleteEnabled are not related to the filter itself but to the UI. They should be moved to the UI state.
searchUrl, featureTypeConfigUrl, showGeneratedFilter are not related to the filter itself but to the data source. They should be moved to the data source configuration.
attributeFields and spatialField are the only fields that are related to the filter itself. Anyway they are reserved to the Query Panel implementation. You can not use them in to store information inside them or they will be used in the Query Panel too.
spatialField is single and it is not possible to have more than one spatial filter. This is not enough to support the use cases described in the motivation section. We mitigated this limitation supporting multiple spatial filters as an array, anyway this breaks the query panel.
groupFields is stored in a separate field. This structure makes the filter not readable and conversions very hard.
crossLayerFilter is stored in a separate field. It represents a function and it stores the function parameters (target layer and a list of attribute, operator, value). This structure makes the filter not readable and conversions very hard.

 {
    "id": 1,
    "logic": "OR",
    "index": 0
},
{
    "id": 1671785737915,
    "logic": "OR",
    "groupId": 1,
    "index": 1
}

In fact this format is very hard to understand and handle.

A newer attempt to improve the filtering capabilities of MapStore has been done with the introduction of FilterBuilder. This object allow to create and combine programmatically OGC filters in this way.

const fb = filterBuilder({ gmlVersion: "3.1.1" });
const {filter, property, and} = fb;
[toFilter(read(cqlFilter))]
const ogcFilter = filter(and(
      ...(layerFilter  && !layerFilter.disabled ? toOGCFilterParts(layerFilter, "1.1.0", "ogc") : []),
      ...(newFilterObj ? toOGCFilterParts(newFilterObj, "1.1.0", "ogc") : []),
      property(geomProp).intersects(geom)))

Basically:

the filterBuilder is used to create the a set of functions to create OGC filters, with the passed parameters (namespace, gmlVersion, srsName...)
code like property(geomProp).intersects(geom) is used to create a OGC filter part (XML string blocks)
toOGCFilterParts is used to convert the a filter object, the one based on query builder, to OGC filter parts (XML string blocks that can be added to the filter object)

All these techniques are used to create and combine filters in MapStore. This is a very complex and hard to understand approach, inherited from the implementation of the Query Panel.

With the proposed improvement we want to simplify the filter format and to make it more extensible, keeping it backward compatible with the current implementation and saved filters.

General idea behind the suggested improvement

The general idea behind the suggested improvement is to simplify the filter format and to make it more extensible, keeping it backward compatible with the current implementation and saved filters.

To do it we will keep all the codebase initially the same, with the addition to the current filter format a new field filters that can be used to append new filters (satisfying the new requirements) and that can be used in the future to store also, in a more readable way, the current filters.

Proposed filter format

The proposed filter format is based on the following assumptions:

the filter format should be extensible. It should be possible to add new fields without breaking the current implementation.
the filter format should be easy to read and to convert to other formats.
the filter format should be easy to use in other contexts.
the filter format should be easy to use in the Query Panel.
the filter format should allow a progressive migration of the current implementation.

In order to support the above assumptions the proposed filter format should have :

a type field to identify the filter format. This field should be used to identify the filter format and to convert it to the current format.
a version field to identify the version of the filter format. This field should be used to identify the version of the filter format and to convert it to the current format.
the current format, that doesn't have a filterFormat and a filterVersion field, should be identified as mapstore and 1.0.0.
a new filters field that should contain the list of filters. This field should be used to store the additional filters to append to the ones already supported by the Query Panel.
In the future we can move attributeFields and spatialField to the filters field. This will allow to have a more generic filter format.

{
  "filterFormat": "mapstore",
  "filterVersion": "1.0.0",
  "attributeFields": [],
  "groupFields": [],
  // ...
  "filters": []
}

(future versions of mapstore filter format may deprecate attributeFields and spatialField to use only the filters field.)

We have now to define a new generic filter format to store in the filters array. In order to reuse the current local representations in MapStore (e.g. MapStore is able to parse cqlFilter and convert them into a JSON structure similar to the OpenLayers 2 format) and to support future extensions like cql2-json, I suggest to store in this filters array a list of objects with the following structure:

{
  "id": "filter-id",
  "filterFormat": "cql2-json",
  "filterVersion": "1.0.0",
  "filter": {
    // ...
  }
}

This will allow to store in the filters array a list of filters with different formats. This will allow to support the use cases described in the motivation section. Moreover the presence of the id field will allow to identify the filters and to update them in it.

Notice that filters array can also contain the mapstore format.

{
  "filterFormat": "mapstore",
  "filterVersion": "1.0.0",
  "attributeFields": [],
  "groupFields": [],
  // ...
  "filters": [
    {
      "id": "my-custom-filter",
      "filterFormat": "mapstore",
      "filterVersion": "1.0.0",
      "attributeFields": [],
      "groupFields": [],
      "filters": []
      // ...
    }
  ]
}

In fact mapstore format, in its 1.0.0 version, is backward compatible with the current format, and is moreover a container for other filters of any type.

We must guarantee for every filterFormat and filterVersion that the conversion at least to the OGC Filter 1.1.0 format and CQL/ECQL 1.1.0 format is possible.

Required changes and Estimation

To provide this initial improvement the changes are a few:

Add to the functions that generates CQL/OGC from filter object, the capability to parse the additional filters part generating and joining these filters to the existing ones (initially the implicit mapstore format is used).
Some changes to query panel to accept changes to the filters array as changes to apply again (to customize the query panel components)
Support for CQL and mapstore formats in filters

Estimated time: 2-3 days of work. This can be considered to be partially absorbed by estimations for https://github.com/geosolutions-it/npa-cgg/issues/291 https://github.com/geosolutions-it/npa-cgg/issues/292 and https://github.com/geosolutions-it/npa-cgg/issues/293

Anyway, other solutions may take more in terms of time and may partially degradate the functionalities of the application. So if this or only solution is not accepted, the estimation can only grow.

Future evolutions

Formats to support

In order to provide an initial support for the required features, we will reuse the mapstore format to store the additional information.

Anyway we can start thinking to support also other formats.

OL 2 JSON format

We have a partial implementation of a JSON format used as intermediate format to parse cql filters into OGC ones. It is inspired to a format used in OpenLayers 2, so I should name it ol2 format. The format is described in the MapStore documentation.

This format already provides a filterBuilder tool that can be used to convert it into ogc format.

CQL2 JSON format

Another format is cql2-json format that is in proposed in this draft CQL2 JSON format documentation;

example:

{
    "op": "and",
    "args": [
      {
        "op": "=",
        "args": [ { "property": "collection" }, "landsat8_l1tp" ]
      }, {
        "op": ">=",
        "args": [ { "property": "datetime" }, "2021-04-08T04:39:23Z" ]
      },
      {
        "op" : "in",
        "args" : [ { "property" : "prop-name" }, [ 123, 456 ] ]
      }
    ]
}

We should have so a filter like this

{
  "id": "my-custom-filter",
  "filterFormat": "cql2-json",
  "filterVersion": "1.0.0",
  "filter": {
    "op": "and",
    "args": [
      {
        "op": "=",
        "args": [ { "property": "collection" }, "landsat8_l1tp" ]
      }, {
        "op": ">=",
        "args": [ { "property": "datetime" }, "2021-04-08T04:39:23Z" ]
      },
      {
        "op" : "in",
        "args" : [ { "property" : "prop-name" }, [ 123, 456 ] ]
      }
    ]
  }
}

Clean up query panel filter format

In the future we can move attributeFields and spatialField to the filters field. This will allow to have a more generic filter format.

{
  "filterFormat": "mapstore",
  "filterVersion": "1.0.0",
  "filters": [
    {
      "id": "query-panel-filter",
      "filterFormat": "mapstore",
      "filterVersion": "1.0.0",
      "attributeFields": [],
      "groupFields": [],
      "filters": []
      // ...
    }
  ]
}

And maybe convert it in a more standard format like cql2-json, keeping the UI info in a separate field.

{
  "filterFormat": "mapstore",
  "filterVersion": "1.0.0",
  "filters": [
    {
      "id": "query-panel-filter",
      "filterFormat": "cql2-json",
      "filterVersion": "1.0.0",
      "filter": {
        // ...
      },
      "ui": {
        "attributeFields": [],
        "groupFields": [],
        // ...
      }
    }
  ]
}

Appendix

Full example of the current filter

"layerFilter": {
    "searchUrl": null,
    "featureTypeConfigUrl": null,
    "showGeneratedFilter": false,
    "attributePanelExpanded": true,
    "spatialPanelExpanded": true,
    "crossLayerExpanded": true,
    "showDetailsPanel": false,
    "groupLevels": 5,
    "useMapProjection": false,
    "toolbarEnabled": true,
    "groupFields": [
        {
            "id": 1,
            "logic": "OR",
            "index": 0
        },
        {
            "id": 1671785737915,
            "logic": "OR",
            "groupId": 1,
            "index": 1
        }
    ],
    "maxFeaturesWPS": 5,
    "filterFields": [
        {
            "rowId": 1671785736331,
            "groupId": 1,
            "attribute": "LAND_KM",
            "operator": ">",
            "value": 1000000,
            "type": "number",
            "fieldOptions": {
                "valuesCount": 0,
                "currentPage": 1
            },
            "exception": null
        },
        {
            "rowId": 1671785739355,
            "groupId": 1671785737915,
            "attribute": "STATE_NAME",
            "operator": "=",
            "value": "Alabama",
            "type": "string",
            "fieldOptions": {
                "valuesCount": 0,
                "currentPage": 1
            },
            "exception": null,
            "loading": false,
            "options": {
                "STATE_NAME": []
            },
            "openAutocompleteMenu": false
        },
        {
            "rowId": 1671785746696,
            "groupId": 1671785737915,
            "attribute": "STATE_NAME",
            "operator": "=",
            "value": "Arizona",
            "type": "string",
            "fieldOptions": {
                "valuesCount": 0,
                "currentPage": 1
            },
            "exception": null,
            "loading": false,
            "options": {
                "STATE_NAME": []
            },
            "openAutocompleteMenu": false
        }
    ],
    "spatialField": {
        "method": "BBOX",
        "operation": "INTERSECTS",
        "geometry": {
            "id": "aefadb00-829f-11ed-b555-8bd9209cf0fa",
            "type": "Polygon",
            "extent": [
                -13188750.608437454,
                3135752.6483710706,
                -8795761.718831802,
                4671831.168789972
            ],
            "center": [
                -10992256.163634628,
                3903791.908580521
            ],
            "coordinates": [
                [
                    [
                        -13188750.608437454,
                        4671831.168789972
                    ],
                    [
                        -13188750.608437454,
                        3135752.6483710706
                    ],
                    [
                        -8795761.718831802,
                        3135752.6483710706
                    ],
                    [
                        -8795761.718831802,
                        4671831.168789972
                    ],
                    [
                        -13188750.608437454,
                        4671831.168789972
                    ]
                ]
            ],
            "style": {},
            "projection": "EPSG:3857"
        },
        "attribute": "the_geom"
    },
    "simpleFilterFields": [],
    "crossLayerFilter": {
        "attribute": "the_geom",
        "collectGeometries": {
            "queryCollection": {
                "typeName": "gs:us_states",
                "filterFields": [
                    {
                        "rowId": 1671785795624,
                        "groupId": 1,
                        "attribute": "STATE_NAME",
                        "operator": "=",
                        "value": "Alabama",
                        "type": "string",
                        "fieldOptions": {
                            "valuesCount": 0,
                            "currentPage": 1
                        },
                        "exception": null,
                        "loading": false,
                        "openAutocompleteMenu": false,
                        "options": {
                            "STATE_NAME": []
                        }
                    },
                    {
                        "rowId": 1671785801840,
                        "groupId": 1,
                        "attribute": "STATE_NAME",
                        "operator": "=",
                        "value": "Arizona",
                        "type": "string",
                        "fieldOptions": {
                            "valuesCount": 0,
                            "currentPage": 1
                        },
                        "exception": null,
                        "loading": false,
                        "openAutocompleteMenu": false,
                        "options": {
                            "STATE_NAME": []
                        }
                    }
                ],
                "geometryName": "the_geom",
                "groupFields": [
                    {
                        "id": 1,
                        "index": 0,
                        "logic": "OR"
                    }
                ]
            }
        },
        "operation": "INTERSECTS"
    },
    "autocompleteEnabled": true
}

Provide feedback

Saved searches

Use saved searches to filter your results more quickly