SNO-172-upgrade-to-elasticsearch7 #296

Bek · 2020-10-05T19:09:33Z

No description provided.

src/snovault/elasticsearch/create_mapping.py

src/snovault/elasticsearch/indexer.py

src/snovault/elasticsearch/searches/queries.py

src/snovault/tests/test_searches_queries.py

keenangraham

This looks close, just a couple of questions.

Also looks like real test failure where cherry^ is no longer returning results (404). I noticed the same query between demo and prod also returning less cherry^ results (looks like Files specifically)..

base.ini

development.ini

src/snovault/elasticsearch/indexer.py

keenangraham · 2020-10-21T21:23:59Z

src/snovault/elasticsearch/create_mapping.py

-        return {
-            'type': 'object',
-            'include_in_all': False,
-            'properties': properties,


If this always had properties why the if/else now?

if there is no mapping directive exists in the schema, object should be stored without its fields being analyzed. else clause prevents unnecessary dynamic mapping and update_mapping events in elasticsearch.

This doesn't seem to be working properly for some fields, e.g. cloud_metadata in files. (Might have something to do with it being a calculated_property.)

Before and after. Now you can't look up cloud_metadata.url=* for example.

Right, previous assumption was that if the field was important, it would be in the schema. This can be fixed, implication is that any other object inserted that doesn't have a mapping directive in the schema will have all of its field values be analyzed as keyword.

keenangraham · 2020-10-21T21:25:39Z

src/snovault/elasticsearch/create_mapping.py

@@ -133,10 +134,6 @@ def schema_mapping(name, schema):
            'type': field_type
        }

-        # these fields are unintentially partially matching some small search
-        # keywords because fields are analyzed by nGram analyzer
-        if name in NON_SUBSTRING_FIELDS:


No equivalent replacement of copy_to: _all here?

Here is the newer commit: afbe476#diff-13be881525721a1251be911ff12f4c7a722ced10eb689a96d8c5821e8d9cd0e8R140

Bek

This looks close, just a couple of questions.

Also looks like real test failure where cherry^ is no longer returning results (404). I noticed the same query between demo and prod also returning less cherry^ results (looks like Files specifically)..

we are back to the same assertions in the test and cherry^ is returning 1 result as before in the tests. With the latest change, the File count is equal now.

Bek · 2020-10-23T16:16:52Z

src/snovault/elasticsearch/create_mapping.py

@@ -133,10 +134,6 @@ def schema_mapping(name, schema):
            'type': field_type
        }

-        # these fields are unintentially partially matching some small search
-        # keywords because fields are analyzed by nGram analyzer
-        if name in NON_SUBSTRING_FIELDS:


Here is the newer commit: afbe476#diff-13be881525721a1251be911ff12f4c7a722ced10eb689a96d8c5821e8d9cd0e8R140

src/snovault/elasticsearch/create_mapping.py

keenangraham · 2020-10-26T22:05:28Z

src/snovault/elasticsearch/create_mapping.py

-        if name in NON_SUBSTRING_FIELDS:
-            sub_mapping['include_in_all'] = False
+        if name not in NON_SUBSTRING_FIELDS:
+            if depth == 1 or (depth == 2 and parent == 'array'):


I think this is the logic I'm most uncomfortable with, though the results certainly look closer than before. What's the implication of not tracking depth, as it didn't seem like it was tracked before? Is it possible to always copy to _all if name not in NON_SUBSTRING_FIELDS?

Older demos showed that setting "copy_to": "_all" always here is equivalent to boosting every single field in the nested objects (minus the NON_SUBSTRING_FIELDS). Depth tracking stops "copy_to": "_all" after level 1. Before, if file or experiment award description has "brain" and "blood" in the text, the files or experiments show up under either "blood" or "brain" search term. Non-specific experiments that showed up in the results in earlier demos were due to "copy_to" being applied to the fields of deeply nested objects. Older include_in_all was applied by default and depth tracking is applied under the hypothesis that, this default application of the setting was weighed more towards top fields rather than nested object fields. Result is that, now all top fields of the object has copy_to_all, and nested fields need to be boosted like so "file.award.description": 1.0. We could have achieved the same matching results by boosting all of the individual fields of a type also without depth tracking, which is still an option but may take more effort to match results.

It seems like maybe defining in boost would make sense given we already have top-level fields there, e.g.:

"boost_values": { "accession": 20.0, "@type": 1.0, "alternate_accessions": 1.0, "assay_term_name": 20.0, "assay_term_id": 1.0, "assay_title": 20.0, "assay_slims": 5.0, "dbxrefs": 1.0, "aliases": 1.0, "biosample_ontology.term_id": 1.0, "biosample_ontology.term_name": 10.0,

this default application of the setting was weighed more towards top fields rather than nested object fields.

I don't think this explains the case of award.description with blood or brain. Even if the weight was low on these they would still show up in results, though at the bottom. However we're not seeing them at all on current prod.

My understanding is that if _all is enabled on old ES then include_in_all is True by default. However you can set it to include_in_all: False for a certain field and it will apply to all subfields unless otherwise specified. I think looking at new and old mappings informative here. For file:

I'm a little concerned since the default has changed from don't include unless specifically set to always include if in first level (probably explains why we are seeing more hits on new demo?). The boost offers more specific control, while always including top-level fields doesn't.

In any case might be close enough for our current purposes, and could be tweaked later. But I think worth looking more into.

You are right that boost offers more specific control. I find that it's not feasible to determine what to boost for the number of types we have multiplied by the number of top fields for each type, make demo for each combination, and compare results with the old ES. ctcf search bringing up File type results was due to unintentional boost on the aliases field. Determining which other objects and fields were important to match the existing result count in the old ES maybe challenging.

I think it's likely the difference is just due to not copying the links and unique_keys stuff to _all, which is by default include_in_all: True on old ES:

Pretty sure it was the alias in unique keys that was matching ctcf, not the the alias in embedded properties. (Can confirm that alias in embedded properties is False for old mapping.)

It doesn't look like any of the embedded properties in old mapping are include_in_all: True unless they are defined in boost values for that schema. Could we go back to only copying embedded properties to _all if they are defined in boost, and just make the dynamic template links/unique keys have copy_to?

ctcf search bringing up File type results was due to unintentional boost on the aliases field

Yeah, looks like only field ctcf is mentioned in file is alias and submitted_file name, but it's probably fine to have these intentionally searchable.

Here is another demo with the changes: https://elasticsearch7-without-depth-tracking.demo.encodedcc.org/

keenangraham · 2020-10-27T19:38:25Z

src/snovault/elasticsearch/create_mapping.py

-        return {
-            'type': 'object',
-            'include_in_all': False,
-            'properties': properties,


This doesn't seem to be working properly for some fields, e.g. cloud_metadata in files. (Might have something to do with it being a calculated_property.)

Before and after. Now you can't look up cloud_metadata.url=* for example.

keenangraham · 2020-10-27T21:07:24Z

One other thing. I suppose we should take a look at the dynamic_template definitions one more time. Looking at the original mapping definition I guess these are included in _all by default. I don't think being able to search principals_allowed is that useful but maybe links and unique_keys would be.

'dynamic_templates': [
            {
                'template_principals_allowed': {
                    'path_match': "principals_allowed.*",
                    'match_mapping_type': "string",
                    'mapping': {
                        'type': 'keyword',
                    },
                },
            },
            {
                'template_unique_keys': {
                    'path_match': "unique_keys.*",
                    'match_mapping_type': "string",
                    'mapping': {
                        'type': 'keyword',
                    },
                },
            },
            {
                'template_links': {
                    'path_match': "links.*",
                    'match_mapping_type': "string",
                    'mapping': {
                        'type': 'keyword',
                    },
                },
....

It does seem like group.admin and system.Everyone give search results on prod, which seems a bit silly right?

Possibly this also explains the file type search differences before you added the depth tracking?

keenangraham · 2020-10-27T21:14:49Z

It looks like links and unique_keys are also by default included in all (though not linked_uuids). Could also explain result differences before adding all top-level fields.

Bek · 2020-10-27T22:31:09Z

One other thing. I suppose we should take a look at the dynamic_template definitions one more time. Looking at the original mapping definition I guess these are included in _all by default. I don't think being able to search principals_allowed is that useful but maybe links and unique_keys would be.
'dynamic_templates': [
            {
                'template_principals_allowed': {
                    'path_match': "principals_allowed.*",
                    'match_mapping_type': "string",
                    'mapping': {
                        'type': 'keyword',
                    },
                },
            },
            {
                'template_unique_keys': {
                    'path_match': "unique_keys.*",
                    'match_mapping_type': "string",
                    'mapping': {
                        'type': 'keyword',
                    },
                },
            },
            {
                'template_links': {
                    'path_match': "links.*",
                    'match_mapping_type': "string",
                    'mapping': {
                        'type': 'keyword',
                    },
                },
....
It does seem like group.admin and system.Everyone give search results on prod, which seems a bit silly right?

Possibly this also explains the file type search differences before you added the depth tracking?

Right, some an existing test rely on searching user uuid, but we don't boost user uuid, so it was getting a hit based on "principals_allowed.edit": ["group.admin", "userid.81a6cc12-2847-4e2e-8f2c-f566699eb29e"] because it contained user uuid...

Bek · 2020-10-27T22:43:00Z

One other thing. I suppose we should take a look at the dynamic_template definitions one more time. Looking at the original mapping definition I guess these are included in _all by default. I don't think being able to search principals_allowed is that useful but maybe links and unique_keys would be.
'dynamic_templates': [
            {
                'template_principals_allowed': {
                    'path_match': "principals_allowed.*",
                    'match_mapping_type': "string",
                    'mapping': {
                        'type': 'keyword',
                    },
                },
            },
            {
                'template_unique_keys': {
                    'path_match': "unique_keys.*",
                    'match_mapping_type': "string",
                    'mapping': {
                        'type': 'keyword',
                    },
                },
            },
            {
                'template_links': {
                    'path_match': "links.*",
                    'match_mapping_type': "string",
                    'mapping': {
                        'type': 'keyword',
                    },
                },
....
It does seem like group.admin and system.Everyone give search results on prod, which seems a bit silly right?

Possibly this also explains the file type search differences before you added the depth tracking?

File type searches not showing up was due to aliases field not being copied to _all. This brings up a worrisome implication for search, which is that if the file name didn't have ctcf in the string of it's name, then it wouldn't be included in the results even if the file was part of the experiment that targets CTCF.

keenangraham · 2020-10-27T23:41:54Z

This brings up a worrisome implication for search, which is that if the file name didn't have ctcf in the string of it's name, then it wouldn't be included in the results even if the file was part of the experiment that targets CTCF.

Think this is fine. File for a long time didn't have any access to experiment biosample or assay information. Now that it does we should probably explicitly add these to boost values.

(In other words it's not that we expect ctcf to be a great search query for files, just that the conspicuous absence of files from new results was worth investigating.)

Bek changed the title ~~ES Upgrade~~ SNO-172 ES Upgrade Oct 5, 2020

keenangraham changed the title ~~SNO-172 ES Upgrade~~ SNO-172-upgrade-to-elasticsearch7 Oct 6, 2020

keenangraham requested changes Oct 6, 2020

View reviewed changes

Bek force-pushed the SNO-172-ES7-upgrade branch from 916faeb to 2b86c6e Compare October 14, 2020 18:25

keenangraham reviewed Oct 14, 2020

View reviewed changes

src/snovault/tests/test_searches_queries.py Outdated Show resolved Hide resolved

Bek force-pushed the SNO-172-ES7-upgrade branch 2 times, most recently from 33d8cba to bdc6355 Compare October 15, 2020 18:08

keenangraham requested changes Oct 21, 2020

View reviewed changes

Bek commented Oct 23, 2020

View reviewed changes

Bek force-pushed the SNO-172-ES7-upgrade branch 2 times, most recently from f1e5cae to 8cb4e1a Compare October 26, 2020 08:51

keenangraham requested changes Oct 26, 2020

View reviewed changes

keenangraham requested a review from pedrohr October 26, 2020 23:52

Bek force-pushed the SNO-172-ES7-upgrade branch from 8cb4e1a to 157a13e Compare October 27, 2020 16:03

keenangraham requested changes Oct 27, 2020

View reviewed changes

Bek added 4 commits November 11, 2020 22:10

ES Upgrade

7b43aeb

tmp: enable object dynamic mapping

80798a3

tmp: remove depth tracking, add links and unique keys to _all

b3813f3

short circuit exists query on select objects

0c22a37

Bek force-pushed the SNO-172-ES7-upgrade branch from 00dc931 to 0c22a37 Compare November 11, 2020 17:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SNO-172-upgrade-to-elasticsearch7 #296

SNO-172-upgrade-to-elasticsearch7 #296

Bek commented Oct 5, 2020

keenangraham left a comment

keenangraham Oct 21, 2020

Bek Oct 23, 2020

keenangraham Oct 27, 2020

Bek Oct 27, 2020

keenangraham Oct 21, 2020

Bek Oct 23, 2020

Bek left a comment

Bek Oct 23, 2020

keenangraham Oct 26, 2020

Bek Oct 27, 2020

keenangraham Oct 27, 2020

Bek Oct 27, 2020

keenangraham Oct 27, 2020

Bek Oct 28, 2020

keenangraham Oct 27, 2020

keenangraham commented Oct 27, 2020 •

edited

keenangraham commented Oct 27, 2020

Bek commented Oct 27, 2020

Bek commented Oct 27, 2020

keenangraham commented Oct 27, 2020 •

edited

SNO-172-upgrade-to-elasticsearch7 #296

Are you sure you want to change the base?

SNO-172-upgrade-to-elasticsearch7 #296

Conversation

Bek commented Oct 5, 2020

keenangraham left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Bek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

keenangraham commented Oct 27, 2020 • edited

keenangraham commented Oct 27, 2020

Bek commented Oct 27, 2020

Bek commented Oct 27, 2020

keenangraham commented Oct 27, 2020 • edited

keenangraham commented Oct 27, 2020 •

edited

keenangraham commented Oct 27, 2020 •

edited