How to use filter_docs with multiple filter from the same key (bug?)

[simonschoenhofen]

Hey there!
Thanks for the great work!

I am currently struggling a bit with the filter_docs:

I have a docsList of a doc like this:

class ImageDoc(BaseDoc):
    url: ImageUrl
    test_index: Optional[int]
    local_path: Optional[str]
    tensor: Optional[ImageTorchTensor]
    embedding: Optional[AnyEmbedding]

Now I want to filter out all the docs which have either test_index 3 or 9 . What would be the query for this? This does not work:

query = {
    '$and': {
        'test_index' : {'$neq' : 3},
        'test_index' : {'$neq' : 9}
    }
}

And only filters out only the 9 (second part of the query). This seems to be always happening when using two times the same index in the query.

What would be the correct approach for this?

Currently I am solving this with a regex expression, but that does not seem right haha
Thanks!

[JoanFM]

This could be a bug. When using different keys in the same and clause, it works?

[simonschoenhofen]

Yes exactly!

Seems when using the same key, it only uses the last part of the clause and ignore the first one

[JoanFM]

Seems like a bug, will open an issue referencing this discussion

[JoanFM]

Hello @simonschoenhofen .

I finally got time to look at this issue.

This does not seem to be a bug, since the query is a dictionary, a dictionary cannot contain 2 entries of the same key and this is why docarray only sees one of the entries.

I would say that the query should be designed differently using the nin (not in a set) clause:

query = {
'test_index' : {'$nin' : [3, 9]},
}

[simonschoenhofen]

Oh thanks! That solves it!

One suggestion: Can you add to the docs what filter querys are possible and currently implemented? I was not aware that

nin

exists

[JoanFM]

Would these documentation changes help #1762?

[simonschoenhofen]

Yes, thanks!