[Bug]: Missing IDs when filtering for all IDs + count

[bostrt]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: 2.4.1
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka): rocksmq
- SDK version(e.g. pymilvus v2.0.0rc2): 2.4.2
- OS(Ubuntu or CentOS): Fedora 38 (podman compose)
- CPU/Memory: 8 CPU, 32GB
- GPU: None

Current Behavior

When I insert around 1,000,000 records, in 1,000 batches and then query to count the final records, some are missing when filtering on the primary ID like this:

>>> from pymilvus import MilvusClient
>>> c = MilvusClient()
>>> c.query(collection_name='test', output_fields=["count(*)"])
[{'count(*)': 999001}]
>>> c.query(collection_name='test', filter='id > 0', output_fields=["count(*)"])
[{'count(*)': 972604}]

This one returns accurate results:

>>> c.query(collection_name='test', filter='id != 0', output_fields=["count(*)"])
[{'count(*)': 999001}]

More troubleshooting:

>>> good_results = c.query(collection_name='test', filter='id != 0', output_fields=["id"])
>>> bad_results = c.query(collection_name='test', filter='id > 0', output_fields=["id"])
>>> good_ids = [x['id'] for x in good_results]
>>> bad_ids = [x['id'] for x in bad_results]

# 1. Remove good ids from bad ids
# 2. Remove bad ids from good ids
>>> unseen = [list(set(bad_ids) - set(good_ids)), list(set(good_ids) - set(bad_ids))]

# All IDs from bad are in good
>>> len(unseen[0])
0
# Good ids have many more than bad 
>>> len(unseen[1])
41444


# Querying with a "bad" id still returns a result
>>> unseen[1][0]
449811121647790610
>>> c.query(collection_name='test', filter='id == 449811121647790610', output_fields=["id"])
[{'id': 449811121647722892}]

# I would have expected this one to fail because of id > 0 given the bug report
>>> c.query(collection_name='test', filter='id > 0 && id == 449811121647790610', output_fields=["id"])
[{'id': 449811121647722892}]

Expected Behavior

I expect any comparison filter that would show all IDs to actually show all IDs.

Steps To Reproduce

1. Start milvus in standalone mode
2. Run `build_db.py` or insert lots of records (1m seems enough to reproduce problem)
3. Query for all IDs greater than zero and count results

Milvus Log

No response

Anything else?

Reproducer script build_db.py. We also tried inserting data sequentially (no batching) and had same issue. We also tried inserting using multi-threading (10 workers) and same issue. The number of "missing" IDs is not the same every time. A co-worker and myself saw different numbers each time we ran the script but it was always reproducible.

cc @rdmullett

from pymilvus import MilvusClient, DataType
import uuid, random

client = MilvusClient()

schema = MilvusClient.create_schema()

schema.add_field("id", datatype=DataType.INT64, is_primary=True, auto_id=True)
schema.add_field(field_name="vector", datatype=DataType.FLOAT_VECTOR, dim=384)
schema.add_field(field_name="color", datatype=DataType.VARCHAR, max_length=256)

# 3.3. Prepare index parameters
index_params = client.prepare_index_params()

# 3.4. Add indexes
index_params.add_index(
    field_name="id"
)

index_params.add_index(
    field_name="vector", 
    index_type="AUTOINDEX",
    metric_type="IP"
)

# 3.5. Create a collection
client.create_collection(
    collection_name="test",
    schema=schema,
    index_params=index_params
)

# Create dataset
colors = ["green", "blue", "yellow", "red", "black", "white", "purple", "pink", "orange", "brown", "grey"]
data = [ {
    "vector": [ random.uniform(-1, 1) for _ in range(384) ], 
    "color": f"{random.choice(colors)}_{str(uuid.uuid4())}" 
} for i in range(0, 1_000_000) ]

print('data is ready to insert')

done = 0
insert_cache = []

# Insert data with 1,000 size batches
for d in data:
    insert_cache.append(d)
    if done % 1000 == 0:
        print('COMPLETE ' + str(done))
        client.insert(collection_name='test', data=insert_cache)
        insert_cache.clear()
    done += 1

[bostrt]

After waiting like 10 minutes, I was running another query and noticed the "missing" ID count dropped from 972604 to:

>>> c.query(collection_name='test', filter='id > 0', output_fields=["count(*)"])
[{'count(*)': 701303}]

EDIT: and again:

>>> c.query(collection_name='test', filter='id > 0', output_fields=["count(*)"])
[{'count(*)': 581652}]
```

[yanliang567]

reproduced on milvus 2.4.1 with pymilvus 2.4.2 after a few attempt, printed more info in one run
the 1st time at round 104

the final results count

[yanliang567]

@congqixia please help to take a look, my cluster
yanliang-241-milvus-standalone-5bc7b96ff6-96vnz 1/1 Running 0 2m58s 10.104.5.184 4am-node12 <none> <none>

/assign @congqixia

[longjiquan]

maybe already fixed by #32858

[yanliang567]

you are right. not reproduced on v2.4.2-20240515-b2d83d33. @bostrt could you please also try on this build?

maybe already fixed by #32858

[yanliang567]

/assign @bostrt
/unassign

[bostrt]

Yes, I just tested and v2.4.2-20240515-b2d83d33 works for me

[bostrt]

/assign @yanliang567
/unassign

[yanliang567]

good to know that. I'd close this issue as Milvus released a new version

[yanliang567]

Milvus 2.4.3 was released, i'd close this issue. thank you all.