[Bug]: [benchmark][standalone][LRU] query raises error `fail to Query on QueryNode 7: worker(7) query failed: Assert "is_system_field_ready()"` in concurrent DQL scene #32136

wangting0128 · 2024-04-10T12:38:34Z

Is there an existing issue for this?

I have searched the existing issues

Environment

- Milvus version: milvus-io-lru-dev-adfed78-20240410
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka): pulsar    
- SDK version(e.g. pymilvus v2.0.0rc2): 2.4.0rc66
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

argo task: lru-fouramf-zp2j9

server:

NAME                                                              READY   STATUS                            RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
lru-autoid-bigdata-etcd-0                                         1/1     Running                           0               32h     10.104.26.109   4am-node32   <none>           <none>
lru-autoid-bigdata-milvus-standalone-78c467bd46-gp47j             1/1     Running                           0               5h3m    10.104.29.126   4am-node35   <none>           <none>
lru-autoid-bigdata-minio-fdfccd4c9-mnht4                          1/1     Running                           1 (25h ago)     32h     10.104.26.110   4am-node32   <none>           <none>
lru-autoid-bigdata-pulsar-bookie-0                                1/1     Running                           0               32h     10.104.26.112   4am-node32   <none>           <none>
lru-autoid-bigdata-pulsar-bookie-1                                1/1     Running                           0               32h     10.104.16.131   4am-node21   <none>           <none>
lru-autoid-bigdata-pulsar-bookie-2                                1/1     Running                           0               32h     10.104.20.216   4am-node22   <none>           <none>
lru-autoid-bigdata-pulsar-broker-0                                1/1     Running                           0               32h     10.104.14.234   4am-node18   <none>           <none>
lru-autoid-bigdata-pulsar-proxy-0                                 1/1     Running                           0               32h     10.104.4.44     4am-node11   <none>           <none>
lru-autoid-bigdata-pulsar-recovery-0                              1/1     Running                           0               32h     10.104.13.133   4am-node16   <none>           <none>
lru-autoid-bigdata-pulsar-zookeeper-0                             1/1     Running                           0               32h     10.104.26.111   4am-node32   <none>           <none>
lru-autoid-bigdata-pulsar-zookeeper-1                             1/1     Running                           0               32h     10.104.27.223   4am-node31   <none>           <none>
lru-autoid-bigdata-pulsar-zookeeper-2                             1/1     Running                           0               32h     10.104.16.134   4am-node21   <none>           <none>

client pod name: lru-fouramf-zp2j9-4098671307
client log:

[2024-04-10 08:01:29,468 - ERROR - fouram]: RPC error: [query], <MilvusException: (code=65535, message=fail to Query on QueryNode 7: worker(7) query failed: Assert "is_system_field_ready()" at /go/src/github.com/milvus-io/milvus/internal/core/src/segcore/SegmentSealedImpl.cpp:1030
 => System field isn't ready when do bulk_insert)>, <Time:{'RPC start': '2024-04-10 07:58:10.910931', 'RPC error': '2024-04-10 08:01:29.467884'}> (decorators.py:146)
[2024-04-10 08:01:29,472 - ERROR - fouram]: (api_response) : [Collection.query] <MilvusException: (code=65535, message=fail to Query on QueryNode 7: worker(7) query failed: Assert "is_system_field_ready()" at /go/src/github.com/milvus-io/milvus/internal/core/src/segcore/SegmentSealedImpl.cpp:1030
 => System field isn't ready when do bulk_insert)>, [requestId: 133905f2-f710-11ee-82ac-be5e0a3a149a] (api_request.py:57)
[2024-04-10 08:01:29,472 - ERROR - fouram]: [CheckFunc] query request check failed, response:<MilvusException: (code=65535, message=fail to Query on QueryNode 7: worker(7) query failed: Assert "is_system_field_ready()" at /go/src/github.com/milvus-io/milvus/internal/core/src/segcore/SegmentSealedImpl.cpp:1030
 => System field isn't ready when do bulk_insert)> (func_check.py:48)
[2024-04-10 09:47:11,121 - ERROR - fouram]: RPC error: [query], <MilvusException: (code=65535, message=fail to Query on QueryNode 7: worker(7) query failed: Assert "is_system_field_ready()" at /go/src/github.com/milvus-io/milvus/internal/core/src/segcore/SegmentSealedImpl.cpp:1030
 => System field isn't ready when do bulk_insert)>, <Time:{'RPC start': '2024-04-10 09:45:55.207041', 'RPC error': '2024-04-10 09:47:11.121420'}> (decorators.py:146)
[2024-04-10 09:47:11,122 - ERROR - fouram]: (api_response) : [Collection.query] <MilvusException: (code=65535, message=fail to Query on QueryNode 7: worker(7) query failed: Assert "is_system_field_ready()" at /go/src/github.com/milvus-io/milvus/internal/core/src/segcore/SegmentSealedImpl.cpp:1030
 => System field isn't ready when do bulk_insert)>, [requestId: 203e5dc4-f71f-11ee-82ac-be5e0a3a149a] (api_request.py:57)
[2024-04-10 09:47:11,122 - ERROR - fouram]: [CheckFunc] query request check failed, response:<MilvusException: (code=65535, message=fail to Query on QueryNode 7: worker(7) query failed: Assert "is_system_field_ready()" at /go/src/github.com/milvus-io/milvus/internal/core/src/segcore/SegmentSealedImpl.cpp:1030
 => System field isn't ready when do bulk_insert)> (func_check.py:48)
[2024-04-10 09:56:31,967 - ERROR - fouram]: RPC error: [query], <MilvusException: (code=65535, message=fail to Query on QueryNode 7: worker(7) query failed: Assert "is_system_field_ready()" at /go/src/github.com/milvus-io/milvus/internal/core/src/segcore/SegmentSealedImpl.cpp:1030
 => System field isn't ready when do bulk_insert)>, <Time:{'RPC start': '2024-04-10 09:53:49.118328', 'RPC error': '2024-04-10 09:56:31.967828'}> (decorators.py:146)
[2024-04-10 09:56:31,971 - ERROR - fouram]: (api_response) : [Collection.query] <MilvusException: (code=65535, message=fail to Query on QueryNode 7: worker(7) query failed: Assert "is_system_field_ready()" at /go/src/github.com/milvus-io/milvus/internal/core/src/segcore/SegmentSealedImpl.cpp:1030
 => System field isn't ready when do bulk_insert)>, [requestId: 3ab77be4-f720-11ee-82ac-be5e0a3a149a] (api_request.py:57)
[2024-04-10 09:56:31,972 - ERROR - fouram]: [CheckFunc] query request check failed, response:<MilvusException: (code=65535, message=fail to Query on QueryNode 7: worker(7) query failed: Assert "is_system_field_ready()" at /go/src/github.com/milvus-io/milvus/internal/core/src/segcore/SegmentSealedImpl.cpp:1030
 => System field isn't ready when do bulk_insert)> (func_check.py:48)
[2024-04-10 09:56:31,989 - ERROR - fouram]: RPC error: [query], <MilvusException: (code=65535, message=fail to Query on QueryNode 7: worker(7) query failed: Assert "is_system_field_ready()" at /go/src/github.com/milvus-io/milvus/internal/core/src/segcore/SegmentSealedImpl.cpp:1030
 => System field isn't ready when do bulk_insert)>, <Time:{'RPC start': '2024-04-10 09:53:49.102826', 'RPC error': '2024-04-10 09:56:31.989447'}> (decorators.py:146)
[2024-04-10 09:56:31,991 - ERROR - fouram]: (api_response) : [Collection.query] <MilvusException: (code=65535, message=fail to Query on QueryNode 7: worker(7) query failed: Assert "is_system_field_ready()" at /go/src/github.com/milvus-io/milvus/internal/core/src/segcore/SegmentSealedImpl.cpp:1030
 => System field isn't ready when do bulk_insert)>, [requestId: 3ab50274-f720-11ee-82ac-be5e0a3a149a] (api_request.py:57)
[2024-04-10 09:56:31,992 - ERROR - fouram]: [CheckFunc] query request check failed, response:<MilvusException: (code=65535, message=fail to Query on QueryNode 7: worker(7) query failed: Assert "is_system_field_ready()" at /go/src/github.com/milvus-io/milvus/internal/core/src/segcore/SegmentSealedImpl.cpp:1030
 => System field isn't ready when do bulk_insert)> (func_check.py:48)
[2024-04-10 10:48:51,615 - ERROR - fouram]: RPC error: [query], <MilvusException: (code=65535, message=fail to Query on QueryNode 7: worker(7) query failed: Assert "is_system_field_ready()" at /go/src/github.com/milvus-io/milvus/internal/core/src/segcore/SegmentSealedImpl.cpp:1030
 => System field isn't ready when do bulk_insert)>, <Time:{'RPC start': '2024-04-10 10:45:05.700705', 'RPC error': '2024-04-10 10:48:51.615682'}> (decorators.py:146)
[2024-04-10 10:48:51,616 - ERROR - fouram]: (api_response) : [Collection.query] <MilvusException: (code=65535, message=fail to Query on QueryNode 7: worker(7) query failed: Assert "is_system_field_ready()" at /go/src/github.com/milvus-io/milvus/internal/core/src/segcore/SegmentSealedImpl.cpp:1030
 => System field isn't ready when do bulk_insert)>, [requestId: 6480b61e-f727-11ee-82ac-be5e0a3a149a] (api_request.py:57)
[2024-04-10 10:48:51,616 - ERROR - fouram]: [CheckFunc] query request check failed, response:<MilvusException: (code=65535, message=fail to Query on QueryNode 7: worker(7) query failed: Assert "is_system_field_ready()" at /go/src/github.com/milvus-io/milvus/internal/core/src/segcore/SegmentSealedImpl.cpp:1030
 => System field isn't ready when do bulk_insert)> (func_check.py:48)

Expected Behavior

No response

Steps To Reproduce

1. create a collection with 3 fields: id(primaryKey, autoID), float_vector(768dim), int64_1(partitionKey=64)
2. build HNSW index
3. prepare 49m data
4. flush collection
5. build index again with the same params
6. load collection
7. concurrent requests:
   - search
   - query <- raises error

Milvus Log

No response

Anything else?

test config:

{
     "dataset_params": {
          "dataset_name": "laion1b_nolang",
          "column_name": "float32_vector",
          "dim": 768,
          "dataset_size": "49m",
          "ni_per": 10000,
          "metric_type": "L2",
          "scalars_params": {
               "int64_1": {
                    "params": {
                         "is_partition_key": true
                    }
               }
          }
     },
     "collection_params": {
          "auto_id": true,
          "other_fields": [
               "int64_1"
          ],
          "num_partitions": 64
     },
     "index_params": {
          "index_type": "HNSW",
          "index_param": {
               "M": 30,
               "efConstruction": 360
          }
     },
     "concurrent_tasks": [
          {
               "type": "search",
               "weight": 1,
               "params": {
                    "top_k": 1,
                    "nq": 10,
                    "search_param": {
                         "ef": 64
                    },
                    "expr": "int64_1 == 1",
                    "timeout": 3000,
                    "random_data": true
               }
          },
          {
               "type": "query",
               "weight": 1,
               "params": {
                    "expr": "int64_1 != 2",
                    "timeout": 3000,
                    "offset": 0,
                    "limit": 10
               }
          }
     ],
     "concurrent_params": {
          "interval": 20,
          "during_time": "12h",
          "concurrent_number": 20
     }
}

The text was updated successfully, but these errors were encountered:

yanliang567 · 2024-04-11T00:51:22Z

@MrPresent-Han the error msg is also very confusing, what's the matter with bulk insert in this case?
/unassign

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>

related: #32136 Signed-off-by: MrPresent-Han <chun.han@zilliz.com>

MrPresent-Han · 2024-04-15T07:27:40Z

@MrPresent-Han the error msg is also very confusing, what's the matter with bulk insert in this case?
/unassign

nothing to do with bulk insert, it's just a wrong-name

wangting0128 · 2024-04-17T02:43:41Z

Recurrent

argo task：lru-fouramf-ngc64-1
image：milvus-io-lru-dev-fc20600-20240416

server：

NAME                                                              READY   STATUS                            RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
lru-verify-32136-etcd-0                                           1/1     Running                           0               12h     10.104.31.147   4am-node34   <none>           <none>
lru-verify-32136-milvus-standalone-7486ff676d-8hpcj               1/1     Running                           1 (4h47m ago)   12h     10.104.15.29    4am-node20   <none>           <none>
lru-verify-32136-minio-55c55c8ff5-k6fxp                           1/1     Running                           0               12h     10.104.31.146   4am-node34   <none>           <none>
lru-verify-32136-pulsar-bookie-0                                  1/1     Running                           0               12h     10.104.30.213   4am-node38   <none>           <none>
lru-verify-32136-pulsar-bookie-1                                  1/1     Running                           0               12h     10.104.20.226   4am-node22   <none>           <none>
lru-verify-32136-pulsar-bookie-2                                  1/1     Running                           0               12h     10.104.31.148   4am-node34   <none>           <none>
lru-verify-32136-pulsar-bookie-init-7j8jp                         0/1     Completed                         0               12h     10.104.6.126    4am-node13   <none>           <none>
lru-verify-32136-pulsar-broker-0                                  1/1     Running                           0               12h     10.104.13.253   4am-node16   <none>           <none>
lru-verify-32136-pulsar-proxy-0                                   1/1     Running                           0               12h     10.104.13.252   4am-node16   <none>           <none>
lru-verify-32136-pulsar-pulsar-init-dns4x                         0/1     Completed                         0               12h     10.104.6.128    4am-node13   <none>           <none>
lru-verify-32136-pulsar-recovery-0                                1/1     Running                           0               12h     10.104.6.130    4am-node13   <none>           <none>
lru-verify-32136-pulsar-zookeeper-0                               1/1     Running                           0               12h     10.104.31.149   4am-node34   <none>           <none>
lru-verify-32136-pulsar-zookeeper-1                               1/1     Running                           0               12h     10.104.23.109   4am-node27   <none>           <none>
lru-verify-32136-pulsar-zookeeper-2                               1/1     Running                           0               12h     10.104.34.73    4am-node37   <none>           <none>

client pod name: lru-fouramf-ngc64-1-3747303924
client log:

client config:

{
     "dataset_params": {
          "dataset_name": "laion1b_nolang",
          "column_name": "float32_vector",
          "dim": 768,
          "dataset_size": "49m",
          "ni_per": 10000,
          "metric_type": "L2",
          "scalars_params": {
               "int64_1": {
                    "params": {
                         "is_partition_key": true
                    }
               }
          }
     },
     "collection_params": {
          "auto_id": true,
          "other_fields": [
               "int64_1"
          ],
          "num_partitions": 64
     },
     "index_params": {
          "index_type": "HNSW",
          "index_param": {
               "M": 30,
               "efConstruction": 360
          }
     },
     "concurrent_tasks": [
          {
               "type": "search",
               "weight": 1,
               "params": {
                    "top_k": 1,
                    "nq": 10,
                    "search_param": {
                         "ef": 64
                    },
                    "expr": "int64_1 == 1",
                    "timeout": 3000,
                    "random_data": true
               }
          },
          {
               "type": "query",
               "weight": 1,
               "params": {
                    "expr": "int64_1 != 2",
                    "timeout": 3000,
                    "offset": 0,
                    "limit": 10
               }
          }
     ],
     "concurrent_params": {
          "interval": 20,
          "during_time": "12h",
          "concurrent_number": 20
     }
}

MrPresent-Han · 2024-04-17T07:48:32Z

the further error is caused by wrong error handling in the segment loader, which result in the following goroutines wrongly thought the segment has been data_loaded an skip the real loading process.
@chyezh will fix it

chyezh · 2024-04-17T08:56:08Z

      defer stateLockGuard.Done(err)

inconsistent state between underlying segment and stateful lock when first loading is failed.

	defer func() {
		stateLockGuard.Done(err)
	}()

wangting0128 · 2024-04-18T02:46:23Z

same scene, different error

argo task：lru-fouramf-k6szn
test image：milvus-io-lru-dev-003dc9f-20240417

server：

NAME                                                              READY   STATUS                            RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
lru-verify-32136-1-etcd-0                                         1/1     Running                           0               14h     10.104.17.21    4am-node23   <none>           <none>
lru-verify-32136-1-milvus-standalone-748cf89bb9-4z7cx             1/1     Running                           2 (6h22m ago)   14h     10.104.33.163   4am-node36   <none>           <none>
lru-verify-32136-1-minio-64c69667c8-vfdrh                         1/1     Running                           0               14h     10.104.18.245   4am-node25   <none>           <none>
lru-verify-32136-1-pulsar-bookie-0                                1/1     Running                           0               14h     10.104.18.246   4am-node25   <none>           <none>
lru-verify-32136-1-pulsar-bookie-1                                1/1     Running                           0               14h     10.104.27.60    4am-node31   <none>           <none>
lru-verify-32136-1-pulsar-bookie-2                                1/1     Running                           0               14h     10.104.15.221   4am-node20   <none>           <none>
lru-verify-32136-1-pulsar-bookie-init-j4gvx                       0/1     Completed                         0               14h     10.104.29.179   4am-node35   <none>           <none>
lru-verify-32136-1-pulsar-broker-0                                1/1     Running                           0               14h     10.104.25.202   4am-node30   <none>           <none>
lru-verify-32136-1-pulsar-proxy-0                                 1/1     Running                           0               14h     10.104.6.125    4am-node13   <none>           <none>
lru-verify-32136-1-pulsar-pulsar-init-4f69w                       0/1     Completed                         0               14h     10.104.29.180   4am-node35   <none>           <none>
lru-verify-32136-1-pulsar-recovery-0                              1/1     Running                           0               14h     10.104.32.196   4am-node39   <none>           <none>
lru-verify-32136-1-pulsar-zookeeper-0                             1/1     Running                           0               14h     10.104.19.104   4am-node28   <none>           <none>
lru-verify-32136-1-pulsar-zookeeper-1                             1/1     Running                           0               14h     10.104.23.96    4am-node27   <none>           <none>
lru-verify-32136-1-pulsar-zookeeper-2                             1/1     Running                           0               14h     10.104.29.184   4am-node35   <none>           <none>

client pod name: lru-fouramf-k6szn-702359166
client log:

client config:

{
     "dataset_params": {
          "dataset_name": "laion1b_nolang",
          "column_name": "float32_vector",
          "dim": 768,
          "dataset_size": "49m",
          "ni_per": 10000,
          "metric_type": "L2",
          "scalars_params": {
               "int64_1": {
                    "params": {
                         "is_partition_key": true
                    }
               }
          }
     },
     "collection_params": {
          "auto_id": true,
          "other_fields": [
               "int64_1"
          ],
          "num_partitions": 64
     },
     "index_params": {
          "index_type": "HNSW",
          "index_param": {
               "M": 30,
               "efConstruction": 360
          }
     },
     "concurrent_tasks": [
          {
               "type": "search",
               "weight": 1,
               "params": {
                    "top_k": 1,
                    "nq": 10,
                    "search_param": {
                         "ef": 64
                    },
                    "expr": "int64_1 == 1",
                    "timeout": 3000,
                    "random_data": true
               }
          },
          {
               "type": "query",
               "weight": 1,
               "params": {
                    "expr": "int64_1 != 2",
                    "timeout": 3000,
                    "offset": 0,
                    "limit": 10
               }
          }
     ],
     "concurrent_params": {
          "interval": 20,
          "during_time": "12h",
          "concurrent_number": 20
     }
}

@MrPresent-Han

related: #32136 Signed-off-by: MrPresent-Han <chun.han@zilliz.com>

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>

related: #32136 Signed-off-by: MrPresent-Han <chun.han@zilliz.com>

Signed-off-by: chyezh <chyezh@outlook.com> Add metric for lru and fix lost delete data when enable lazy load (milvus-io#31868) Signed-off-by: chyezh <chyezh@outlook.com> feat: Support stream reduce v1 (milvus-io#31873) related: milvus-io#31410 --------- Signed-off-by: MrPresent-Han <chun.han@zilliz.com> Change do wait lru dev (milvus-io#31878) Signed-off-by: sunby <sunbingyi1992@gmail.com> enhance: add config for disk cache (milvus-io#31881) fix config not initialized (milvus-io#31890) Signed-off-by: sunby <sunbingyi1992@gmail.com> fix error handle in search (milvus-io#31895) Signed-off-by: sunby <sunbingyi1992@gmail.com> fix: thread safe vector (milvus-io#31898) fix: insert record cannot reinsert (milvus-io#31900) enhance: cancel concurrency restrict for stream reduce and add metrics (milvus-io#31892) Signed-off-by: MrPresent-Han <chun.han@zilliz.com> fix: bit set (milvus-io#31905) fix bitset clear to reset (milvus-io#31908) Signed-off-by: MrPresent-Han <chun.han@zilliz.com> Fix 0404 lru dev (milvus-io#31914) fix: 1. sealed_segment num_rows reset to std::null opt 2. sealed_segment lazy_load reset to true after evicting to avoid shortcut --------- Signed-off-by: MrPresent-Han <chun.han@zilliz.com> fix possible block due to unpin fifo activating principle (milvus-io#31924) Signed-off-by: MrPresent-Han <chun.han@zilliz.com> Add lru reloader lru dev (milvus-io#31952) Signed-off-by: sunby <sunbingyi1992@gmail.com> fix query limit (milvus-io#32060) Signed-off-by: sunby <sunbingyi1992@gmail.com> fix: lru cache lost delete and wrong mem size (milvus-io#32072) issue: milvus-io#30361 Signed-off-by: chyezh <chyezh@outlook.com> enhance: add more metrics for cache and search (milvus-io#31777) (milvus-io#32097) issue: milvus-io#30931 Signed-off-by: chyezh <chyezh@outlook.com> fix:panic due to empty search result when stream reducing(milvus-io#32009) (milvus-io#32083) related: milvus-io#32009 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> fix: sealed segment may not exist when throw (milvus-io#32098) issue: milvus-io#30361 Signed-off-by: chyezh <chyezh@outlook.com> Major compaction 1st edition (milvus-io#31804) (milvus-io#32116) Signed-off-by: wayblink <anyang.wang@zilliz.com> Signed-off-by: Cai Zhang <cai.zhang@zilliz.com> Signed-off-by: chasingegg <chao.gao@zilliz.com> Co-authored-by: chasingegg <chao.gao@zilliz.com> fix: inconsistent between state lock and load state (milvus-io#32171) issue: milvus-io#30361 Signed-off-by: chyezh <chyezh@outlook.com> enhance: Throw error instead of crash when index cannot be built (milvus-io#31844) issue: milvus-io#27589 --------- Signed-off-by: Cai Zhang <cai.zhang@zilliz.com> (cherry picked from commit 1b76766) Signed-off-by: jaime <yun.zhang@zilliz.com> update knowhere to support clustering (milvus-io#32188) Signed-off-by: chasingegg <chao.gao@zilliz.com> fix: segment release is not sync with cache (milvus-io#32212) issue: milvus-io#32206 Signed-off-by: chyezh <chyezh@outlook.com> fix: incorrect pinCount resulting unexpected eviction(milvus-io#32136) (milvus-io#32238) related: milvus-io#32136 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> fix: possible panic when stream reducing (milvus-io#32247) related: milvus-io#32009 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> enhance: [lru-dev] add the related data size for the read apis (milvus-io#32274) cherry-pick: milvus-io#31816 --------- Signed-off-by: SimFG <bang.fu@zilliz.com> add debug log (milvus-io#32303) Signed-off-by: Cai Zhang <cai.zhang@zilliz.com> Refine code for analyze task scheduler (milvus-io#32122) Signed-off-by: Cai Zhang <cai.zhang@zilliz.com> fix: memory leak on stream reduce (milvus-io#32345) related: milvus-io#32304 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> feat: adding cache stats support (milvus-io#32344) See milvus-io#32067 Signed-off-by: Ted Xu <ted.xu@zilliz.com> Fix bug for version (milvus-io#32363) Signed-off-by: Cai Zhang <cai.zhang@zilliz.com> fix: remove sub entity in load delta log, update entity num in segment itself (milvus-io#32350) issue: milvus-io#30361 Signed-off-by: chyezh <chyezh@outlook.com> fix: clear data when loading failure (milvus-io#32370) issue: milvus-io#30361 Signed-off-by: chyezh <chyezh@outlook.com> fix: stream reduce memory leak for failing to release stream reducer(milvus-io#32345) (milvus-io#32381) related: milvus-io#32345 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> Keep InProgress state when getting task state is init (milvus-io#32394) Signed-off-by: Cai Zhang <cai.zhang@zilliz.com> add log for search failed (milvus-io#32367) related: milvus-io#32136 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> enable asan by default (milvus-io#32423) Signed-off-by: sunby <sunbingyi1992@gmail.com> Major compaction refactoring (milvus-io#32149) Signed-off-by: wayblink <anyang.wang@zilliz.com> Lru dev debug (milvus-io#32414) Co-authored-by: wayblink <anyang.wang@zilliz.com> fix: protect loadInfo with atomic, remove rlock at cache to avoid dead lock (milvus-io#32436) issue: milvus-io#32435 Signed-off-by: chyezh <chyezh@outlook.com> fix: use Get but not GetBy of SegmentManager (milvus-io#32438) issue: milvus-io#32435 Signed-off-by: chyezh <chyezh@outlook.com> fix: return growing segment when sealed (milvus-io#32460) issue: milvus-io#32435 Signed-off-by: chyezh <chyezh@outlook.com> enhance: add request resource for lru loading process(milvus-io#32205) (milvus-io#32452) related: milvus-io#32205 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> fix: unexpected deleted index files when lazy loading(milvus-io#32136) (milvus-io#32469) related: milvus-io#32136 Signed-off-by: MrPresent-Han <chun.han@zilliz.com> fix: reference count leak cause release blocked (milvus-io#32465) issue: milvus-io#32379 Signed-off-by: chyezh <chyezh@outlook.com> Fix compaction fail (milvus-io#32473) Signed-off-by: wayblink <anyang.wang@zilliz.com>

issue: #32136 related PR: #32469 Signed-off-by: chyezh <chyezh@outlook.com>

issue: milvus-io#32136 related PR: milvus-io#32469 Signed-off-by: chyezh <chyezh@outlook.com>

stale · 2024-05-18T19:13:57Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

yanliang567 · 2024-05-20T01:19:02Z

keep it

wangting0128 added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. test/benchmark benchmark test labels Apr 10, 2024

wangting0128 assigned yanliang567 and MrPresent-Han Apr 10, 2024

sre-ci-robot unassigned yanliang567 Apr 11, 2024

yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Apr 11, 2024

yanliang567 added this to the 2.4.1 milestone Apr 11, 2024

MrPresent-Han added a commit to MrPresent-Han/milvus that referenced this issue Apr 13, 2024

fix: incorrect pinCount resulting unexpected eviction(milvus-io#32136)

fe7f9bb

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>

MrPresent-Han added a commit to MrPresent-Han/milvus that referenced this issue Apr 13, 2024

fix: incorrect pinCount resulting unexpected eviction(milvus-io#32136)

f240583

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>

MrPresent-Han mentioned this issue Apr 13, 2024

fix: incorrect pinCount resulting unexpected eviction(#32136) #32238

Merged

jaime0815 pushed a commit that referenced this issue Apr 15, 2024

fix: incorrect pinCount resulting unexpected eviction(#32136) (#32238)

9e7ed00

related: #32136 Signed-off-by: MrPresent-Han <chun.han@zilliz.com>

MrPresent-Han mentioned this issue Apr 17, 2024

add log for search failed #32367

Merged

chyezh mentioned this issue Apr 17, 2024

fix: remove sub entity in load delta log, update entity num in segment itself #32350

Merged

jaime0815 pushed a commit that referenced this issue Apr 18, 2024

add log for search failed (#32367)

08574b1

related: #32136 Signed-off-by: MrPresent-Han <chun.han@zilliz.com>

MrPresent-Han added a commit to MrPresent-Han/milvus that referenced this issue Apr 19, 2024

fix: unexpected deleted index files when lazy loading(milvus-io#32136)

66ab1eb

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>

MrPresent-Han mentioned this issue Apr 19, 2024

fix: unexpected deleted index files when lazy loading(#32136) #32469

Merged

jaime0815 pushed a commit that referenced this issue Apr 19, 2024

fix: unexpected deleted index files when lazy loading(#32136) (#32469)

514722a

related: #32136 Signed-off-by: MrPresent-Han <chun.han@zilliz.com>

This was referenced Apr 25, 2024

fix: clear data when loading failure #32370

Merged

fix: protect loadInfo with atomic, remove rlock at cache to avoid dead lock #32436

Merged

fix: garbage_collector_test unittest failure #32640

Merged

jaime0815 pushed a commit that referenced this issue Apr 26, 2024

fix: garbage_collector_test unittest failure (#32640)

e1126a5

issue: #32136 related PR: #32469 Signed-off-by: chyezh <chyezh@outlook.com>

sunby pushed a commit to sunby/milvus that referenced this issue Apr 26, 2024

fix: garbage_collector_test unittest failure (milvus-io#32640)

2df0339

issue: milvus-io#32136 related PR: milvus-io#32469 Signed-off-by: chyezh <chyezh@outlook.com>

sunby pushed a commit to sunby/milvus that referenced this issue Apr 26, 2024

fix: garbage_collector_test unittest failure (milvus-io#32640)

b0c9849

issue: milvus-io#32136 related PR: milvus-io#32469 Signed-off-by: chyezh <chyezh@outlook.com>

stale bot added the stale indicates no udpates for 30 days label May 18, 2024

stale bot removed the stale indicates no udpates for 30 days label May 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: [benchmark][standalone][LRU] query raises error `fail to Query on QueryNode 7: worker(7) query failed: Assert "is_system_field_ready()"` in concurrent DQL scene #32136

[Bug]: [benchmark][standalone][LRU] query raises error `fail to Query on QueryNode 7: worker(7) query failed: Assert "is_system_field_ready()"` in concurrent DQL scene #32136

wangting0128 commented Apr 10, 2024

yanliang567 commented Apr 11, 2024

MrPresent-Han commented Apr 15, 2024

wangting0128 commented Apr 17, 2024

MrPresent-Han commented Apr 17, 2024 •

edited

chyezh commented Apr 17, 2024

wangting0128 commented Apr 18, 2024

stale bot commented May 18, 2024

yanliang567 commented May 20, 2024

[Bug]: [benchmark][standalone][LRU] query raises error fail to Query on QueryNode 7: worker(7) query failed: Assert "is_system_field_ready()" in concurrent DQL scene #32136

[Bug]: [benchmark][standalone][LRU] query raises error fail to Query on QueryNode 7: worker(7) query failed: Assert "is_system_field_ready()" in concurrent DQL scene #32136

Comments

wangting0128 commented Apr 10, 2024

Is there an existing issue for this?

Environment

Current Behavior

Expected Behavior

Steps To Reproduce

Milvus Log

Anything else?

yanliang567 commented Apr 11, 2024

MrPresent-Han commented Apr 15, 2024

wangting0128 commented Apr 17, 2024

Recurrent

MrPresent-Han commented Apr 17, 2024 • edited

chyezh commented Apr 17, 2024

wangting0128 commented Apr 18, 2024

same scene, different error

stale bot commented May 18, 2024

yanliang567 commented May 20, 2024

[Bug]: [benchmark][standalone][LRU] query raises error `fail to Query on QueryNode 7: worker(7) query failed: Assert "is_system_field_ready()"` in concurrent DQL scene #32136

[Bug]: [benchmark][standalone][LRU] query raises error `fail to Query on QueryNode 7: worker(7) query failed: Assert "is_system_field_ready()"` in concurrent DQL scene #32136

MrPresent-Han commented Apr 17, 2024 •

edited