[Bug]: [benchmark][standalone][LRU] milvus crash in multi-collections scene when building index and search at the same time

[wangting0128]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: milvus-io-lru-dev-9e7ed00-20240415
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka): pulsar   
- SDK version(e.g. pymilvus v2.0.0rc2): 2.4.0rc66
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

prepare 49m data argo task: lru-big-data-tb7g4
concurrent clients argo task: lru-multi-client-lhpp9

server:

NAME                                                              READY   STATUS                            RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
lru-big-data-etcd-0                                               1/1     Running                           0               28h     10.104.17.116   4am-node23   <none>           <none>
lru-big-data-milvus-standalone-77d4cff797-pz9pp                   1/1     Running                           9 (57m ago)     28h     10.104.21.126   4am-node24   <none>           <none>
lru-big-data-minio-5bd4bc5cd6-6wfkx                               1/1     Running                           0               28h     10.104.21.125   4am-node24   <none>           <none>
lru-big-data-pulsar-bookie-0                                      1/1     Running                           0               28h     10.104.31.44    4am-node34   <none>           <none>
lru-big-data-pulsar-bookie-1                                      1/1     Running                           0               28h     10.104.30.211   4am-node38   <none>           <none>
lru-big-data-pulsar-bookie-2                                      1/1     Running                           0               28h     10.104.17.117   4am-node23   <none>           <none>
lru-big-data-pulsar-bookie-init-xdp22                             0/1     Completed                         0               28h     10.104.1.201    4am-node10   <none>           <none>
lru-big-data-pulsar-broker-0                                      1/1     Running                           0               28h     10.104.6.186    4am-node13   <none>           <none>
lru-big-data-pulsar-proxy-0                                       1/1     Running                           0               28h     10.104.6.185    4am-node13   <none>           <none>
lru-big-data-pulsar-pulsar-init-vh64q                             0/1     Completed                         0               28h     10.104.1.200    4am-node10   <none>           <none>
lru-big-data-pulsar-recovery-0                                    1/1     Running                           0               28h     10.104.1.199    4am-node10   <none>           <none>
lru-big-data-pulsar-zookeeper-0                                   1/1     Running                           0               28h     10.104.17.115   4am-node23   <none>           <none>
lru-big-data-pulsar-zookeeper-1                                   1/1     Running                           0               28h     10.104.29.76    4am-node35   <none>           <none>
lru-big-data-pulsar-zookeeper-2                                   1/1     Running                           0               28h     10.104.34.168   4am-node37   <none>           <none>

client 1 pod name: lru-multi-client-lhpp9-3526480825
build index

client 2 pod name: lru-multi-client-lhpp9-4203639264
search

Expected Behavior

No response

Steps To Reproduce

1. deploy a standalone milvus with LRU enabled and insert 49m data
2. concurrent 2 clients， each one execute steps as follow：
   a. create a collection with int64(pk), float_vector, int64 three fields
   b. build HNSW index
   c. insert 1million 768dim data
   d. flush collection
   e. build index with the same params again
   f. load collection
   g. serial search

Milvus Log

No response

Anything else?

prepare 49m data test result:

[2024-04-15 11:53:11,468 -  INFO - fouram]: Print locust final stats. (locust_runner.py:56)
[2024-04-15 11:53:11,469 -  INFO - fouram]: Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s (stats.py:789)
[2024-04-15 11:53:11,469 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-04-15 11:53:11,469 -  INFO - fouram]: grpc     search                                                                            18    2(11.11%) | 195943  155326  547367 166000 |    0.01        0.00 (stats.py:789)
[2024-04-15 11:53:11,469 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-04-15 11:53:11,469 -  INFO - fouram]:          Aggregated                                                                        18    2(11.11%) | 195943  155326  547367 166000 |    0.01        0.00 (stats.py:789)
[2024-04-15 11:53:11,469 -  INFO - fouram]:  (stats.py:790)
[2024-04-15 11:53:11,470 -  INFO - fouram]: [PerfTemplate] Report data: 
{'server': {'deploy_tool': 'helm',
            'deploy_mode': 'standalone',
            'config_name': 'standalone_8c16m',
            'config': {'standalone': {'resources': {'limits': {'cpu': '16',
                                                               'memory': '8Gi',
                                                               'ephemeral-storage': '48Gi'},
                                                    'requests': {'cpu': '16',
                                                                 'memory': '8Gi'}},
                                      'messageQueue': 'pulsar',
                                      'extraEnv': [{'name': 'LOCAL_STORAGE_SIZE',
                                                    'value': '48'}],
                                      'disk': {'size': {'enabled': True}}},
                       'cluster': {'enabled': False},
                       'etcd': {'replicaCount': 1,
                                'metrics': {'enabled': True,
                                            'podMonitor': {'enabled': True}}},
                       'minio': {'mode': 'standalone',
                                 'metrics': {'podMonitor': {'enabled': True}},
                                 'persistence': {'size': '320Gi'}},
                       'pulsar': {'enabled': True},
                       'metrics': {'serviceMonitor': {'enabled': True}},
                       'log': {'level': 'debug'},
                       'extraConfigFiles': {'user.yaml': 'queryNode:\n'
                                                         '  mmap:\n'
                                                         '    mmapEnabled: '
                                                         'true\n'
                                                         '  lazyloadEnabled: '
                                                         'true\n'
                                                         '  '
                                                         'useStreamComputing: '
                                                         'true\n'
                                                         '  cache:\n'
                                                         '    warmup: sync\n'
                                                         '  '
                                                         'lazyloadWaitTimeout: '
                                                         '300000\n'},
                       'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus',
                                         'tag': 'milvus-io-lru-dev-9e7ed00-20240415'}}},
            'host': 'lru-big-data-milvus.qa-milvus.svc.cluster.local',
            'port': '19530',
            'uri': ''},
 'client': {'test_case_type': 'ConcurrentClientBase',
            'test_case_name': 'test_concurrent_locust_custom_parameters',
            'test_case_params': {'dataset_params': {'metric_type': 'L2',
                                                    'column_name': 'float32_vector',
                                                    'dim': 768,
                                                    'extra_partitions': {'partitions': 49,
                                                                         'data_repeated': False},
                                                    'dataset_name': 'laion1b_nolang',
                                                    'dataset_size': '49m',
                                                    'ni_per': 10000},
                                 'collection_params': {'other_fields': ['int64_1'],
                                                       'collection_name': 'fouram_49m'},
                                 'index_params': {'index_type': 'HNSW',
                                                  'index_param': {'M': 30,
                                                                  'efConstruction': 360}},
                                 'concurrent_params': {'concurrent_number': 1,
                                                       'during_time': '1h',
                                                       'interval': 20,
                                                       'spawn_rate': None},
                                 'concurrent_tasks': [{'type': 'search',
                                                       'weight': 1,
                                                       'params': {'top_k': 1,
                                                                  'nq': 1,
                                                                  'search_param': {'ef': 64},
                                                                  'timeout': 3000,
                                                                  'random_data': True}}]},
            'run_id': 2024041543369779,
            'datetime': '2024-04-15 04:12:16.687881',
            'client_version': '2.4.0'},
 'result': {'test_result': {'index': {'RT': 9692.2451},
                            'insert': {'total_time': 13074.2016,
                                       'VPS': 3761.3472,
                                       'batch_time': 2.6682,
                                       'batch': 10000.0},
                            'flush': {'RT': 2.6428},
                            'load': {'RT': 8.6401},
                            'Locust': {'Aggregated': {'Requests': 18,
                                                      'Fails': 2,
                                                      'RPS': 0.01,
                                                      'fail_s': 0.11,
                                                      'RT_max': 547367.41,
                                                      'RT_avg': 195943.46,
                                                      'TP50': 166000.0,
                                                      'TP99': 547000.0},
                                       'search': {'Requests': 18,
                                                  'Fails': 2,
                                                  'RPS': 0.01,
                                                  'fail_s': 0.11,
                                                  'RT_max': 547367.41,
                                                  'RT_avg': 195943.46,
                                                  'TP50': 166000.0,
                                                  'TP99': 547000.0}}}}}

client 1 test result:

{'server': {'deploy_tool': '',
            'deploy_mode': '',
            'config_name': '',
            'config': {},
            'host': 'lru-big-data-milvus.qa-milvus.svc.cluster.local',
            'port': '19530',
            'uri': ''},
 'client': {'test_case_type': 'ConcurrentClientBase',
            'test_case_name': 'test_concurrent_locust_custom_parameters',
            'test_case_params': {'dataset_params': {'metric_type': 'L2',
                                                    'column_name': 'float32_vector',
                                                    'dim': 768,
                                                    'dataset_name': 'laion1b_nolang',
                                                    'dataset_size': '1m',
                                                    'ni_per': 10000},
                                 'collection_params': {'other_fields': ['int64_1']},
                                 'index_params': {'index_type': 'HNSW',
                                                  'index_param': {'M': 30,
                                                                  'efConstruction': 360}},
                                 'concurrent_params': {'concurrent_number': 20,
                                                       'during_time': '1h',
                                                       'interval': 20,
                                                       'spawn_rate': None},
                                 'concurrent_tasks': [{'type': 'search',
                                                       'weight': 1,
                                                       'params': {'top_k': 1,
                                                                  'nq': 1,
                                                                  'search_param': {'ef': 64},
                                                                  'timeout': 3000}}]},
            'run_id': 2024041603447508,
            'datetime': '2024-04-16 06:52:24.290324',
            'client_version': '2.4.0'},
 'result': {'test_result': {'index': {'RT': 617.916},
                            'insert': {'total_time': 276.0907,
                                       'VPS': 3621.9981,
                                       'batch_time': 2.7609,
                                       'batch': 10000},
                            'flush': {'RT': 2.5228},
                            'load': {'RT': 4.584},
                            'Locust': {'Aggregated': {'Requests': 1535883,
                                                      'Fails': 0,
                                                      'RPS': 426.63,
                                                      'fail_s': 0.0,
                                                      'RT_max': 330466.04,
                                                      'RT_avg': 42.1,
                                                      'TP50': 19,
                                                      'TP99': 270.0},
                                       'search': {'Requests': 1535883,
                                                  'Fails': 0,
                                                  'RPS': 426.63,
                                                  'fail_s': 0.0,
                                                  'RT_max': 330466.04,
                                                  'RT_avg': 42.1,
                                                  'TP50': 19,
                                                  'TP99': 270.0}}}}}

client 2 test result:

{'server': {'deploy_tool': '',
            'deploy_mode': '',
            'config_name': '',
            'config': {},
            'host': 'lru-big-data-milvus.qa-milvus.svc.cluster.local',
            'port': '19530',
            'uri': ''},
 'client': {'test_case_type': 'ConcurrentClientBase',
            'test_case_name': 'test_concurrent_locust_custom_parameters',
            'test_case_params': {'dataset_params': {'metric_type': 'L2',
                                                    'column_name': 'float32_vector',
                                                    'dim': 768,
                                                    'dataset_name': 'laion1b_nolang',
                                                    'dataset_size': '1m',
                                                    'ni_per': 10000},
                                 'collection_params': {'other_fields': ['int64_1']},
                                 'index_params': {'index_type': 'HNSW',
                                                  'index_param': {'M': 30,
                                                                  'efConstruction': 360}},
                                 'concurrent_params': {'concurrent_number': 20,
                                                       'during_time': '1h',
                                                       'interval': 20,
                                                       'spawn_rate': None},
                                 'concurrent_tasks': [{'type': 'search',
                                                       'weight': 1,
                                                       'params': {'top_k': 1,
                                                                  'nq': 1,
                                                                  'search_param': {'ef': 64},
                                                                  'timeout': 3000}}]},
            'run_id': 2024041603559129,
            'datetime': '2024-04-16 06:52:35.030710',
            'client_version': '2.4.0'},
 'result': {'test_result': {'index': {'RT': 420.1551},
                            'insert': {'total_time': 281.663,
                                       'VPS': 3550.3421,
                                       'batch_time': 2.8166,
                                       'batch': 10000},
                            'flush': {'RT': 2.524},
                            'load': {'RT': 4.0307},
                            'Locust': {'Aggregated': {'Requests': 1376361,
                                                      'Fails': 0,
                                                      'RPS': 382.32,
                                                      'fail_s': 0.0,
                                                      'RT_max': 208670.61,
                                                      'RT_avg': 47.6,
                                                      'TP50': 19,
                                                      'TP99': 480.0},
                                       'search': {'Requests': 1376361,
                                                  'Fails': 0,
                                                  'RPS': 382.32,
                                                  'fail_s': 0.0,
                                                  'RT_max': 208670.61,
                                                  'RT_avg': 47.6,
                                                  'TP50': 19,
                                                  'TP99': 480.0}}}}} 

[sunby]

/assign

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

[yanliang567]

keep it