Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: [benchmark][standalone][LRU] Search 100 million data and milvus OOM #32205

Open
1 task done
wangting0128 opened this issue Apr 12, 2024 · 4 comments
Open
1 task done

[Bug]: [benchmark][standalone][LRU] Search 100 million data and milvus OOM #32205

wangting0128 opened this issue Apr 12, 2024 · 4 comments
Assignees
@sunby
Labels
kind/bug Issues or changes related a bug test/benchmark benchmark test triage/accepted Indicates an issue or PR is ready to be actively worked on.
Milestone
2.4.lru

Comments

@wangting0128
Copy link
Contributor

wangting0128 commented Apr 12, 2024 •
edited by jaime0815

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version: milvus-io-lru-dev-6e67e5a-20240411
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka): pulsar    
- SDK version(e.g. pymilvus v2.0.0rc2): 2.4.0rc66
- OS(Ubuntu or CentOS): 
- CPU/Memory:  16core8G
- GPU: 
- Others:

Current Behavior

argo task: lru-fouramf-hdgvb

server:

NAME                                                              READY   STATUS                            RESTARTS          AGE     IP              NODE         NOMINATED NODE   READINESS GATES
lru-bingyi-etcd-0                                                 1/1     Running                           0                 19h     10.104.18.146   4am-node25   <none>           <none>
lru-bingyi-milvus-standalone-7ccc5b96f4-n5jr4                     1/1     Running                           43 (22m ago)      19h     10.104.33.123   4am-node36   <none>           <none>
lru-bingyi-minio-dbb847758-9sgdv                                  1/1     Running                           0                 19h     10.104.33.122   4am-node36   <none>           <none>
lru-bingyi-pulsar-bookie-0                                        1/1     Running                           0                 19h     10.104.18.148   4am-node25   <none>           <none>
lru-bingyi-pulsar-bookie-1                                        1/1     Running                           0                 3h35m   10.104.16.212   4am-node21   <none>           <none>
lru-bingyi-pulsar-bookie-2                                        1/1     Running                           0                 19h     10.104.15.72    4am-node20   <none>           <none>
lru-bingyi-pulsar-broker-0                                        1/1     Running                           0                 77m     10.104.13.24    4am-node16   <none>           <none>
lru-bingyi-pulsar-proxy-0                                         1/1     Running                           0                 19h     10.104.15.65    4am-node20   <none>           <none>
lru-bingyi-pulsar-recovery-0                                      1/1     Running                           0                 3h34m   10.104.16.238   4am-node21   <none>           <none>
lru-bingyi-pulsar-zookeeper-0                                     1/1     Running                           0                 19h     10.104.18.147   4am-node25   <none>           <none>
lru-bingyi-pulsar-zookeeper-1                                     1/1     Running                           0                 19h     10.104.17.254   4am-node23   <none>           <none>
lru-bingyi-pulsar-zookeeper-2                                     1/1     Running                           0                 3h34m   10.104.16.228   4am-node21   <none>           <none>
截屏2024-04-12 12 15 14 截屏2024-04-12 12 15 55

client pod name: lru-fouramf-hdgvb-2023823411
client log:

截屏2024-04-12 12 14 19

Expected Behavior

No response

Steps To Reproduce

1. create a collection
2. build HNSW index
3. insert 100 million data(128dim)
4. flush collection
5. build index again with the same params
6. load collection
7. concurrent request:
   - search <- raises error

Milvus Log

No response

Anything else?

test result:

[2024-04-11 15:06:54,814 -  INFO - fouram]: Print locust final stats. (locust_runner.py:56)
[2024-04-11 15:06:54,815 -  INFO - fouram]: Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s (stats.py:789)
[2024-04-11 15:06:54,815 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-04-11 15:06:54,815 -  INFO - fouram]: grpc     search                                                                         11744     0(0.00%) |  34024     715 1634512    830 |    3.30        0.00 (stats.py:789)
[2024-04-11 15:06:54,815 -  INFO - fouram]: --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|----------- (stats.py:789)
[2024-04-11 15:06:54,815 -  INFO - fouram]:          Aggregated                                                                     11744     0(0.00%) |  34024     715 1634512    830 |    3.30        0.00 (stats.py:789)
[2024-04-11 15:06:54,815 -  INFO - fouram]:  (stats.py:790)
[2024-04-11 15:06:54,816 -  INFO - fouram]: [PerfTemplate] Report data: 
{'server': {'deploy_tool': 'helm',
            'deploy_mode': 'standalone',
            'config_name': 'standalone_8c16m',
            'config': {'standalone': {'resources': {'limits': {'cpu': '16',
                                                               'memory': '8Gi',
                                                               'ephemeral-storage': '48Gi'},
                                                    'requests': {'cpu': '16',
                                                                 'memory': '8Gi'}},
                                      'messageQueue': 'pulsar',
                                      'extraEnv': [{'name': 'LOCAL_STORAGE_SIZE',
                                                    'value': '48'}],
                                      'disk': {'size': {'enabled': True}}},
                       'cluster': {'enabled': False},
                       'etcd': {'replicaCount': 1,
                                'metrics': {'enabled': True,
                                            'podMonitor': {'enabled': True}}},
                       'minio': {'mode': 'standalone',
                                 'metrics': {'podMonitor': {'enabled': True}},
                                 'persistence': {'size': '320Gi'}},
                       'pulsar': {'enabled': True},
                       'metrics': {'serviceMonitor': {'enabled': True}},
                       'log': {'level': 'debug'},
                       'extraConfigFiles': {'user.yaml': 'queryNode:\n'
                                                         '  mmap:\n'
                                                         '    mmapEnabled: '
                                                         'true\n'
                                                         '  lazyloadEnabled: '
                                                         'true\n'
                                                         '  '
                                                         'useStreamComputing: '
                                                         'true\n'
                                                         '  cache:\n'
                                                         '    warmup: sync\n'
                                                         '  '
                                                         'lazyloadWaitTimeout: '
                                                         '300000\n'},
                       'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus',
                                         'tag': 'milvus-io-lru-dev-6e67e5a-20240411'}}},
            'host': 'lru-bingyi-milvus.qa-milvus.svc.cluster.local',
            'port': '19530',
            'uri': ''},
 'client': {'test_case_type': 'ConcurrentClientBase',
            'test_case_name': 'test_concurrent_locust_custom_parameters',
            'test_case_params': {'dataset_params': {'metric_type': 'L2',
                                                    'dim': 128,
                                                    'dataset_name': 'sift',
                                                    'dataset_size': '100m',
                                                    'ni_per': 50000},
                                 'collection_params': {'other_fields': [],
                                                       'shards_num': 2,
                                                       'collection_name': 'fouram_100m'},
                                 'index_params': {'index_type': 'HNSW',
                                                  'index_param': {'M': 8,
                                                                  'efConstruction': 200}},
                                 'concurrent_params': {'concurrent_number': 200,
                                                       'during_time': '1h',
                                                       'interval': 20,
                                                       'spawn_rate': None},
                                 'concurrent_tasks': [{'type': 'search',
                                                       'weight': 1,
                                                       'params': {'nq': 1,
                                                                  'top_k': 10,
                                                                  'search_param': {'ef': 16},
                                                                  'timeout': 3600}}]},
            'run_id': 2024041140507530,
            'datetime': '2024-04-11 08:27:30.351650',
            'client_version': '2.2'},
 'result': {'test_result': {'index': {'RT': 4030.6373},
                            'insert': {'total_time': 3816.0353,
                                       'VPS': 26205.2083,
                                       'batch_time': 1.908,
                                       'batch': 50000},
                            'flush': {'RT': 3.0232},
                            'load': {'RT': 5.6172},
                            'Locust': {'Aggregated': {'Requests': 11744,
                                                      'Fails': 0,
                                                      'RPS': 3.3,
                                                      'fail_s': 0.0,
                                                      'RT_max': 1634512.96,
                                                      'RT_avg': 34024.53,
                                                      'TP50': 830.0,
                                                      'TP99': 1408000.0},
                                       'search': {'Requests': 11744,
                                                  'Fails': 0,
                                                  'RPS': 3.3,
                                                  'fail_s': 0.0,
                                                  'RT_max': 1634512.96,
                                                  'RT_avg': 34024.53,
                                                  'TP50': 830.0,
                                                  'TP99': 1408000.0}}}}}
@wangting0128 wangting0128 added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. test/benchmark benchmark test labels Apr 12, 2024
@yanliang567 yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Apr 12, 2024
@yanliang567 yanliang567 added this to the 2.4.1 milestone Apr 12, 2024
@yanliang567 yanliang567 removed their assignment Apr 12, 2024
@MrPresent-Han
Copy link
Contributor

should have been fixed by:#32381
please help verify it @wangting0128

@wangting0128
Copy link
Contributor Author

Recurrent

argo task: lru-fouramf-lgh42
test image：milvus-io-lru-dev-0446b4c-20240418

server：

NAME                                                              READY   STATUS                            RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
lru-verify-32205-etcd-0                                           1/1     Running                           0               170m    10.104.33.99    4am-node36   <none>           <none>
lru-verify-32205-milvus-standalone-748564c7c7-n52kl               0/1     CrashLoopBackOff                  3 (8s ago)      170m    10.104.29.56    4am-node35   <none>           <none>
lru-verify-32205-minio-647b55ff4d-4w8mc                           1/1     Running                           0               170m    10.104.33.100   4am-node36   <none>           <none>
lru-verify-32205-pulsar-bookie-0                                  1/1     Running                           0               170m    10.104.26.181   4am-node32   <none>           <none>
lru-verify-32205-pulsar-bookie-1                                  1/1     Running                           0               170m    10.104.34.31    4am-node37   <none>           <none>
lru-verify-32205-pulsar-bookie-2                                  1/1     Running                           0               170m    10.104.18.53    4am-node25   <none>           <none>
lru-verify-32205-pulsar-bookie-init-xf9c4                         0/1     Completed                         0               170m    10.104.6.43     4am-node13   <none>           <none>
lru-verify-32205-pulsar-broker-0                                  1/1     Running                           0               170m    10.104.1.127    4am-node10   <none>           <none>
lru-verify-32205-pulsar-proxy-0                                   1/1     Running                           0               170m    10.104.9.116    4am-node14   <none>           <none>
lru-verify-32205-pulsar-pulsar-init-b87nl                         0/1     Completed                         0               170m    10.104.6.42     4am-node13   <none>           <none>
lru-verify-32205-pulsar-recovery-0                                1/1     Running                           0               170m    10.104.6.41     4am-node13   <none>           <none>
lru-verify-32205-pulsar-zookeeper-0                               1/1     Running                           0               170m    10.104.18.52    4am-node25   <none>           <none>
lru-verify-32205-pulsar-zookeeper-1                               1/1     Running                           0               169m    10.104.32.215   4am-node39   <none>           <none>
lru-verify-32205-pulsar-zookeeper-2                               1/1     Running                           0               168m    10.104.28.165   4am-node33   <none>           <none>
截屏2024-04-18 14 14 46

client pod name:
client log: lru-fouramf-lgh42-3194488815

截屏2024-04-18 14 15 42

client config:

{
     "dataset_params": {
          "metric_type": "L2",
          "dim": 128,
          "dataset_name": "sift",
          "dataset_size": "100m",
          "ni_per": 50000
     },
     "collection_params": {
          "other_fields": [],
          "shards_num": 2,
          "collection_name": "fouram_100m"
     },
     "index_params": {
          "index_type": "HNSW",
          "index_param": {
               "M": 8,
               "efConstruction": 200
          }
     },
     "concurrent_params": {
          "concurrent_number": [
               20,
               50,
               100,
               200
          ],
          "during_time": "1h",
          "interval": 20
     },
     "concurrent_tasks": [
          {
               "type": "search",
               "weight": 1,
               "params": {
                    "nq": 1,
                    "top_k": 10,
                    "search_param": {
                         "ef": 16
                    },
                    "timeout": 3600
               }
          }
     ]
}

@MrPresent-Han please help to check

MrPresent-Han added a commit to MrPresent-Han/milvus that referenced this issue Apr 18, 2024
@MrPresent-Han
add request resource for lru(milvus-io#32205)
de97dc9 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
MrPresent-Han added a commit to MrPresent-Han/milvus that referenced this issue Apr 18, 2024
@MrPresent-Han
add request resource for lru(milvus-io#32205)
24dd5c1 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
MrPresent-Han added a commit to MrPresent-Han/milvus that referenced this issue Apr 18, 2024
@MrPresent-Han
add request resource for lru(milvus-io#32205)
d8edbd1 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
MrPresent-Han added a commit to MrPresent-Han/milvus that referenced this issue Apr 18, 2024
@MrPresent-Han
add request resource for lru(milvus-io#32205)
3501bab 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
MrPresent-Han added a commit to MrPresent-Han/milvus that referenced this issue Apr 19, 2024
@MrPresent-Han
enhance: add request resource for lru loading process(milvus-io#32205)
8986018 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
@MrPresent-Han MrPresent-Han mentioned this issue Apr 19, 2024
Merged
MrPresent-Han added a commit to MrPresent-Han/milvus that referenced this issue Apr 19, 2024
@MrPresent-Han
enhance: add request resource for lru loading process(milvus-io#32205)
ad4c937 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
MrPresent-Han added a commit to MrPresent-Han/milvus that referenced this issue Apr 19, 2024
@MrPresent-Han
enhance: add request resource for lru loading process(milvus-io#32205)
ad5ad7a 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
tedxu pushed a commit that referenced this issue Apr 19, 2024
@MrPresent-Han
enhance: add request resource for lru loading process(#32205) (#32452)
27cb486 
related: #32205

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
sunby pushed a commit to sunby/milvus that referenced this issue Apr 22, 2024
@chyezh @sunby
Fix partital release (milvus-io#31866)
4c583a5 
Signed-off-by: chyezh <chyezh@outlook.com>

Add metric for lru and fix lost delete data when enable lazy load  (milvus-io#31868)

Signed-off-by: chyezh <chyezh@outlook.com>

feat: Support stream reduce v1 (milvus-io#31873)

related: milvus-io#31410

---------

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>

Change do wait lru dev (milvus-io#31878)

Signed-off-by: sunby <sunbingyi1992@gmail.com>

enhance: add config for disk cache (milvus-io#31881)

fix config not initialized (milvus-io#31890)

Signed-off-by: sunby <sunbingyi1992@gmail.com>

fix error handle in search (milvus-io#31895)

Signed-off-by: sunby <sunbingyi1992@gmail.com>

fix: thread safe vector (milvus-io#31898)

fix: insert record cannot reinsert (milvus-io#31900)

enhance: cancel concurrency restrict for stream reduce and add metrics (milvus-io#31892)

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>

fix: bit set (milvus-io#31905)

fix bitset clear to reset (milvus-io#31908)

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>

Fix 0404 lru dev (milvus-io#31914)

fix:
1. sealed_segment num_rows reset to std::null opt
2. sealed_segment lazy_load reset to true after evicting to avoid
shortcut

---------

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>

fix possible block due to unpin fifo activating principle (milvus-io#31924)

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>

Add lru reloader lru dev (milvus-io#31952)

Signed-off-by: sunby <sunbingyi1992@gmail.com>

fix query limit (milvus-io#32060)

Signed-off-by: sunby <sunbingyi1992@gmail.com>

fix: lru cache lost delete and wrong mem size (milvus-io#32072)

issue: milvus-io#30361

Signed-off-by: chyezh <chyezh@outlook.com>

enhance: add more metrics for cache and search (milvus-io#31777) (milvus-io#32097)

issue: milvus-io#30931

Signed-off-by: chyezh <chyezh@outlook.com>

fix:panic due to empty search result when stream reducing(milvus-io#32009) (milvus-io#32083)

related: milvus-io#32009

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>

fix: sealed segment may not exist when throw (milvus-io#32098)

issue: milvus-io#30361

Signed-off-by: chyezh <chyezh@outlook.com>

Major compaction 1st edition (milvus-io#31804) (milvus-io#32116)

Signed-off-by: wayblink <anyang.wang@zilliz.com>
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Signed-off-by: chasingegg <chao.gao@zilliz.com>
Co-authored-by: chasingegg <chao.gao@zilliz.com>

fix: inconsistent between state lock and load state (milvus-io#32171)

issue: milvus-io#30361

Signed-off-by: chyezh <chyezh@outlook.com>

enhance: Throw error instead of crash when index cannot be built (milvus-io#31844)

issue: milvus-io#27589

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>

(cherry picked from commit 1b76766)
Signed-off-by: jaime <yun.zhang@zilliz.com>

update knowhere to support clustering (milvus-io#32188)

Signed-off-by: chasingegg <chao.gao@zilliz.com>

fix: segment release is not sync with cache (milvus-io#32212)

issue: milvus-io#32206

Signed-off-by: chyezh <chyezh@outlook.com>

fix: incorrect pinCount resulting unexpected eviction(milvus-io#32136) (milvus-io#32238)

related: milvus-io#32136

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>

fix: possible panic when stream reducing (milvus-io#32247)

related: milvus-io#32009

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>

enhance: [lru-dev] add the related data size for the read apis (milvus-io#32274)

cherry-pick: milvus-io#31816

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>

add debug log (milvus-io#32303)

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>

Refine code for analyze task scheduler (milvus-io#32122)

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>

fix: memory leak on stream reduce (milvus-io#32345)

related: milvus-io#32304

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>

feat: adding cache stats support (milvus-io#32344)

See milvus-io#32067

Signed-off-by: Ted Xu <ted.xu@zilliz.com>

Fix bug for version (milvus-io#32363)

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>

fix: remove sub entity in load delta log, update entity num in segment itself (milvus-io#32350)

issue: milvus-io#30361

Signed-off-by: chyezh <chyezh@outlook.com>

fix: clear data when loading failure (milvus-io#32370)

issue: milvus-io#30361

Signed-off-by: chyezh <chyezh@outlook.com>

fix: stream reduce memory leak for failing to release stream reducer(milvus-io#32345) (milvus-io#32381)

related: milvus-io#32345

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>

Keep InProgress state when getting task state is init (milvus-io#32394)

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>

add log for search failed (milvus-io#32367)

related: milvus-io#32136

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>

enable asan by default (milvus-io#32423)

Signed-off-by: sunby <sunbingyi1992@gmail.com>

Major compaction refactoring (milvus-io#32149)

Signed-off-by: wayblink <anyang.wang@zilliz.com>

Lru dev debug (milvus-io#32414)

Co-authored-by: wayblink <anyang.wang@zilliz.com>

fix: protect loadInfo with atomic, remove rlock at cache to avoid dead lock (milvus-io#32436)

issue: milvus-io#32435

Signed-off-by: chyezh <chyezh@outlook.com>

fix: use Get but not GetBy of SegmentManager (milvus-io#32438)

issue: milvus-io#32435

Signed-off-by: chyezh <chyezh@outlook.com>

fix: return growing segment when sealed (milvus-io#32460)

issue: milvus-io#32435

Signed-off-by: chyezh <chyezh@outlook.com>

enhance: add request resource for lru loading process(milvus-io#32205) (milvus-io#32452)

related: milvus-io#32205

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>

fix: unexpected deleted index files when lazy loading(milvus-io#32136) (milvus-io#32469)

related: milvus-io#32136

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>

fix: reference count leak cause release blocked (milvus-io#32465)

issue: milvus-io#32379

Signed-off-by: chyezh <chyezh@outlook.com>

Fix compaction fail (milvus-io#32473)

Signed-off-by: wayblink <anyang.wang@zilliz.com>
@stale Stale
Copy link

stale bot commented May 18, 2024

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

@stale stale bot added the stale indicates no udpates for 30 days label May 18, 2024
@yanliang567
Copy link
Contributor

keep it

@stale stale bot removed the stale indicates no udpates for 30 days label May 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sunby sunby

Labels
kind/bug Issues or changes related a bug test/benchmark benchmark test triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
2.4.lru
Development

No branches or pull requests
4 participants
@sunby @wangting0128 @yanliang567 @MrPresent-Han