Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: [Nightly] Load collection will occasionally fail for timeout #31522

Open
1 task done
NicoYuan1986 opened this issue Mar 22, 2024 · 10 comments
Open
1 task done

[Bug]: [Nightly] Load collection will occasionally fail for timeout #31522

NicoYuan1986 opened this issue Mar 22, 2024 · 10 comments
Assignees
@weiliu1031 @NicoYuan1986
Labels
ci/e2e kind/bug Issues or changes related a bug priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Milestone
2.4.5

Comments

@NicoYuan1986
Copy link
Contributor

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version: 2.3-20240321-f8496dbc-amd64
- Deployment mode(standalone or cluster): cluster
- MQ type(rocksmq, pulsar or kafka):    kafka
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

Load collection will occasionally fail for timeout.

[2024-03-21T23:23:40.704Z] [2024-03-21 22:15:15 - DEBUG - ci_test]: (api_request)  : [Collection.load] args: [None, 1, 180], kwargs: {} (api_request.py:62)
[2024-03-21T23:23:40.704Z] [2024-03-21 22:15:20 - ERROR - pymilvus.decorators]: RPC error: [load_collection], <MilvusException: (code=65535, message=stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
[2024-03-21T23:23:40.704Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:553 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call
[2024-03-21T23:23:40.704Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:567 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall
[2024-03-21T23:23:40.704Z] /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:107 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]
[2024-03-21T23:23:40.704Z] /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:275 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).ShowPartitions
[2024-03-21T23:23:40.704Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:97 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetPartitions
[2024-03-21T23:23:40.704Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/job_load.go:118 github.com/milvus-io/milvus/internal/querycoordv2/job.(*LoadCollectionJob).Execute
[2024-03-21T23:23:40.705Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/scheduler.go:159 github.com/milvus-io/milvus/internal/querycoordv2/job.(*Scheduler).process
[2024-03-21T23:23:40.705Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/scheduler.go:135 github.com/milvus-io/milvus/internal/querycoordv2/job.(*Scheduler).processQueue
[2024-03-21T23:23:40.705Z] /usr/local/go/src/runtime/asm_amd64.s:1598 runtime.goexit: rpc error: code = DeadlineExceeded desc = context deadline exceeded)>, <Time:{'RPC start': '2024-03-21 22:15:15.452549', 'RPC error': '2024-03-21 22:15:20.464219'}> (decorators.py:139)
[2024-03-21T23:23:40.705Z] [2024-03-21 22:15:20 - ERROR - ci_test]: Traceback (most recent call last):
[2024-03-21T23:23:40.705Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 32, in inner_wrapper
[2024-03-21T23:23:40.705Z]     res = func(*args, **_kwargs)
[2024-03-21T23:23:40.705Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 63, in api_request
[2024-03-21T23:23:40.705Z]     return func(*arg, **kwargs)
[2024-03-21T23:23:40.705Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/orm/collection.py", line 419, in load
[2024-03-21T23:23:40.705Z]     conn.load_collection(
[2024-03-21T23:23:40.705Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 140, in handler
[2024-03-21T23:23:40.705Z]     raise e from e
[2024-03-21T23:23:40.705Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 136, in handler
[2024-03-21T23:23:40.705Z]     return func(*args, **kwargs)
[2024-03-21T23:23:40.705Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 175, in handler
[2024-03-21T23:23:40.705Z]     return func(self, *args, **kwargs)
[2024-03-21T23:23:40.705Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 115, in handler
[2024-03-21T23:23:40.705Z]     raise e from e
[2024-03-21T23:23:40.705Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 86, in handler
[2024-03-21T23:23:40.705Z]     return func(*args, **kwargs)
[2024-03-21T23:23:40.705Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 1046, in load_collection
[2024-03-21T23:23:40.705Z]     check_status(response)
[2024-03-21T23:23:40.705Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/client/utils.py", line 54, in check_status
[2024-03-21T23:23:40.705Z]     raise MilvusException(status.code, status.reason, status.error_code)
[2024-03-21T23:23:40.705Z] pymilvus.exceptions.MilvusException: <MilvusException: (code=65535, message=stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
[2024-03-21T23:23:40.705Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:553 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call
[2024-03-21T23:23:40.705Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:567 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall
[2024-03-21T23:23:40.705Z] /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:107 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]
[2024-03-21T23:23:40.705Z] /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:275 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).ShowPartitions
[2024-03-21T23:23:40.705Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:97 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetPartitions
[2024-03-21T23:23:40.705Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/job_load.go:118 github.com/milvus-io/milvus/internal/querycoordv2/job.(*LoadCollectionJob).Execute
[2024-03-21T23:23:40.705Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/scheduler.go:159 github.com/milvus-io/milvus/internal/querycoordv2/job.(*Scheduler).process
[2024-03-21T23:23:40.705Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/scheduler.go:135 github.com/milvus-io/milvus/internal/querycoordv2/job.(*Scheduler).processQueue
[2024-03-21T23:23:40.705Z] /usr/local/go/src/runtime/asm_amd64.s:1598 runtime.goexit: rpc error: code = DeadlineExceeded desc = context deadline exceeded)>
[2024-03-21T23:23:40.705Z]  (api_request.py:45)
[2024-03-21T23:23:40.705Z] [2024-03-21 22:15:20 - ERROR - ci_test]: (api_response) : <MilvusException: (code=65535, message=stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
[2024-03-21T23:23:40.705Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:553 github.com/milvus-io/milvus/internal/util/grpcclient.(*Cli...... (api_request.py:46)[create_path] folder(/tmp/ci_logs/test) is not exist.

Expected Behavior

load successfully

Steps To Reproduce

No response

Milvus Log

  1. link: https://jenkins.milvus.io:18080/blue/organizations/jenkins/Milvus%20Nightly%20CI/detail/2.3/140/pipeline/205/
  2. log: artifacts-milvus-distributed-kafka-nightly-140-pymilvus-e2e-logs.tar.gz
  3. collection name: search_collection_So5wlFEE
  4. failed time: [2024-03-21T22:15:22.118Z] [gw4] [ 49%] FAILED testcases/test_search.py::TestCollectionSearch::test_search_after_different_index_with_params[128-True-True-True-IVF_SQ8-params2]

Anything else?

No response
@NicoYuan1986 NicoYuan1986 added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Mar 22, 2024
@NicoYuan1986 NicoYuan1986 added this to the 2.3.12 milestone Mar 22, 2024
@yanliang567
Copy link
Contributor

/assign @weiliu1031
/unassign

@yanliang567 yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Mar 22, 2024
@yanliang567 yanliang567 modified the milestones: 2.3.12, 2.3.13 Mar 22, 2024
@weiliu1031
Copy link
Contributor

should be fixed by #31798, please verify this with latest image

@weiliu1031
Copy link
Contributor

/assign @NicoYuan1986

@NicoYuan1986
Copy link
Contributor Author

The error has not occurred since the pr merged. So close the issue.
I will reopen it if the issue reproduces.

@NicoYuan1986
Copy link
Contributor Author

reproduced on 2.3 branch.

  1. link: https://jenkins.milvus.io:18080/blue/organizations/jenkins/Milvus%20Nightly%20CI/detail/2.3/171/pipeline/237
  2. log: artifacts-milvus-distributed-pulsar-nightly-171-pymilvus-e2e-logs.tar.gz
  3. collection name: search_collection_VKAMMKtC
  4. failed time: [2024-04-17T23:04:45.481Z] [gw0] [ 66%] FAILED testcases/test_search.py::TestCollectionSearch::test_search_with_consistency_bounded[200-128-True-True-True]
  5. error message:
[2024-04-17T23:44:54.981Z] [2024-04-17 23:04:39 - DEBUG - ci_test]: (api_request)  : [Collection.load] args: [None, 1, 180], kwargs: {} (api_request.py:62)
[2024-04-17T23:44:54.981Z] [2024-04-17 23:04:44 - ERROR - pymilvus.decorators]: RPC error: [load_collection], <MilvusException: (code=65535, message=stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:553 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:567 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:107 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:275 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).ShowPartitions
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:97 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetPartitions
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/job_load.go:118 github.com/milvus-io/milvus/internal/querycoordv2/job.(*LoadCollectionJob).Execute
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/scheduler.go:159 github.com/milvus-io/milvus/internal/querycoordv2/job.(*Scheduler).process
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/scheduler.go:135 github.com/milvus-io/milvus/internal/querycoordv2/job.(*Scheduler).processQueue
[2024-04-17T23:44:54.981Z] /usr/local/go/src/runtime/asm_amd64.s:1598 runtime.goexit: rpc error: code = DeadlineExceeded desc = context deadline exceeded)>, <Time:{'RPC start': '2024-04-17 23:04:39.053386', 'RPC error': '2024-04-17 23:04:44.068447'}> (decorators.py:146)
[2024-04-17T23:44:54.981Z] [2024-04-17 23:04:44 - ERROR - ci_test]: Traceback (most recent call last):
[2024-04-17T23:44:54.981Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 32, in inner_wrapper
[2024-04-17T23:44:54.981Z]     res = func(*args, **_kwargs)
[2024-04-17T23:44:54.981Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 63, in api_request
[2024-04-17T23:44:54.981Z]     return func(*arg, **kwargs)
[2024-04-17T23:44:54.981Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/orm/collection.py", line 419, in load
[2024-04-17T23:44:54.981Z]     conn.load_collection(
[2024-04-17T23:44:54.981Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 147, in handler
[2024-04-17T23:44:54.981Z]     raise e from e
[2024-04-17T23:44:54.981Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 143, in handler
[2024-04-17T23:44:54.981Z]     return func(*args, **kwargs)
[2024-04-17T23:44:54.981Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 182, in handler
[2024-04-17T23:44:54.981Z]     return func(self, *args, **kwargs)
[2024-04-17T23:44:54.981Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 122, in handler
[2024-04-17T23:44:54.981Z]     raise e from e
[2024-04-17T23:44:54.981Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 87, in handler
[2024-04-17T23:44:54.981Z]     return func(*args, **kwargs)
[2024-04-17T23:44:54.981Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 1046, in load_collection
[2024-04-17T23:44:54.981Z]     check_status(response)
[2024-04-17T23:44:54.981Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/client/utils.py", line 54, in check_status
[2024-04-17T23:44:54.981Z]     raise MilvusException(status.code, status.reason, status.error_code)
[2024-04-17T23:44:54.981Z] pymilvus.exceptions.MilvusException: <MilvusException: (code=65535, message=stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:553 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:567 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:107 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:275 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).ShowPartitions
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:97 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetPartitions
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/job_load.go:118 github.com/milvus-io/milvus/internal/querycoordv2/job.(*LoadCollectionJob).Execute
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/scheduler.go:159 github.com/milvus-io/milvus/internal/querycoordv2/job.(*Scheduler).process
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/scheduler.go:135 github.com/milvus-io/milvus/internal/querycoordv2/job.(*Scheduler).processQueue
[2024-04-17T23:44:54.981Z] /usr/local/go/src/runtime/asm_amd64.s:1598 runtime.goexit: rpc error: code = DeadlineExceeded desc = context deadline exceeded)>
[2024-04-17T23:44:54.981Z]  (api_request.py:45)
[2024-04-17T23:44:54.981Z] [2024-04-17 23:04:44 - ERROR - ci_test]: (api_response) : <MilvusException: (code=65535, message=stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:553 github.com/milvus-io/milvus/internal/util/grpcclient.(*Cli...... (api_request.py:46)

@NicoYuan1986 NicoYuan1986 reopened this Apr 18, 2024
@stale Stale
Copy link

stale bot commented May 18, 2024

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

@stale stale bot added the stale indicates no udpates for 30 days label May 18, 2024
@yanliang567
Copy link
Contributor

keep it as i was told that it reproduces recently again.

@stale stale bot removed the stale indicates no udpates for 30 days label May 20, 2024
@yanliang567 yanliang567 added priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. ci/e2e labels May 21, 2024
@yanliang567 yanliang567 modified the milestones: 2.3.13, 2.4.2 May 21, 2024
@NicoYuan1986
Copy link
Contributor Author

update:

  1. link: https://qa-jenkins.milvus.io/blue/organizations/jenkins/E2E%20Test/detail/E2E%20Test/468/pipeline/
  2. log: artifacts-e2e-test-468-server-logs.tar.gz
  3. collection name: delete_luiZAzlB
  4. failed time: [2024-05-17T10:32:41.809Z] [gw0] [ 16%] FAILED test_delete.py::TestDeleteComplexExpr::test_delete_string_expressions_normal[True-NewStr not in ["0", "1", "2"]]
  5. error message:
[2024-05-17T11:09:13.732Z] [2024-05-17 10:32:28 - DEBUG - ci_test]: (api_request)  : [Collection.load] args: [None, 1, 180], kwargs: {} (api_request.py:62)
[2024-05-17T11:09:13.733Z] [2024-05-17 10:32:36 - ERROR - pymilvus.decorators]: RPC error: [load_collection], <MilvusException: (code=65535, message=stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
[2024-05-17T11:09:13.733Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:556 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call
[2024-05-17T11:09:13.733Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:570 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall
[2024-05-17T11:09:13.733Z] /go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:107 github.com/milvus-io/milvus/internal/distributed/datacoord/client.wrapGrpcCall[...]
[2024-05-17T11:09:13.733Z] /go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:749 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).ListIndexes
[2024-05-17T11:09:13.733Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:296 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).ListIndexes
[2024-05-17T11:09:13.733Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/utils.go:88 github.com/milvus-io/milvus/internal/querycoordv2/job.loadPartitions
[2024-05-17T11:09:13.733Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/job_load.go:176 github.com/milvus-io/milvus/internal/querycoordv2/job.(*LoadCollectionJob).Execute
[2024-05-17T11:09:13.733Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/scheduler.go:159 github.com/milvus-io/milvus/internal/querycoordv2/job.(*Scheduler).process
[2024-05-17T11:09:13.733Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/scheduler.go:135 github.com/milvus-io/milvus/internal/querycoordv2/job.(*Scheduler).processQueue: rpc error: code = DeadlineExceeded desc = context deadline exceeded)>, <Time:{'RPC start': '2024-05-17 10:32:28.275153', 'RPC error': '2024-05-17 10:32:36.210539'}> (decorators.py:139)
[2024-05-17T11:09:13.733Z] [2024-05-17 10:32:36 - ERROR - ci_test]: Traceback (most recent call last):
[2024-05-17T11:09:13.733Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 32, in inner_wrapper
[2024-05-17T11:09:13.733Z]     res = func(*args, **_kwargs)
[2024-05-17T11:09:13.733Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 63, in api_request
[2024-05-17T11:09:13.733Z]     return func(*arg, **kwargs)
[2024-05-17T11:09:13.733Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/orm/collection.py", line 424, in load
[2024-05-17T11:09:13.733Z]     conn.load_collection(
[2024-05-17T11:09:13.733Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 140, in handler
[2024-05-17T11:09:13.733Z]     raise e from e
[2024-05-17T11:09:13.733Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 136, in handler
[2024-05-17T11:09:13.733Z]     return func(*args, **kwargs)
[2024-05-17T11:09:13.733Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 175, in handler
[2024-05-17T11:09:13.733Z]     return func(self, *args, **kwargs)
[2024-05-17T11:09:13.733Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 115, in handler
[2024-05-17T11:09:13.733Z]     raise e from e
[2024-05-17T11:09:13.733Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 86, in handler
[2024-05-17T11:09:13.733Z]     return func(*args, **kwargs)
[2024-05-17T11:09:13.733Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 1148, in load_collection
[2024-05-17T11:09:13.733Z]     check_status(response)
[2024-05-17T11:09:13.733Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/utils.py", line 63, in check_status
[2024-05-17T11:09:13.733Z]     raise MilvusException(status.code, status.reason, status.error_code)
[2024-05-17T11:09:13.733Z] pymilvus.exceptions.MilvusException: <MilvusException: (code=65535, message=stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
[2024-05-17T11:09:13.733Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:556 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call
[2024-05-17T11:09:13.733Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:570 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall
[2024-05-17T11:09:13.733Z] /go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:107 github.com/milvus-io/milvus/internal/distributed/datacoord/client.wrapGrpcCall[...]
[2024-05-17T11:09:13.733Z] /go/src/github.com/milvus-io/milvus/internal/distributed/datacoord/client/client.go:749 github.com/milvus-io/milvus/internal/distributed/datacoord/client.(*Client).ListIndexes
[2024-05-17T11:09:13.733Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:296 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).ListIndexes
[2024-05-17T11:09:13.733Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/utils.go:88 github.com/milvus-io/milvus/internal/querycoordv2/job.loadPartitions
[2024-05-17T11:09:13.733Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/job_load.go:176 github.com/milvus-io/milvus/internal/querycoordv2/job.(*LoadCollectionJob).Execute
[2024-05-17T11:09:13.733Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/scheduler.go:159 github.com/milvus-io/milvus/internal/querycoordv2/job.(*Scheduler).process
[2024-05-17T11:09:13.733Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/scheduler.go:135 github.com/milvus-io/milvus/internal/querycoordv2/job.(*Scheduler).processQueue: rpc error: code = DeadlineExceeded desc = context deadline exceeded)>
[2024-05-17T11:09:13.733Z]  (api_request.py:45)
[2024-05-17T11:09:13.733Z] [2024-05-17 10:32:36 - ERROR - ci_test]: (api_response) : <MilvusException: (code=65535, message=stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
[2024-05-17T11:09:13.733Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:556 github.com/milvus-io/milvus/internal/util/grpcclient.(*Cli...... (api_request.py:46)

@NicoYuan1986
Copy link
Contributor Author

update:

  1. link: https://jenkins.milvus.io:18080/blue/organizations/jenkins/Milvus%20Nightly%20CI/detail/2.3/205/pipeline/205/
  2. log: artifacts-milvus-distributed-kafka-nightly-205-pymilvus-e2e-logs.tar.gz
  3. collection name: coll__ffQ9yake
  4. failed time: [2024-05-21T21:19:01.431Z] [gw0] [ 23%] FAILED testcases/test_partition.py::TestPartitionParams::test_load_partition_after_load_partition
  5. error message:
2024-05-21T23:00:09.199Z] [2024-05-21 21:18:55 - DEBUG - ci_test]: (api_request)  : [Partition.load] args: [1, 180], kwargs: {} (api_request.py:62)
[2024-05-21T23:00:09.199Z] [2024-05-21 21:19:00 - ERROR - pymilvus.decorators]: RPC error: [load_partitions], <MilvusException: (code=65535, message=stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
[2024-05-21T23:00:09.199Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:553 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call
[2024-05-21T23:00:09.199Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:567 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall
[2024-05-21T23:00:09.199Z] /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:107 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]
[2024-05-21T23:00:09.199Z] /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:183 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).DescribeCollection
[2024-05-21T23:00:09.199Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:78 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetCollectionSchema
[2024-05-21T23:00:09.199Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/utils.go:76 github.com/milvus-io/milvus/internal/querycoordv2/job.loadPartitions
[2024-05-21T23:00:09.199Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/job_load.go:352 github.com/milvus-io/milvus/internal/querycoordv2/job.(*LoadPartitionJob).Execute
[2024-05-21T23:00:09.199Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/scheduler.go:159 github.com/milvus-io/milvus/internal/querycoordv2/job.(*Scheduler).process
[2024-05-21T23:00:09.199Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/scheduler.go:135 github.com/milvus-io/milvus/internal/querycoordv2/job.(*Scheduler).processQueue: rpc error: code = DeadlineExceeded desc = context deadline exceeded)>, <Time:{'RPC start': '2024-05-21 21:18:55.092973', 'RPC error': '2024-05-21 21:19:00.113220'}> (decorators.py:139)
[2024-05-21T23:00:09.199Z] [2024-05-21 21:19:00 - ERROR - ci_test]: Traceback (most recent call last):
[2024-05-21T23:00:09.199Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 32, in inner_wrapper
[2024-05-21T23:00:09.199Z]     res = func(*args, **_kwargs)
[2024-05-21T23:00:09.199Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 63, in api_request
[2024-05-21T23:00:09.199Z]     return func(*arg, **kwargs)
[2024-05-21T23:00:09.199Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/orm/partition.py", line 199, in load
[2024-05-21T23:00:09.199Z]     return conn.load_partitions(
[2024-05-21T23:00:09.199Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 140, in handler
[2024-05-21T23:00:09.199Z]     raise e from e
[2024-05-21T23:00:09.199Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 136, in handler
[2024-05-21T23:00:09.199Z]     return func(*args, **kwargs)
[2024-05-21T23:00:09.199Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 175, in handler
[2024-05-21T23:00:09.199Z]     return func(self, *args, **kwargs)
[2024-05-21T23:00:09.199Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 115, in handler
[2024-05-21T23:00:09.199Z]     raise e from e
[2024-05-21T23:00:09.199Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 86, in handler
[2024-05-21T23:00:09.199Z]     return func(*args, **kwargs)
[2024-05-21T23:00:09.199Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 1127, in load_partitions
[2024-05-21T23:00:09.199Z]     check_status(response)
[2024-05-21T23:00:09.199Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/client/utils.py", line 54, in check_status
[2024-05-21T23:00:09.199Z]     raise MilvusException(status.code, status.reason, status.error_code)
[2024-05-21T23:00:09.199Z] pymilvus.exceptions.MilvusException: <MilvusException: (code=65535, message=stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
[2024-05-21T23:00:09.199Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:553 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call
[2024-05-21T23:00:09.199Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:567 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall
[2024-05-21T23:00:09.199Z] /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:107 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]
[2024-05-21T23:00:09.199Z] /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:183 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).DescribeCollection
[2024-05-21T23:00:09.199Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:78 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetCollectionSchema
[2024-05-21T23:00:09.199Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/utils.go:76 github.com/milvus-io/milvus/internal/querycoordv2/job.loadPartitions
[2024-05-21T23:00:09.199Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/job_load.go:352 github.com/milvus-io/milvus/internal/querycoordv2/job.(*LoadPartitionJob).Execute
[2024-05-21T23:00:09.199Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/scheduler.go:159 github.com/milvus-io/milvus/internal/querycoordv2/job.(*Scheduler).process
[2024-05-21T23:00:09.199Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/scheduler.go:135 github.com/milvus-io/milvus/internal/querycoordv2/job.(*Scheduler).processQueue: rpc error: code = DeadlineExceeded desc = context deadline exceeded)>
[2024-05-21T23:00:09.199Z]  (api_request.py:45)
[2024-05-21T23:00:09.200Z] [2024-05-21 21:19:00 - ERROR - ci_test]: (api_response) : <MilvusException: (code=65535, message=stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
[2024-05-21T23:00:09.200Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:553 github.com/milvus-io/milvus/internal/util/grpcclient.(*Cli...... (api_request.py:46)

@NicoYuan1986
Copy link
Contributor Author

update: https://jenkins.milvus.io:18080/blue/organizations/jenkins/Milvus%20Nightly%20CI/detail/2.3/206/pipeline/237

@yanliang567 yanliang567 removed this from the 2.4.2 milestone May 24, 2024
@yanliang567 yanliang567 modified the milestones: 2.4.3, 2.4.4 May 24, 2024
@yanliang567 yanliang567 modified the milestones: 2.4.4, 2.4.5 Jun 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@weiliu1031 weiliu1031

@NicoYuan1986 NicoYuan1986

Labels
ci/e2e kind/bug Issues or changes related a bug priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
2.4.5
Development

No branches or pull requests
3 participants
@yanliang567 @weiliu1031 @NicoYuan1986