Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fail to restart broker Due to org.apache.pulsar.client.api.PulsarClientException$BrokerMetadataException #11774

Closed
Richard-Yi opened this issue Aug 25, 2021 · 2 comments
Closed

Fail to restart broker Due to org.apache.pulsar.client.api.PulsarClientException$BrokerMetadataException #11774

Richard-Yi opened this issue Aug 25, 2021 · 2 comments
Labels
type/bug The PR fixed a bug or issue reported a bug

Comments

@Richard-Yi
Copy link

Describe the bug
When I restart broker after installing kafka-source plugin into the running broker, the broker restart failed.
the error log as belows:

15:56:47.533 [pulsar-client-io-65-1] WARN  org.apache.pulsar.client.impl.ClientCnx - [id: 0xa042a253, L:/10.70.20.106:46632 - R:10.70.20.106/10.70.20.106:6650] Received error from server: Failed to get batch size for entry java.lang.IllegalArgumentException: Invalid unknonwn tag type: 6
15:56:47.534 [pulsar-client-io-65-1] ERROR org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/functions/metadata][c-pulsar-cluster-zk-1-fw-10.70.20.106-8080-function-metadata-tailer-reader-1b67bf0771] Failed getLastMessageId command
15:56:47.535 [pulsar-client-io-65-1] ERROR org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/functions/metadata][c-pulsar-cluster-zk-1-fw-10.70.20.106-8080-function-metadata-tailer-reader-1b67bf0771] Failed getLastMessageId command
15:56:47.540 [pulsar-io-4-1] INFO  org.apache.pulsar.broker.service.ServerCnx - [/10.70.20.106:46632] Closing consumer: consumerId=1
15:56:47.540 [pulsar-io-4-1] INFO  org.apache.pulsar.broker.service.AbstractDispatcherSingleActiveConsumer - Removing consumer Consumer{subscription=PersistentSubscription{topic=persistent://public/functions/metadata, name=c-pulsar-cluster-zk-1-fw-10.70.20.106-8080-function-metadata-tailer-reader-1b67bf0771}, consumerId=1, consumerName=c-pulsar-cluster-zk-1-fw-10.70.20.106-8080-function-metadata-tailer, address=/10.70.20.106:46632}
15:56:47.541 [pulsar-io-4-1] INFO  org.apache.pulsar.broker.service.persistent.PersistentSubscription - [persistent://public/functions/metadata][c-pulsar-cluster-zk-1-fw-10.70.20.106-8080-function-metadata-tailer-reader-1b67bf0771] Successfully closed subscription [NonDurableCursorImpl{ledger=public/functions/persistent/metadata, ackPos=182:-1, readPos=182:1}]
15:56:47.542 [pulsar-io-4-1] INFO  org.apache.pulsar.broker.service.persistent.PersistentSubscription - [persistent://public/functions/metadata][c-pulsar-cluster-zk-1-fw-10.70.20.106-8080-function-metadata-tailer-reader-1b67bf0771] Successfully closed dispatcher for reader
15:56:47.543 [pulsar-io-4-1] INFO  org.apache.pulsar.broker.service.ServerCnx - [/10.70.20.106:46632] Closed consumer, consumerId=1
15:56:47.545 [pulsar-client-io-65-1] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/functions/metadata] [c-pulsar-cluster-zk-1-fw-10.70.20.106-8080-function-metadata-tailer-reader-1b67bf0771] Closed consumer
15:56:47.545 [main] ERROR org.apache.pulsar.functions.worker.FunctionMetaDataManager - Failed to initialize meta data store
org.apache.pulsar.client.api.PulsarClientException$BrokerMetadataException: The subscription c-pulsar-cluster-zk-1-fw-10.70.20.106-8080-function-metadata-tailer-reader-1b67bf0771 of the topic persistent://public/functions/metadata gets the last message id was failed
Failed to get batch size for entry java.lang.IllegalArgumentException: Invalid unknonwn tag type: 6
	at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:987) ~[org.apache.pulsar-pulsar-client-api-2.8.0.jar:2.8.0]
	at org.apache.pulsar.client.impl.ConsumerImpl.hasMessageAvailable(ConsumerImpl.java:1876) ~[org.apache.pulsar-pulsar-client-original-2.8.0.jar:2.8.0]
	at org.apache.pulsar.client.impl.ReaderImpl.hasMessageAvailable(ReaderImpl.java:168) ~[org.apache.pulsar-pulsar-client-original-2.8.0.jar:2.8.0]
	at org.apache.pulsar.functions.worker.FunctionMetaDataManager.initialize(FunctionMetaDataManager.java:110) [org.apache.pulsar-pulsar-functions-worker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:496) [org.apache.pulsar-pulsar-functions-worker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1490) [org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:793) [org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:259) [org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:331) [org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
15:56:47.545 [main] ERROR org.apache.pulsar.functions.worker.PulsarWorkerService - Error Starting up in worker
java.lang.RuntimeException: Failed to initialize Metadata Manager
	at org.apache.pulsar.functions.worker.FunctionMetaDataManager.initialize(FunctionMetaDataManager.java:116) ~[org.apache.pulsar-pulsar-functions-worker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:496) [org.apache.pulsar-pulsar-functions-worker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1490) [org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:793) [org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:259) [org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:331) [org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
Caused by: org.apache.pulsar.client.api.PulsarClientException$BrokerMetadataException: The subscription c-pulsar-cluster-zk-1-fw-10.70.20.106-8080-function-metadata-tailer-reader-1b67bf0771 of the topic persistent://public/functions/metadata gets the last message id was failed
Failed to get batch size for entry java.lang.IllegalArgumentException: Invalid unknonwn tag type: 6
	at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:987) ~[org.apache.pulsar-pulsar-client-api-2.8.0.jar:2.8.0]
	at org.apache.pulsar.client.impl.ConsumerImpl.hasMessageAvailable(ConsumerImpl.java:1876) ~[org.apache.pulsar-pulsar-client-original-2.8.0.jar:2.8.0]
	at org.apache.pulsar.client.impl.ReaderImpl.hasMessageAvailable(ReaderImpl.java:168) ~[org.apache.pulsar-pulsar-client-original-2.8.0.jar:2.8.0]
	at org.apache.pulsar.functions.worker.FunctionMetaDataManager.initialize(FunctionMetaDataManager.java:110) ~[org.apache.pulsar-pulsar-functions-worker-2.8.0.jar:2.8.0]
	... 5 more
15:56:47.545 [main] ERROR org.apache.pulsar.broker.PulsarService - Failed to start Pulsar service: java.lang.RuntimeException: Failed to initialize Metadata Manager
java.lang.RuntimeException: java.lang.RuntimeException: Failed to initialize Metadata Manager
	at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:571) ~[org.apache.pulsar-pulsar-functions-worker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1490) ~[org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:793) [org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:259) [org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:331) [org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
Caused by: java.lang.RuntimeException: Failed to initialize Metadata Manager
	at org.apache.pulsar.functions.worker.FunctionMetaDataManager.initialize(FunctionMetaDataManager.java:116) ~[org.apache.pulsar-pulsar-functions-worker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:496) ~[org.apache.pulsar-pulsar-functions-worker-2.8.0.jar:2.8.0]
	... 4 more
Caused by: org.apache.pulsar.client.api.PulsarClientException$BrokerMetadataException: The subscription c-pulsar-cluster-zk-1-fw-10.70.20.106-8080-function-metadata-tailer-reader-1b67bf0771 of the topic persistent://public/functions/metadata gets the last message id was failed
Failed to get batch size for entry java.lang.IllegalArgumentException: Invalid unknonwn tag type: 6
	at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:987) ~[org.apache.pulsar-pulsar-client-api-2.8.0.jar:2.8.0]
	at org.apache.pulsar.client.impl.ConsumerImpl.hasMessageAvailable(ConsumerImpl.java:1876) ~[org.apache.pulsar-pulsar-client-original-2.8.0.jar:2.8.0]
	at org.apache.pulsar.client.impl.ReaderImpl.hasMessageAvailable(ReaderImpl.java:168) ~[org.apache.pulsar-pulsar-client-original-2.8.0.jar:2.8.0]
	at org.apache.pulsar.functions.worker.FunctionMetaDataManager.initialize(FunctionMetaDataManager.java:110) ~[org.apache.pulsar-pulsar-functions-worker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:496) ~[org.apache.pulsar-pulsar-functions-worker-2.8.0.jar:2.8.0]
	... 4 more
15:56:47.545 [main] ERROR org.apache.pulsar.PulsarBrokerStarter - Failed to start pulsar service.
org.apache.pulsar.broker.PulsarServerException: java.lang.RuntimeException: java.lang.RuntimeException: Failed to initialize Metadata Manager
	at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:821) ~[org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:259) ~[org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:331) [org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: Failed to initialize Metadata Manager
	at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:571) ~[org.apache.pulsar-pulsar-functions-worker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1490) ~[org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:793) ~[org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
	... 2 more
Caused by: java.lang.RuntimeException: Failed to initialize Metadata Manager
	at org.apache.pulsar.functions.worker.FunctionMetaDataManager.initialize(FunctionMetaDataManager.java:116) ~[org.apache.pulsar-pulsar-functions-worker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:496) ~[org.apache.pulsar-pulsar-functions-worker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1490) ~[org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:793) ~[org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
	... 2 more
Caused by: org.apache.pulsar.client.api.PulsarClientException$BrokerMetadataException: The subscription c-pulsar-cluster-zk-1-fw-10.70.20.106-8080-function-metadata-tailer-reader-1b67bf0771 of the topic persistent://public/functions/metadata gets the last message id was failed
Failed to get batch size for entry java.lang.IllegalArgumentException: Invalid unknonwn tag type: 6
	at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:987) ~[org.apache.pulsar-pulsar-client-api-2.8.0.jar:2.8.0]
	at org.apache.pulsar.client.impl.ConsumerImpl.hasMessageAvailable(ConsumerImpl.java:1876) ~[org.apache.pulsar-pulsar-client-original-2.8.0.jar:2.8.0]
	at org.apache.pulsar.client.impl.ReaderImpl.hasMessageAvailable(ReaderImpl.java:168) ~[org.apache.pulsar-pulsar-client-original-2.8.0.jar:2.8.0]
	at org.apache.pulsar.functions.worker.FunctionMetaDataManager.initialize(FunctionMetaDataManager.java:110) ~[org.apache.pulsar-pulsar-functions-worker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:496) ~[org.apache.pulsar-pulsar-functions-worker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1490) ~[org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:793) ~[org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
	... 2 more

Additional context

  • pulsar version: 2.8.0
  • pulsar mode: cluster
@Richard-Yi Richard-Yi added the type/bug The PR fixed a bug or issue reported a bug label Aug 25, 2021
@sijie sijie mentioned this issue Aug 25, 2021
Closed
@lhotari
Copy link
Member

lhotari commented Aug 25, 2021 •
edited

This problem is most likely fixed by #10968 which will be included in Pulsar 2.8.1 . Cherry-picked commit in branch-2.8 is 9c3e904 .

@lhotari
Copy link
Member

lhotari commented Aug 25, 2021 •
edited

this issue duplicates #10967 . Closing.

@lhotari lhotari closed this as completed Aug 25, 2021
@Raven888888 Raven888888 mentioned this issue Oct 21, 2021
Open
@sijie sijie mentioned this issue Oct 21, 2021
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/bug The PR fixed a bug or issue reported a bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@lhotari @Richard-Yi