Skip to content
This repository has been archived by the owner on Apr 1, 2024. It is now read-only.

ISSUE-11774: Fail to restart broker Due to org.apache.pulsar.client.api.PulsarClientException$BrokerMetadataException #2956

Closed
sijie opened this issue Aug 25, 2021 · 0 comments
Closed

ISSUE-11774: Fail to restart broker Due to org.apache.pulsar.client.api.PulsarClientException$BrokerMetadataException #2956

sijie opened this issue Aug 25, 2021 · 0 comments
Labels
type/bug

Comments

@sijie
Copy link
Member

sijie commented Aug 25, 2021

Original Issue: apache#11774

Describe the bug
When I restart broker after installing kafka-source plugin into the running broker, the broker restart failed.
the error log as belows:

15:56:47.533 [pulsar-client-io-65-1] WARN  org.apache.pulsar.client.impl.ClientCnx - [id: 0xa042a253, L:/10.70.20.106:46632 - R:10.70.20.106/10.70.20.106:6650] Received error from server: Failed to get batch size for entry java.lang.IllegalArgumentException: Invalid unknonwn tag type: 6
15:56:47.534 [pulsar-client-io-65-1] ERROR org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/functions/metadata][c-pulsar-cluster-zk-1-fw-10.70.20.106-8080-function-metadata-tailer-reader-1b67bf0771] Failed getLastMessageId command
15:56:47.535 [pulsar-client-io-65-1] ERROR org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/functions/metadata][c-pulsar-cluster-zk-1-fw-10.70.20.106-8080-function-metadata-tailer-reader-1b67bf0771] Failed getLastMessageId command
15:56:47.540 [pulsar-io-4-1] INFO  org.apache.pulsar.broker.service.ServerCnx - [/10.70.20.106:46632] Closing consumer: consumerId=1
15:56:47.540 [pulsar-io-4-1] INFO  org.apache.pulsar.broker.service.AbstractDispatcherSingleActiveConsumer - Removing consumer Consumer{subscription=PersistentSubscription{topic=persistent://public/functions/metadata, name=c-pulsar-cluster-zk-1-fw-10.70.20.106-8080-function-metadata-tailer-reader-1b67bf0771}, consumerId=1, consumerName=c-pulsar-cluster-zk-1-fw-10.70.20.106-8080-function-metadata-tailer, address=/10.70.20.106:46632}
15:56:47.541 [pulsar-io-4-1] INFO  org.apache.pulsar.broker.service.persistent.PersistentSubscription - [persistent://public/functions/metadata][c-pulsar-cluster-zk-1-fw-10.70.20.106-8080-function-metadata-tailer-reader-1b67bf0771] Successfully closed subscription [NonDurableCursorImpl{ledger=public/functions/persistent/metadata, ackPos=182:-1, readPos=182:1}]
15:56:47.542 [pulsar-io-4-1] INFO  org.apache.pulsar.broker.service.persistent.PersistentSubscription - [persistent://public/functions/metadata][c-pulsar-cluster-zk-1-fw-10.70.20.106-8080-function-metadata-tailer-reader-1b67bf0771] Successfully closed dispatcher for reader
15:56:47.543 [pulsar-io-4-1] INFO  org.apache.pulsar.broker.service.ServerCnx - [/10.70.20.106:46632] Closed consumer, consumerId=1
15:56:47.545 [pulsar-client-io-65-1] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/functions/metadata] [c-pulsar-cluster-zk-1-fw-10.70.20.106-8080-function-metadata-tailer-reader-1b67bf0771] Closed consumer
15:56:47.545 [main] ERROR org.apache.pulsar.functions.worker.FunctionMetaDataManager - Failed to initialize meta data store
org.apache.pulsar.client.api.PulsarClientException$BrokerMetadataException: The subscription c-pulsar-cluster-zk-1-fw-10.70.20.106-8080-function-metadata-tailer-reader-1b67bf0771 of the topic persistent://public/functions/metadata gets the last message id was failed
Failed to get batch size for entry java.lang.IllegalArgumentException: Invalid unknonwn tag type: 6
	at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:987) ~[org.apache.pulsar-pulsar-client-api-2.8.0.jar:2.8.0]
	at org.apache.pulsar.client.impl.ConsumerImpl.hasMessageAvailable(ConsumerImpl.java:1876) ~[org.apache.pulsar-pulsar-client-original-2.8.0.jar:2.8.0]
	at org.apache.pulsar.client.impl.ReaderImpl.hasMessageAvailable(ReaderImpl.java:168) ~[org.apache.pulsar-pulsar-client-original-2.8.0.jar:2.8.0]
	at org.apache.pulsar.functions.worker.FunctionMetaDataManager.initialize(FunctionMetaDataManager.java:110) [org.apache.pulsar-pulsar-functions-worker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:496) [org.apache.pulsar-pulsar-functions-worker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1490) [org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:793) [org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:259) [org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:331) [org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
15:56:47.545 [main] ERROR org.apache.pulsar.functions.worker.PulsarWorkerService - Error Starting up in worker
java.lang.RuntimeException: Failed to initialize Metadata Manager
	at org.apache.pulsar.functions.worker.FunctionMetaDataManager.initialize(FunctionMetaDataManager.java:116) ~[org.apache.pulsar-pulsar-functions-worker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:496) [org.apache.pulsar-pulsar-functions-worker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1490) [org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:793) [org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:259) [org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:331) [org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
Caused by: org.apache.pulsar.client.api.PulsarClientException$BrokerMetadataException: The subscription c-pulsar-cluster-zk-1-fw-10.70.20.106-8080-function-metadata-tailer-reader-1b67bf0771 of the topic persistent://public/functions/metadata gets the last message id was failed
Failed to get batch size for entry java.lang.IllegalArgumentException: Invalid unknonwn tag type: 6
	at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:987) ~[org.apache.pulsar-pulsar-client-api-2.8.0.jar:2.8.0]
	at org.apache.pulsar.client.impl.ConsumerImpl.hasMessageAvailable(ConsumerImpl.java:1876) ~[org.apache.pulsar-pulsar-client-original-2.8.0.jar:2.8.0]
	at org.apache.pulsar.client.impl.ReaderImpl.hasMessageAvailable(ReaderImpl.java:168) ~[org.apache.pulsar-pulsar-client-original-2.8.0.jar:2.8.0]
	at org.apache.pulsar.functions.worker.FunctionMetaDataManager.initialize(FunctionMetaDataManager.java:110) ~[org.apache.pulsar-pulsar-functions-worker-2.8.0.jar:2.8.0]
	... 5 more
15:56:47.545 [main] ERROR org.apache.pulsar.broker.PulsarService - Failed to start Pulsar service: java.lang.RuntimeException: Failed to initialize Metadata Manager
java.lang.RuntimeException: java.lang.RuntimeException: Failed to initialize Metadata Manager
	at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:571) ~[org.apache.pulsar-pulsar-functions-worker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1490) ~[org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:793) [org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:259) [org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:331) [org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
Caused by: java.lang.RuntimeException: Failed to initialize Metadata Manager
	at org.apache.pulsar.functions.worker.FunctionMetaDataManager.initialize(FunctionMetaDataManager.java:116) ~[org.apache.pulsar-pulsar-functions-worker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:496) ~[org.apache.pulsar-pulsar-functions-worker-2.8.0.jar:2.8.0]
	... 4 more
Caused by: org.apache.pulsar.client.api.PulsarClientException$BrokerMetadataException: The subscription c-pulsar-cluster-zk-1-fw-10.70.20.106-8080-function-metadata-tailer-reader-1b67bf0771 of the topic persistent://public/functions/metadata gets the last message id was failed
Failed to get batch size for entry java.lang.IllegalArgumentException: Invalid unknonwn tag type: 6
	at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:987) ~[org.apache.pulsar-pulsar-client-api-2.8.0.jar:2.8.0]
	at org.apache.pulsar.client.impl.ConsumerImpl.hasMessageAvailable(ConsumerImpl.java:1876) ~[org.apache.pulsar-pulsar-client-original-2.8.0.jar:2.8.0]
	at org.apache.pulsar.client.impl.ReaderImpl.hasMessageAvailable(ReaderImpl.java:168) ~[org.apache.pulsar-pulsar-client-original-2.8.0.jar:2.8.0]
	at org.apache.pulsar.functions.worker.FunctionMetaDataManager.initialize(FunctionMetaDataManager.java:110) ~[org.apache.pulsar-pulsar-functions-worker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:496) ~[org.apache.pulsar-pulsar-functions-worker-2.8.0.jar:2.8.0]
	... 4 more
15:56:47.545 [main] ERROR org.apache.pulsar.PulsarBrokerStarter - Failed to start pulsar service.
org.apache.pulsar.broker.PulsarServerException: java.lang.RuntimeException: java.lang.RuntimeException: Failed to initialize Metadata Manager
	at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:821) ~[org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:259) ~[org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:331) [org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: Failed to initialize Metadata Manager
	at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:571) ~[org.apache.pulsar-pulsar-functions-worker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1490) ~[org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:793) ~[org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
	... 2 more
Caused by: java.lang.RuntimeException: Failed to initialize Metadata Manager
	at org.apache.pulsar.functions.worker.FunctionMetaDataManager.initialize(FunctionMetaDataManager.java:116) ~[org.apache.pulsar-pulsar-functions-worker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:496) ~[org.apache.pulsar-pulsar-functions-worker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1490) ~[org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:793) ~[org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
	... 2 more
Caused by: org.apache.pulsar.client.api.PulsarClientException$BrokerMetadataException: The subscription c-pulsar-cluster-zk-1-fw-10.70.20.106-8080-function-metadata-tailer-reader-1b67bf0771 of the topic persistent://public/functions/metadata gets the last message id was failed
Failed to get batch size for entry java.lang.IllegalArgumentException: Invalid unknonwn tag type: 6
	at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:987) ~[org.apache.pulsar-pulsar-client-api-2.8.0.jar:2.8.0]
	at org.apache.pulsar.client.impl.ConsumerImpl.hasMessageAvailable(ConsumerImpl.java:1876) ~[org.apache.pulsar-pulsar-client-original-2.8.0.jar:2.8.0]
	at org.apache.pulsar.client.impl.ReaderImpl.hasMessageAvailable(ReaderImpl.java:168) ~[org.apache.pulsar-pulsar-client-original-2.8.0.jar:2.8.0]
	at org.apache.pulsar.functions.worker.FunctionMetaDataManager.initialize(FunctionMetaDataManager.java:110) ~[org.apache.pulsar-pulsar-functions-worker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:496) ~[org.apache.pulsar-pulsar-functions-worker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1490) ~[org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
	at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:793) ~[org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
	... 2 more

Additional context

  • pulsar version: 2.8.0
  • pulsar mode: cluster
@sijie sijie added the type/bug label Aug 25, 2021
@sijie sijie closed this as completed Aug 25, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
type/bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@sijie