[CDCSDK] [PG Parity] Consistency with Nemesis run had an error: Database connection failed during resolving unknown type

[shamanthchandra-yb]

Jira Link: DB-11301

Description

Its BankWorkload

We have following types:

id -> bigint
balance -> varchar
filler -> varchar

In one of the run, it failed with after ~2 hours:

2024-05-11 15:30:03,105 ERROR  ||  WorkerSourceTask{id=yugabyte_connector-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted   [org.apache.kafka.connect.runtime.WorkerTask]
org.apache.kafka.connect.errors.ConnectException: Database connection failed during resolving unknown type
        at io.debezium.connector.postgresql.TypeRegistry.resolveUnknownType(TypeRegistry.java:404)
        at io.debezium.connector.postgresql.TypeRegistry.get(TypeRegistry.java:179)
        at io.debezium.connector.postgresql.PostgresType$Builder.build(PostgresType.java:304)
        at io.debezium.connector.postgresql.TypeRegistry.prime(TypeRegistry.java:338)
        at io.debezium.connector.postgresql.TypeRegistry.<init>(TypeRegistry.java:125)
        at io.debezium.connector.postgresql.connection.PostgresConnection.<init>(PostgresConnection.java:113)
        at io.debezium.connector.postgresql.connection.PostgresConnection.<init>(PostgresConnection.java:121)
        at io.debezium.connector.postgresql.PostgresConnectorTask.lambda$start$1(PostgresConnectorTask.java:95)
        at io.debezium.jdbc.DefaultMainConnectionProvidingConnectionFactory.newConnection(DefaultMainConnectionProvidingConnectionFactory.java:26)
        at io.debezium.connector.postgresql.PostgresConnectorTask.start(PostgresConnectorTask.java:119)
        at io.debezium.connector.common.BaseSourceTask.startIfNeededAndPossible(BaseSourceTask.java:268)
        at io.debezium.connector.common.BaseSourceTask.poll(BaseSourceTask.java:178)
        at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.poll(AbstractWorkerSourceTask.java:469)
        at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.execute(AbstractWorkerSourceTask.java:357)
        at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:204)
        at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:259)
        at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.run(AbstractWorkerSourceTask.java:77)
        at org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:236)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.yugabyte.util.PSQLException: ERROR: recvmsg error: Connection refused
        at com.yugabyte.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2675)
        at com.yugabyte.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2365)
        at com.yugabyte.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:355)
        at com.yugabyte.jdbc.PgStatement.executeInternal(PgStatement.java:490)
        at com.yugabyte.jdbc.PgStatement.execute(PgStatement.java:408)
        at com.yugabyte.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:162)
        at com.yugabyte.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:114)
        at io.debezium.connector.postgresql.TypeRegistry.loadType(TypeRegistry.java:409)
        at io.debezium.connector.postgresql.TypeRegistry.resolveUnknownType(TypeRegistry.java:400)
        ... 22 more

There was rolling restart VMs happening when this issue occured.

Source connector version

fourpointfour/ybdb-debezium:0.6

Connector configuration

add connector connector_name='yugabyte_connector' stream_id='rs_cdc_9aebf7_d004' db_name='cdc_9aebf7' connector_host='172.151.28.147' table_list=['test_cdc_a830de_0'] {'name': 'yugabyte_connector', 'config': {'database.master.addresses': '172.151.31.35:7100,172.151.24.188:7100,172.151.28.132:7100', 'database.hostname': '172.151.31.35:5433,172.151.24.188:5433,172.151.28.132:5433', 'database.port': 5433, 'database.masterhost': '172.151.28.132', 'database.masterport': '7100', 'database.user': 'yugabyte', 'database.password': 'yugabyte', 'database.dbname': 'cdc_9aebf7', 'snapshot.mode': 'never', 'admin.operation.timeout.ms': 600000, 'socket.read.timeout.ms': 300000, 'max.connector.retries': '10', 'operation.timeout.ms': 600000, 'topic.creation.default.compression.type': 'lz4', 'topic.creation.default.cleanup.policy': 'delete', 'topic.creation.default.partitions': 1, 'topic.creation.default.replication.factor': '1', 'tasks.max': '1', 'connector.class': 'io.debezium.connector.postgresql.PostgresConnector', 'topic.prefix': 'db_cdc', 'plugin.name': 'pgoutput', 'slot.name': 'rs_cdc_9aebf7_d004_from_con', 'publication.autocreate.mode': 'filtered', 'publication.name': 'pn_yugabyte_connector', 'table.include.list': 'public.test_cdc_a830de_0', 'transforms': 'Reroute', 'transforms.Reroute.topic.regex': '(.*)', 'transforms.Reroute.topic.replacement': 'db_cdc_all_events', 'transforms.Reroute.type': 'io.debezium.transforms.ByLogicalTableRouter', 'transforms.Reroute.key.field.regex': 'db_cdc(.*)', 'transforms.Reroute.key.field.replacement': '$1', 'max.poll.interval.ms': '5000', 'exactly.once.support': 'requested', 'transaction.ordering': 'true', 'provide.transaction.metadata': 'true'}}

YugabyteDB version

2024.1.1.0-b11

Issue Type

kind/bug

Warning: Please confirm that this issue does not contain any sensitive information

• I confirm this issue does not contain any sensitive information.