[BUG] Kafka consumer stop consuming new message, and present wrong max offset and leader epoch -1, confluent-kafka-go v2.1.0

[vx416]

Description

I create a kafka consumer group and use Assign method for consuming specific topic with specific partition. If this topic has 7 partition, I will create 7 consumer group and assign specific partition and offset for it and each consumer group is running in isolated goroutine.

This consumer group per goroutine was working fine, after I scale partition to eight. Each consumer will consume first offset, then stop consume next offset:

afka-production-1.kafka-svc.production.svc.cluster.local:9093/1: Fetch topic SOL_TWD_OUT [1] at offset 5215904 (leader epoch 0, current leader epoch -1, v3)
%7|1705555265.819|FETCH|rdkafka#consumer-6| [thrd:kafka-production-1.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-1.kafka-svc.production.svc.cluster.local:9093/1: Fetch 1/1/3 toppar(s)
%7|1705555265.819|SEND|rdkafka#consumer-6| [thrd:kafka-production-1.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-1.kafka-svc.production.svc.cluster.local:9093/1: Sent FetchRequest (v11, 101 bytes @ 0, CorrId 534)
%7|1705555265.939|RECV|rdkafka#consumer-9| [thrd:kafka-production-1.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-1.kafka-svc.production.svc.cluster.local:9093/1: Received FetchResponse (v11, 73 bytes, CorrId 533, rtt 499.79ms)
%7|1705555265.940|FETCH|rdkafka#consumer-9| [thrd:kafka-production-1.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-1.kafka-svc.production.svc.cluster.local:9093/1: Topic SOL_TWD_OUT [4] MessageSet size 0, error "Success", MaxOffset 5204949, LSO 5204949, Ver 3/3
%7|1705555265.940|FETCH|rdkafka#consumer-9| [thrd:kafka-production-1.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-1.kafka-svc.production.svc.cluster.local:9093/1: Fetch topic SOL_TWD_OUT [4] at offset 5204949 (leader epoch 0, current leader epoch -1, v3)
%7|1705555265.940|FETCH|rdkafka#consumer-9| [thrd:kafka-production-1.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-1.kafka-svc.production.svc.cluster.local:9093/1: Fetch 1/1/3 toppar(s)
%7|1705555265.940|SEND|rdkafka#consumer-9| [thrd:kafka-production-1.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-1.kafka-svc.production.svc.cluster.local:9093/1: Sent FetchRequest (v11, 101 bytes @ 0, CorrId 534)
%7|1705555266.115|RECV|rdkafka#consumer-2| [thrd:kafka-production-2.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-2.kafka-svc.production.svc.cluster.local:9093/2: Received FetchResponse (v11, 79 bytes, CorrId 530, rtt 500.74ms)
%7|1705555266.115|FETCH|rdkafka#consumer-2| [thrd:kafka-production-2.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-2.kafka-svc.production.svc.cluster.local:9093/2: Topic EXCHANGE_RATE_TWD [0] MessageSet size 0, error "Success", MaxOffset 1353609, LSO 1353609, Ver 3/3
%7|1705555266.115|FETCH|rdkafka#consumer-2| [thrd:kafka-production-2.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-2.kafka-svc.production.svc.cluster.local:9093/2: Fetch topic EXCHANGE_RATE_TWD [0] at offset 1353609 (leader epoch 0, current leader epoch -1, v3)
%7|1705555266.115|FETCH|rdkafka#consumer-2| [thrd:kafka-production-2.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-2.kafka-svc.production.svc.cluster.local:9093/2: Fetch 1/1/1 toppar(s)
%7|1705555266.115|SEND|rdkafka#consumer-2| [thrd:kafka-production-2.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-2.kafka-svc.production.svc.cluster.local:9093/2: Sent FetchRequest (v11, 107 bytes @ 0, CorrId 531)
%7|1705555266.119|RECV|rdkafka#consumer-10| [thrd:kafka-production-2.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-2.kafka-svc.production.svc.cluster.local:9093/2: Received FetchResponse (v11, 73 bytes, CorrId 532, rtt 501.06ms)
%7|1705555266.119|FETCH|rdkafka#consumer-10| [thrd:kafka-production-2.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-2.kafka-svc.production.svc.cluster.local:9093/2: Topic SOL_TWD_OUT [5] MessageSet size 0, error "Success", MaxOffset 5218092, LSO 5218092, Ver 3/3
%7|1705555266.119|FETCH|rdkafka#consumer-10| [thrd:kafka-production-2.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-2.kafka-svc.production.svc.cluster.local:9093/2: Fetch topic SOL_TWD_OUT [5] at offset 5218092 (leader epoch 0, current leader epoch -1, v3)
%7|1705555266.119|FETCH|rdkafka#consumer-10| [thrd:kafka-production-2.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-2.kafka-svc.production.svc.cluster.local:9093/2: Fetch 1/1/3 toppar(s)
%7|1705555266.119|SEND|rdkafka#consumer-10| [thrd:kafka-production-2.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-2.kafka-svc.production.svc.cluster.local:9093/2: Sent FetchRequest (v11, 101 bytes @ 0, CorrId 533)
%7|1705555266.120|RECV|rdkafka#consumer-13| [thrd:kafka-production-2.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-2.kafka-svc.production.svc.cluster.local:9093/2: Received FetchResponse (v11, 73 bytes, CorrId 530, rtt 501.18ms)
%7|1705555266.120|FETCH|rdkafka#consumer-13| [thrd:kafka-production-2.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-2.kafka-svc.production.svc.cluster.local:9093/2: Topic SOL_TWD_OUT [8] MessageSet size 0, error "Success", MaxOffset 9109, LSO 9109, Ver 3/3
%7|1705555266.120|FETCH|rdkafka#consumer-13| [thrd:kafka-production-2.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-2.kafka-svc.production.svc.cluster.local:9093/2: Fetch topic SOL_TWD_OUT [8] at offset 9109 (leader epoch 0, current leader epoch -1, v3)
%7|1705555266.120|FETCH|rdkafka#consumer-13| [thrd:kafka-production-2.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-2.kafka-svc.production.svc.cluster.local:9093/2: Fetch 1/1/3 toppar(s)
%7|1705555266.120|SEND|rdkafka#consumer-13| [thrd:kafka-production-2.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-2.kafka-svc.production.svc.cluster.local:9093/2: Sent FetchRequest (v11, 101 bytes @ 0, CorrId 531)
%7|1705555266.187|RECV|rdkafka#consumer-3| [thrd:kafka-production-0.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-0.kafka-svc.production.svc.cluster.local:9093/0: Received FetchResponse (v11, 80 bytes, CorrId 529, rtt 557.51ms)
%7|1705555266.187|FETCH|rdkafka#consumer-3| [thrd:kafka-production-0.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-0.kafka-svc.production.svc.cluster.local:9093/0: Topic EXCHANGE_RATE_BITO [0] MessageSet size 0, error "Success", MaxOffset 1353607, LSO 1353607, Ver 3/3
%7|1705555266.187|FETCH|rdkafka#consumer-3| [thrd:kafka-production-0.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-0.kafka-svc.production.svc.cluster.local:9093/0: Fetch topic EXCHANGE_RATE_BITO [0] at offset 1353607 (leader epoch 0, current leader epoch -1, v3)
%7|1705555266.187|FETCH|rdkafka#consumer-3| [thrd:kafka-production-0.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-0.kafka-svc.production.svc.cluster.local:9093/0: Fetch 1/1/1 toppar(s)
%7|1705555266.187|SEND|rdkafka#consumer-3| [thrd:kafka-production-0.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-0.kafka-svc.production.svc.cluster.local:9093/0: Sent FetchRequest (v11, 108 bytes @ 0, CorrId 530)
%7|1705555266.275|RECV|rdkafka#consumer-8| [thrd:kafka-production-0.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-0.kafka-svc.production.svc.cluster.local:9093/0: Received FetchResponse (v11, 73 bytes, CorrId 532, rtt 501.16ms)
%7|1705555266.275|FETCH|rdkafka#consumer-8| [thrd:kafka-production-0.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-0.kafka-svc.production.svc.cluster.local:9093/0: Topic SOL_TWD_OUT [3] MessageSet size 0, error "Success", MaxOffset 5208504, LSO 5208504, Ver 3/3
%7|1705555266.275|FETCH|rdkafka#consumer-8| [thrd:kafka-production-0.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-0.kafka-svc.production.svc.cluster.local:9093/0: Fetch topic SOL_TWD_OUT [3] at offset 5208504 (leader epoch 0, current leader epoch -1, v3)
%7|1705555266.275|FETCH|rdkafka#consumer-8| [thrd:kafka-production-0.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-0.kafka-svc.production.svc.cluster.local:9093/0: Fetch 1/1/3 toppar(s)
%7|1705555266.275|SEND|rdkafka#consumer-8| [thrd:kafka-production-0.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-0.kafka-svc.production.svc.cluster.local:9093/0: Sent FetchRequest (v11, 101 bytes @ 0, CorrId 533)
%7|1705555266.282|RECV|rdkafka#consumer-5| [thrd:kafka-production-0.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-0.kafka-svc.production.svc.cluster.local:9093/0: Received FetchResponse (v11, 73 bytes, CorrId 531, rtt 502.26ms)
%7|1705555266.282|RECV|rdkafka#consumer-11| [thrd:kafka-production-0.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-0.kafka-svc.production.svc.cluster.local:9093/0: Received FetchResponse (v11, 73 bytes, CorrId 531, rtt 502.19ms)
%7|1705555266.282|FETCH|rdkafka#consumer-5| [thrd:kafka-production-0.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-0.kafka-svc.production.svc.cluster.local:9093/0: Topic SOL_TWD_OUT [0] MessageSet size 0, error "Success", MaxOffset 14259034, LSO 14259034, Ver 3/3
%7|1705555266.282|FETCH|rdkafka#consumer-11| [thrd:kafka-production-0.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-0.kafka-svc.production.svc.cluster.local:9093/0: Topic SOL_TWD_OUT [6] MessageSet size 0, error "Success", MaxOffset 8431, LSO 8431, Ver 3/3
%7|1705555266.282|FETCH|rdkafka#consumer-5| [thrd:kafka-production-0.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-0.kafka-svc.production.svc.cluster.local:9093/0: Fetch topic SOL_TWD_OUT [0] at offset 14259034 (leader epoch 0, current leader epoch -1, v3)
%7|1705555266.282|FETCH|rdkafka#consumer-5| [thrd:kafka-production-0.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-0.kafka-svc.production.svc.cluster.local:9093/0: Fetch 1/1/3 toppar(s)
%7|1705555266.282|SEND|rdkafka#consumer-5| [thrd:kafka-production-0.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-0.kafka-svc.production.svc.cluster.local:9093/0: Sent FetchRequest (v11, 101 bytes @ 0, CorrId 532)
%7|1705555266.282|FETCH|rdkafka#consumer-11| [thrd:kafka-production-0.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-0.kafka-svc.production.svc.cluster.local:9093/0: Fetch topic SOL_TWD_OUT [6] at offset 8431 (leader epoch 0, current leader epoch -1, v3)
%7|1705555266.282|FETCH|rdkafka#consumer-11| [thrd:kafka-production-0.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-0.kafka-svc.production.svc.cluster.local:9093/0: Fetch 1/1/3 toppar(s)
%7|1705555266.282|SEND|rdkafka#consumer-11| [thrd:kafka-production-0.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-0.kafka-svc.production.svc.cluster.local:9093/0: Sent FetchRequest (v11, 101 bytes @ 0, CorrId 532)
%7|1705555266.318|RECV|rdkafka#consumer-12| [thrd:kafka-production-1.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-1.kafka-svc.production.svc.cluster.local:9093/1: Received FetchResponse (v11, 73 bytes, CorrId 534, rtt 500.33ms)
%7|1705555266.318|FETCH|rdkafka#consumer-12| [thrd:kafka-production-1.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-1.kafka-svc.production.svc.cluster.local:9093/1: Topic SOL_TWD_OUT [7] MessageSet size 0, error "Success", MaxOffset 12332, LSO 12332, Ver 3/3
%7|1705555266.318|FETCH|rdkafka#consumer-12| [thrd:kafka-production-1.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-1.kafka-svc.production.svc.cluster.local:9093/1: Fetch topic SOL_TWD_OUT [7] at offset 12332 (leader epoch 0, current leader epoch -1, v3)
%7|1705555266.318|FETCH|rdkafka#consumer-12| [thrd:kafka-production-1.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-1.kafka-svc.production.svc.cluster.local:9093/1: Fetch 1/1/3 toppar(s)
%7|1705555266.318|SEND|rdkafka#consumer-12| [thrd:kafka-production-1.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-1.kafka-svc.production.svc.cluster.local:9093/1: Sent FetchRequest (v11, 101 bytes @ 0, CorrId 535)
%7|1705555266.318|RECV|rdkafka#consumer-7| [thrd:kafka-production-2.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-2.kafka-svc.production.svc.cluster.local:9093/2: Received FetchResponse (v11, 73 bytes, CorrId 531, rtt 501.71ms)
%7|1705555266.318|FETCH|rdkafka#consumer-7| [thrd:kafka-production-2.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-2.kafka-svc.production.svc.cluster.local:9093/2: Topic SOL_TWD_OUT [2] MessageSet size 0, error "Success", MaxOffset 5215033, LSO 5215033, Ver 3/3
%7|1705555266.319|FETCH|rdkafka#consumer-7| [thrd:kafka-production-2.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-2.kafka-svc.production.svc.cluster.local:9093/2: Fetch topic SOL_TWD_OUT [2] at offset 5215033 (leader epoch 0, current leader epoch -1, v3)
%7|1705555266.319|FETCH|rdkafka#consumer-7| [thrd:kafka-production-2.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-2.kafka-svc.production.svc.cluster.local:9093/2: Fetch 1/1/3 toppar(s)
%7|1705555266.319|SEND|rdkafka#consumer-7| [thrd:kafka-production-2.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-2.kafka-svc.production.svc.cluster.local:9093/2: Sent FetchRequest (v11, 101 bytes @ 0, CorrId 532)
%7|1705555266.319|RECV|rdkafka#consumer-6| [thrd:kafka-production-1.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-1.kafka-svc.production.svc.cluster.local:9093/1: Received FetchResponse (v11, 73 bytes, CorrId 534, rtt 500.37ms)
%7|1705555266.319|FETCH|rdkafka#consumer-6| [thrd:kafka-production-1.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-1.kafka-svc.production.svc.cluster.local:9093/1: Topic SOL_TWD_OUT [1] MessageSet size 0, error "Success", MaxOffset 5215904, LSO 5215904, Ver 3/3
%7|1705555266.319|FETCH|rdkafka#consumer-6| [thrd:kafka-production-1.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-1.kafka-svc.production.svc.cluster.local:9093/1: Fetch topic SOL_TWD_OUT [1] at offset 5215904 (leader epoch 0, current leader epoch -1, v3)
%7|1705555266.319|FETCH|rdkafka#consumer-6| [thrd:kafka-production-1.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-1.kafka-svc.production.svc.cluster.local:9093/1: Fetch 1/1/3 toppar(s)
%7|1705555266.319|SEND|rdkafka#consumer-6| [thrd:kafka-production-1.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-1.kafka-svc.production.svc.cluster.local:9093/1: Sent FetchRequest (v11, 101 bytes @ 0, CorrId 535)
%7|1705555266.440|RECV|rdkafka#consumer-9| [thrd:kafka-production-1.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-1.kafka-svc.production.svc.cluster.local:9093/1: Received FetchResponse (v11, 73 bytes, CorrId 534, rtt 499.96ms)
%7|1705555266.440|FETCH|rdkafka#consumer-9| [thrd:kafka-production-1.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-1.kafka-svc.production.svc.cluster.local:9093/1: Topic SOL_TWD_OUT [4] MessageSet size 0, error "Success", MaxOffset 5204949, LSO 5204949, Ver 3/3
%7|1705555266.440|FETCH|rdkafka#consumer-9| [thrd:kafka-production-1.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-1.kafka-svc.production.svc.cluster.local:9093/1: Fetch topic SOL_TWD_OUT [4] at offset 5204949 (leader epoch 0, current leader epoch -1, v3)
%7|1705555266.440|FETCH|rdkafka#consumer-9| [thrd:kafka-production-1.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-1.kafka-svc.production.svc.cluster.local:9093/1: Fetch 1/1/3 toppar(s)

We can see above log, fetch topic SOL_TWD_OUT [1] request always fetches 5215904 offset, It should fetch 5215905 offset, Although log say MaxOffset is 5215904, but our kafka dashboard present max offset is greater than 5215904.
I also found that current leader epoch is -1, which smaller thane leader epoch. Is it mean that consumer might get wrong metadata so that get wrong max offset

%7|1705555004.030|TOPBRK|rdkafka#producer-4| [thrd::0/internal]: :0/internal: Topic SOL_TWD_OUT [7]: not joining broker (next broker kafka-production-1.kafka-svc.production.svc.cluster.local:9093/1)
%7|1705555004.030|TOPBRK|rdkafka#producer-4| [thrd::0/internal]: :0/internal: Topic SOL_TWD_OUT [8]: not joining broker (next broker kafka-production-2.kafka-svc.production.svc.cluster.local:9093/2)
%7|1705555004.030|TOPBRK|rdkafka#producer-4| [thrd::0/internal]: :0/internal: Topic SOL_TWD_OUT [0]: leaving broker (0 messages in xmitq, next broker kafka-production-0.kafka-svc.production.svc.cluster.local:9093/0, rktp 0x7fb288c4c050)
%7|1705555004.030|FETCHADD|rdkafka#producer-4| [thrd::0/internal]: :0/internal: Removed SOL_TWD_OUT [0] from active list (6 entries, opv 0): leaving
%7|1705555004.030|BROADCAST|rdkafka#producer-4| [thrd::0/internal]: Broadcasting state change
%7|1705555004.030|TOPBRK|rdkafka#producer-4| [thrd:kafka-production-0.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-0.kafka-svc.production.svc.cluster.local:9093/0: Topic SOL_TWD_OUT [0]: joining broker (rktp 0x7fb288c4c050, 0 message(s) queued)
%7|1705555004.030|TOPBRK|rdkafka#producer-4| [thrd:kafka-production-2.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-2.kafka-svc.production.svc.cluster.local:9093/2: Topic SOL_TWD_OUT [8]: joining broker (rktp 0x7fb288ba6560, 0 message(s) queued)
%7|1705555004.030|FETCHADD|rdkafka#producer-4| [thrd:kafka-production-0.kafka-svc.production.svc.cluster.local:9093/]: kafka-production-0.kafka-svc.production.svc.cluster.local:9093/0: Added SOL_TWD_OUT [0] to active list (1 entries, opv 0, 0 messages queued): joining
%7|1705555004.031|BROADCAST|rdkafka#producer-4| [thrd:kafka-production-0.kafka-svc.production.svc.cluster.local:9093/]: Broadcasting state change
%7|1705555004.030|TOPBRK|rdkafka#producer-4| [thrd::0/internal]: :0/internal: Topic SOL_TWD_OUT [1]: leaving broker (0 messages in xmitq, next broker kafka-production-1.kafka-svc.production.svc.cluster.local:9093/1, rktp 0x7fb288ba8050)
%7|1705555004.031|FETCHADD|rdkafka#producer-4| [thrd::0/internal]: :0/internal: Removed SOL_TWD_OUT [1] from active list (5 entries, opv 0): leaving
%7|1705555004.031|BROADCAST|rdkafka#producer-4| [thrd::0/internal]: Broadcasting state change
%7|1705555004.031|TOPBRK|rdkafka#producer-4| [thrd::0/internal]: :0/internal: Topic SOL_TWD_OUT [2]: leaving broker (0 messages in xmitq, next broker kafka-production-2.kafka-svc.production.svc.cluster.local:9093/2, rktp 0x7fb288ba8590)
%7|1705555004.031|FETCHADD|rdkafka#producer-4| [thrd::0/internal]: :0/internal: Removed SOL_TWD_OUT [2] from active list (4 entries, opv 0): leaving
%7|1705555004.031|BROADCAST|rdkafka#producer-4| [thrd::0/internal]: Broadcasting state change
%7|1705555004.031|TOPBRK|rdkafka#producer-4| [thrd::0/internal]: :0/internal: Topic SOL_TWD_OUT [3]: leaving broker (0 messages in xmitq, next broker kafka-production-0.kafka-svc.production.svc.cluster.local:9093/0, rktp 0x7fb288ba8ad0)
%7|1705555004.031|FETCHADD|rdkafka#producer-4| [thrd::0/internal]: :0/internal: Removed SOL_TWD_OUT [3] from active list (3 entries, opv 0): leaving
%7|1705555004.031|BROADCAST|rdkafka#producer-4| [thrd::0/internal]: Broadcasting state change
%7|1705555004.031|TOPBRK|rdkafka#producer-4| [thrd::0/internal]: :0/internal: Topic SOL_TWD_OUT [4]: leaving broker (0 messages in xmitq, next broker kafka-production-1.kafka-svc.production.svc.cluster.local:9093/1, rktp 0x7fb288ba7010)
%7|1705555004.031|FETCHADD|rdkafka#producer-4| [thrd::0/internal]: :0/internal: Removed SOL_TWD_OUT [4] from active list (2 entries, opv 0): leaving
%7|1705555004.031|BROADCAST|rdkafka#producer-4| [thrd::0/internal]: Broadcasting state change
%7|1705555004.031|TOPBRK|rdkafka#producer-4| [thrd::0/internal]: :0/internal: Topic SOL_TWD_OUT [5]: leaving broker (0 messages in xmitq, next broker kafka-production-2.kafka-svc.production.svc.cluster.local:9093/2, rktp 0x7fb288ba7550)
%7|1705555004.031|FETCHADD|rdkafka#producer-4| [thrd::0/internal]: :0/internal: Removed SOL_TWD_OUT [5] from active list (1 entries, opv 0): leaving
%7|1705555004.031|BROADCAST|rdkafka#producer-4| [thrd::0/internal]: Broadcasting state change
%7|1705555004.031|TOPBRK|rdkafka#producer-4| [thrd::0/internal]: :0/internal: Topic SOL_TWD_OUT [6]: leaving broker (0 messages in xmitq, next broker kafka-production-0.kafka-svc.production.svc.cluster.local:9093/0, rktp 0x7fb288ba7a90)
%7|1705555004.031|FETCHADD|rdkafka#producer-4| [thrd::0/internal]: :0/internal: Removed SOL_TWD_OUT [6] from active list (0 entries, opv 0): leaving
%7|1705555004.031|BROADCAST|rdkafka#producer-4| [thrd::0/internal]: Broadcasting state change

According to above log, consumer is leaving from active list after consumer connected.

• confluent-kafka-go version : github.com/confluentinc/confluent-kafka-go/v2 v2.1.0

• Apache Kafka broker version : 3.2.1 with java8

• clientCofnig :
  socket.timeout.ms: "180000"
  session.timeout.ms: "2000"
  max.poll.interval.ms: "300000"
  enable.auto.commit: true
  enable.auto.offset.store: false

• os: alpine3.16

How to reproduce

Checklist

Please provide the following information:

• confluent-kafka-go and librdkafka version (LibraryVersion()):
• Apache Kafka broker version:
• Client configuration: ConfigMap{...}
• Operating system:
• Provide client logs (with "debug": ".." as necessary)
• Provide broker log excerpts
• Critical issue