Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Current offset is higher than last offset #2863

Open
vector-mj opened this issue Apr 17, 2024 · 0 comments
Open

[Bug]: Current offset is higher than last offset #2863

vector-mj opened this issue Apr 17, 2024 · 0 comments

Comments

@vector-mj
Copy link

vector-mj commented Apr 17, 2024

Description

After a network incident(without any PVC issue or data loss at disk level) in our data center, we see that the current offset(consumer group offset) of some partitions in some topics is higher than the last offset(last record offset in the partition) of that partition.
I read relevant Kafka documents about this issue, then I noticed that this issue never should happen because when you try to fetch or commit an offset that doesn't exist(higher or lower than Kafka offset range), Kafka returns an OffsetOutOfRangeException
also when there is no committed position in consumer group Kafka will use auto.offset.reset configuration.

Question:

  • Can this issue be from Sarama?
Versions
Sarama Kafka Go
v1.41.3 v3.2.0 1.21
Configuration
// Consumer
config.Consumer.Offsets.Initial = sarama.OffsetOldest
config.Consumer.Group.Rebalance.Strategy = sarama.BalanceStrategySticky
config.Consumer.Group.Rebalance.Timeout = 2 * time.Minute
config.Consumer.Group.Rebalance.Retry.Max = 100
config.Consumer.Return.Errors = true
config.Consumer.MaxWaitTime = time.Second

// Producer
config.Producer.RequiredAcks = sarama.WaitForLocal
config.Producer.Retry.Max = 10
config.Producer.Partitioner = sarama.NewManualPartitioner
config.Producer.Return.Successes = false
config.Producer.Compression = sarama.CompressionGZIP
config.Producer.Return.Errors = true
config.Producer.Flush.Frequency = time.Millisecond * 10
Additional Context

photo_2024-04-28_10-59-13

related issue: #1734

@vector-mj vector-mj changed the title Bug: Current offset is higher than last offset [Bug]: Current offset is higher than last offset Apr 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant