Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MarkOffset then exiting with AutoCommit false still commits offset #1893

Closed
oss92 opened this issue Mar 2, 2021 · 1 comment
Closed

MarkOffset then exiting with AutoCommit false still commits offset #1893

oss92 opened this issue Mar 2, 2021 · 1 comment

Comments

@oss92
Copy link

oss92 commented Mar 2, 2021

Versions

Please specify real version numbers or git SHAs, not just "Latest" since that changes fairly regularly.

Sarama Kafka Go
v1.27.0 x2.6.x go1.14.4 darwin/amd64

Configuration

What configuration values are you using for Sarama and Kafka?

saramaCfg := sarama.NewConfig()
// client is responsible for committing
saramaCfg.Consumer.Offsets.AutoCommit.Enable = false
saramaCfg.Consumer.Offsets.Initial = sarama.OffsetOldest
saramaCfg.Consumer.Group.Rebalance.Strategy = sarama.BalanceStrategySticky
saramaCfg.Net.DialTimeout = 15 * time.Second
saramaCfg.Version = sarama.V2_6_0_0

Problem Description

I have a consumer and even if message processing fails I call MarkOffset but I do not commit.

Expected Behavior
The consumer would not commit any offset and the lag should be greater than or equal to 1. The current instance should continue processing next messages but if I restart the consumer it should process from the offset that was not committed.

And this is according to the documentation

// MarkOffset marks the provided offset, alongside a metadata string
// that represents the state of the partition consumer at that point in time. The
// metadata string can be used by another consumer to restore that state, so it
// can resume consumption.
//
// To follow upstream conventions, you are expected to mark the offset of the
// next message to read, not the last message read. Thus, when calling `MarkOffset`
// you should typically add one to the offset of the last consumed message.
//
// Note: calling MarkOffset does not necessarily commit the offset to the backend
// store immediately for efficiency reasons, and it may never be committed if
// your application crashes. This means that you may end up processing the same
// message twice, and your processing should ideally be idempotent.

Current Behavior
Even though Consumer.Offsets.AutoCommit.Enable is set to false and there is no Commit() call in the code. The offset is committed regardless.

Before calling MarkOffset
Screen Shot 2021-03-02 at 17 59 11

After calling MarkOffset without committing and no auto-commit
Screen Shot 2021-03-02 at 17 59 18

That does not seem like expected functionality. Or am I missing something?

@oss92
Copy link
Author

oss92 commented Mar 2, 2021

Closing as it is duplicate of #1843.
Resolution: issue was fixed in Sarama 1.28.0

@oss92 oss92 closed this as completed Mar 2, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant