Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add finalizers to prevent memory leaking #1

Merged
merged 1 commit into from Aug 28, 2020

Conversation

ains-stripe
Copy link
Contributor

We've seen some processes leaking memory when using this library.
After looking at this issue
klauspost/compress#264

It seems like it was fixed in kafka-go by using finalizers to ensure that Close was called in the wrapped objects when the GC cleans them up, otherwise the goroutines in compress will leak.
https://github.com/segmentio/kafka-go/blob/master/zstd/zstd.go#L39-L47

Copy link
Owner

@mostynb mostynb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix! I will watch the referenced issue to see if this can be simplified later.

I wonder if we should be using WithDecoderConcurrency/WithEncoderConcurrency when creating these?

@mostynb mostynb merged commit 60a87b5 into mostynb:main Aug 28, 2020
@ains-stripe
Copy link
Contributor Author

That's another way of getting around the bug (disabling the concurrency) if you use any at all though you'll need to cleanup the goroutines.

Thanks for merging though! Would you mind tagging a new release?

@mostynb
Copy link
Owner

mostynb commented Aug 28, 2020

Would you mind tagging a new release?

Done.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants