Releases: karafka/karafka
Releases · karafka/karafka
v2.4.2
v2.4.1
- [Enhancement] Allow for usage of producer variants and alternative producers with ActiveJob Jobs (Pro).
- [Enhancement] Support
:earliest
and:latest
inKarafka::Admin#seek_consumer_group
. - [Enhancement] Align configuration attributes mapper with exact librdkafka version used and not master.
- [Maintenance] Use
base64
from RubyGems as it will no longer be part of standard library in Ruby 3.4. - [Fix] Support migrating via aliases and plan with aliases usage.
- [Fix] Active with default set to
false
cannot be overwritten - [Fix] Fix inheritance of ActiveJob adapter
karafka_options
partitioner and dispatch method.
v2.4.0
This release contains BREAKING changes. Make sure to read and apply upgrade notes.
- [Breaking] Drop Ruby
2.7
support. - [Breaking] Drop the concept of consumer group mapping.
- [Breaking]
karafka topics migrate
will now perform declarative topics configuration alignment. - [Breaking] Replace
deserializer
config with#deserializers
in routing to support key and lazy header deserializers. - [Breaking] Rename
Karafka::Serializers::JSON::Deserializer
toKarafka::Deserializers::Payload
to reflect its role. - [Feature] Support custom OAuth providers (with a lot of help from bruce-szalwinski-he and hotelengine.com).
- [Feature] Provide
karafka topics alter
for declarative topics alignment. - [Feature] Introduce ability to use direct assignments (Pro).
- [Feature] Provide consumer piping API (Pro).
- [Feature] Introduce
karafka topics plan
to describe changes that will be applied when migrating. - [Feature] Introduce ability to use custom message key deserializers.
- [Feature] Introduce ability to use custom message headers deserializers.
- [Feature] Provide
Karafka::Admin::Configs
API for cluster and topics configuration management. - [Enhancement] Protect critical
rdkafka
thread executable code sections. - [Enhancement] Assign names to internal threads for better debuggability when on
TTIN
. - [Enhancement] Provide
log_polling
setting to theKarafka::Instrumentation::LoggerListener
to silence polling in any non-debug mode. - [Enhancement] Provide
metadata#message
to be able to retrieve message from metadata. - [Enhancement] Include number of attempts prior to DLQ message being dispatched including the dispatch one (Pro).
- [Enhancement] Provide ability to decide how to dispatch from DLQ (sync / async).
- [Enhancement] Provide ability to decide how to mark as consumed from DLQ (sync / async).
- [Enhancement] Allow for usage of a custom Appsignal namespace when logging.
- [Enhancement] Do not run periodic jobs when LRJ job is running despite polling (LRJ can still start when Periodic runs).
- [Enhancement] Improve accuracy of periodic jobs and make sure they do not run too early after saturated work.
- [Enhancement] Introduce ability to async lock other subscription groups polling.
- [Enhancement] Improve shutdown when using long polling setup (high
max_wait_time
). - [Enhancement] Provide
Karafka::Admin#read_lags_with_offsets
for ability to query lags and offsets of a given CG. - [Enhancement] Allow direct assignments granular distribution in the Swarm (Pro).
- [Enhancement] Add a buffer to the supervisor supervision on shutdown to prevent a potential race condition when signal pass lags.
- [Enhancement] Provide ability to automatically generate and validate fingerprints of encrypted payload.
- [Enhancement] Support
enable.partition.eof
fast yielding. - [Enhancement] Provide
#mark_as_consumed
and#mark_as_consumed!
to the iterator. - [Enhancement] Introduce graceful
#stop
to the iterator instead of recommending of usage ofbreak
. - [Enhancement] Do not run jobs schedulers and other interval based operations on each job queue unlock.
- [Enhancement] Publish listeners status lifecycle events.
- [Enhancement] Use proxy wrapper for Admin metadata requests.
- [Enhancement] Use limited scope topic info data when operating on direct topics instead of full cluster queries.
- [Enhancement] No longer raise
Karafka::UnsupportedCaseError
for not recognized error types to support dynamic errors reporting. - [Change] Do not create new proxy object to Rdkafka with certain low-level operations and re-use existing.
- [Change] Update
karafka.erb
template with a placeholder for waterdrop and karafka error instrumentation. - [Change] Replace
statistics.emitted.error
error type withcallbacks.statistics.error
to align naming conventions. - [Fix] Pro Swarm liveness listener can report incorrect failure when dynamic multiplexing scales down.
- [Fix] K8s liveness listener can report incorrect failure when dynamic multiplexing scales down.
- [Fix] Fix a case where connection conductor would not be released during manager state changes.
- [Fix] Make sure, that all
Admin
operations go through stabilization proxy. - [Fix] Fix an issue where coordinator running jobs would not count periodic jobs and revocations.
- [Fix] Fix a case where critically crashed supervisor would raise incorrect error.
- [Fix] Re-raise critical supervisor errors before shutdown.
- [Fix] Fix a case when right-open (infinite) swarm matching would not pass validations.
- [Fix] Make
#enqueue_all
output compatible withActiveJob.perform_all_later
(oozzal) - [Fix] Seek consumer group on a topic level is updating only recent partition.
Upgrade Notes
PLEASE MAKE SURE TO READ AND APPLY THEM!
Available here.
v2.4.0.rc1
This release contains BREAKING changes. Make sure to read and apply upgrade notes.
- [Breaking] Drop Ruby
2.7
support. - [Breaking] Drop the concept of consumer group mapping.
- [Breaking]
karafka topics migrate
will now perform declarative topics configuration alignment. - [Breaking] Replace
deserializer
config with#deserializers
in routing to support key and lazy header deserializers. - [Breaking] Rename
Karafka::Serializers::JSON::Deserializer
toKarafka::Deserializers::Payload
to reflect its role. - [Feature] Support custom OAuth providers (with a lot of help from bruce-szalwinski-he and hotelengine.com).
- [Feature] Provide
karafka topics alter
for declarative topics alignment. - [Feature] Introduce ability to use direct assignments (Pro).
- [Feature] Provide consumer piping API (Pro).
- [Feature] Introduce
karafka topics plan
to describe changes that will be applied when migrating. - [Feature] Introduce ability to use custom message key deserializers.
- [Feature] Introduce ability to use custom message headers deserializers.
- [Feature] Provide
Karafka::Admin::Configs
API for cluster and topics configuration management. - [Enhancement] Assign names to internal threads for better debuggability when on
TTIN
. - [Enhancement] Provide
log_polling
setting to theKarafka::Instrumentation::LoggerListener
to silence polling in any non-debug mode. - [Enhancement] Provide
metadata#message
to be able to retrieve message from metadata. - [Enhancement] Include number of attempts prior to DLQ message being dispatched including the dispatch one (Pro).
- [Enhancement] Provide ability to decide how to dispatch from DLQ (sync / async).
- [Enhancement] Provide ability to decide how to mark as consumed from DLQ (sync / async).
- [Enhancement] Allow for usage of a custom Appsignal namespace when logging.
- [Enhancement] Do not run periodic jobs when LRJ job is running despite polling (LRJ can still start when Periodic runs).
- [Enhancement] Improve accuracy of periodic jobs and make sure they do not run too early after saturated work.
- [Enhancement] Introduce ability to async lock other subscription groups polling.
- [Enhancement] Improve shutdown when using long polling setup (high
max_wait_time
). - [Enhancement] Provide
Karafka::Admin#read_lags_with_offsets
for ability to query lags and offsets of a given CG. - [Enhancement] Allow direct assignments granular distribution in the Swarm (Pro).
- [Enhancement] Add a buffer to the supervisor supervision on shutdown to prevent a potential race condition when signal pass lags.
- [Enhancement] Provide ability to automatically generate and validate fingerprints of encrypted payload.
- [Enhancement] Support
enable.partition.eof
fast yielding. - [Enhancement] Provide
#mark_as_consumed
and#mark_as_consumed!
to the iterator. - [Enhancement] Introduce graceful
#stop
to the iterator instead of recommending of usage ofbreak
. - [Enhancement] Do not run jobs schedulers and other interval based operations on each job queue unlock.
- [Enhancement] Publish listeners status lifecycle events.
- [Enhancement] Use proxy wrapper for Admin metadata requests.
- [Enhancement] Use limited scope topic info data when operating on direct topics instead of full cluster queries.
- [Enhancement] No longer raise
Karafka::UnsupportedCaseError
for not recognized error types to support dynamic errors reporting. - [Change] Do not create new proxy object to Rdkafka with certain low-level operations and re-use existing.
- [Change] Update
karafka.erb
template with a placeholder for waterdrop and karafka error instrumentation. - [Fix] Pro Swarm liveness listener can report incorrect failure when dynamic multiplexing scales down.
- [Fix] K8s liveness listener can report incorrect failure when dynamic multiplexing scales down.
- [Fix] Fix a case where connection conductor would not be released during manager state changes.
- [Fix] Make sure, that all
Admin
operations go through stabilization proxy. - [Fix] Fix an issue where coordinator running jobs would not count periodic jobs and revocations.
- [Fix] Fix a case where critically crashed supervisor would raise incorrect error.
- [Fix] Re-raise critical supervisor errors before shutdown.
- [Fix] Fix a case when right-open (infinite) swarm matching would not pass validations.
- [Fix] Make
#enqueue_all
output compatible withActiveJob.perform_all_later
(oozzal) - [Fix] Seek consumer group on a topic level is updating only recent partition.
Upgrade Notes
PLEASE MAKE SURE TO READ AND APPLY THEM!
Available here.
v2.4.0.beta2
This release contains BREAKING changes. Make sure to read and apply upgrade notes.
- [Breaking] Drop Ruby
2.7
support. - [Breaking] Drop the concept of consumer group mapping.
- [Breaking]
karafka topics migrate
will now perform declarative topics configuration alignment. - [Breaking] Replace
deserializer
config with#deserializers
in routing to support key and lazy header deserializers. - [Breaking] Rename
Karafka::Serializers::JSON::Deserializer
toKarafka::Deserializers::Payload
to reflect its role. - [Feature] Support custom OAuth providers (with a lot of help from bruce-szalwinski-he and hotelengine.com).
- [Feature] Provide
karafka topics alter
for declarative topics alignment. - [Feature] Introduce ability to use direct assignments (Pro).
- [Feature] Provide consumer piping API (Pro).
- [Feature] Introduce
karafka topics plan
to describe changes that will be applied when migrating. - [Feature] Introduce ability to use custom message key deserializers.
- [Feature] Introduce ability to use custom message headers deserializers.
- [Feature] Provide
Karafka::Admin::Configs
API for cluster and topics configuration management. - [Enhancement] Assign names to internal threads for better debuggability when on
TTIN
. - [Enhancement] Provide
log_polling
setting to theKarafka::Instrumentation::LoggerListener
to silence polling in any non-debug mode. - [Enhancement] Provide
metadata#message
to be able to retrieve message from metadata. - [Enhancement] Include number of attempts prior to DLQ message being dispatched including the dispatch one (Pro).
- [Enhancement] Provide ability to decide how to dispatch from DLQ (sync / async).
- [Enhancement] Provide ability to decide how to mark as consumed from DLQ (sync / async).
- [Enhancement] Allow for usage of a custom Appsignal namespace when logging.
- [Enhancement] Do not run periodic jobs when LRJ job is running despite polling (LRJ can still start when Periodic runs).
- [Enhancement] Improve accuracy of periodic jobs and make sure they do not run too early after saturated work.
- [Enhancement] Introduce ability to async lock other subscription groups polling.
- [Enhancement] Improve shutdown when using long polling setup (high
max_wait_time
). - [Enhancement] Provide
Karafka::Admin#read_lags_with_offsets
for ability to query lags and offsets of a given CG. - [Enhancement] Allow direct assignments granular distribution in the Swarm (Pro).
- [Enhancement] Add a buffer to the supervisor supervision on shutdown to prevent a potential race condition when signal pass lags.
- [Enhancement] Provide ability to automatically generate and validate fingerprints of encrypted payload.
- [Enhancement] Support
enable.partition.eof
fast yielding. - [Enhancement] Provide
#mark_as_consumed
and#mark_as_consumed!
to the iterator. - [Enhancement] Introduce graceful
#stop
to the iterator instead of recommending of usage ofbreak
. - [Enhancement] Do not run jobs schedulers and other interval based operations on each job queue unlock.
- [Enhancement] Publish listeners status lifecycle events.
- [Enhancement] Use proxy wrapper for Admin metadata requests.
- [Enhancement] Use limited scope topic info data when operating on direct topics instead of full cluster queries.
- [Enhancement] No longer raise
Karafka::UnsupportedCaseError
for not recognized error types to support dynamic errors reporting. - [Change] Do not create new proxy object to Rdkafka with certain low-level operations and re-use existing.
- [Change] Update
karafka.erb
template with a placeholder for waterdrop and karafka error instrumentation. - [Fix] Pro Swarm liveness listener can report incorrect failure when dynamic multiplexing scales down.
- [Fix] K8s liveness listener can report incorrect failure when dynamic multiplexing scales down.
- [Fix] Fix a case where connection conductor would not be released during manager state changes.
- [Fix] Make sure, that all
Admin
operations go through stabilization proxy. - [Fix] Fix an issue where coordinator running jobs would not count periodic jobs and revocations.
- [Fix] Fix a case where critically crashed supervisor would raise incorrect error.
- [Fix] Re-raise critical supervisor errors before shutdown.
- [Fix] Fix a case when right-open (infinite) swarm matching would not pass validations.
- [Fix] Make
#enqueue_all
output compatible withActiveJob.perform_all_later
(oozzal) - [Fix] Seek consumer group on a topic level is updating only recent partition.
Upgrade Notes
PLEASE MAKE SURE TO READ AND APPLY THEM!
Available here.
v2.3.4
v2.4.0.beta1
This release contains BREAKING changes. Make sure to read and apply upgrade notes.
- [Breaking] Drop Ruby
2.7
support. - [Breaking] Drop the concept of consumer group mapping.
- [Breaking]
karafka topics migrate
will now perform declarative topics configuration alignment. - [Breaking] Replace
deserializer
config with#deserializers
in routing to support key and lazy header deserializers. - [Breaking] Rename
Karafka::Serializers::JSON::Deserializer
toKarafka::Deserializers::Payload
to reflect its role. - [Feature] Support custom OAuth providers (with a lot of help from bruce-szalwinski-he and hotelengine.com).
- [Feature] Provide
karafka topics alter
for declarative topics alignment. - [Feature] Introduce ability to use direct assignments (Pro).
- [Feature] Provide consumer piping API (Pro).
- [Feature] Introduce
karafka topics plan
to describe changes that will be applied when migrating. - [Feature] Introduce ability to use custom message key deserializers.
- [Feature] Introduce ability to use custom message headers deserializers.
- [Enhancement] Assign names to internal threads for better debuggability when on
TTIN
. - [Enhancement] Provide
log_polling
setting to theKarafka::Instrumentation::LoggerListener
to silence polling in any non-debug mode. - [Enhancement] Provide
metadata#message
to be able to retrieve message from metadata. - [Enhancement] Include number of attempts prior to DLQ message being dispatched including the dispatch one (Pro).
- [Enhancement] Provide ability to decide how to dispatch from DLQ (sync / async).
- [Enhancement] Provide ability to decide how to mark as consumed from DLQ (sync / async).
- [Enhancement] Allow for usage of a custom Appsignal namespace when logging.
- [Enhancement] Do not run periodic jobs when LRJ job is running despite polling (LRJ can still start when Periodic runs).
- [Enhancement] Improve accuracy of periodic jobs and make sure they do not run too early after saturated work.
- [Enhancement] Introduce ability to async lock other subscription groups polling.
- [Enhancement] Improve shutdown when using long polling setup (high
max_wait_time
). - [Enhancement] Provide
Karafka::Admin#read_lags_with_offsets
for ability to query lags and offsets of a given CG. - [Enhancement] Allow direct assignments granular distribution in the Swarm (Pro).
- [Enhancement] Add a buffer to the supervisor supervision on shutdown to prevent a potential race condition when signal pass lags.
- [Enhancement] Provide ability to automatically generate and validate fingerprints of encrypted payload.
- [Enhancement] Support
enable.partition.eof
fast yielding. - [Enhancement] Provide
#mark_as_consumed
and#mark_as_consumed!
to the iterator. - [Enhancement] Introduce graceful
#stop
to the iterator instead of recommending of usage ofbreak
. - [Enhancement] Do not run jobs schedulers and other interval based operations on each job queue unlock.
- [Enhancement] Publish listeners status lifecycle events.
- [Enhancement] Use proxy wrapper for Admin metadata requests.
- [Enhancement] Use limited scope topic info data when operating on direct topics instead of full cluster queries.
- [Enhancement] No longer raise
Karafka::UnsupportedCaseError
for not recognized error types to support dynamic errors reporting. - [Change] Do not create new proxy object to Rdkafka with certain low-level operations and re-use existing.
- [Change] Update
karafka.erb
template with a placeholder for waterdrop and karafka error instrumentation. - [Fix] Pro Swarm liveness listener can report incorrect failure when dynamic multiplexing scales down.
- [Fix] K8s liveness listener can report incorrect failure when dynamic multiplexing scales down.
- [Fix] Fix a case where connection conductor would not be released during manager state changes.
- [Fix] Make sure, that all
Admin
operations go through stabilization proxy. - [Fix] Fix an issue where coordinator running jobs would not count periodic jobs and revocations.
- [Fix] Fix a case where critically crashed supervisor would raise incorrect error.
- [Fix] Re-raise critical supervisor errors before shutdown.
- [Fix] Fix a case when right-open (infinite) swarm matching would not pass validations.
- [Fix] Make
#enqueue_all
output compatible withActiveJob.perform_all_later
(oozzal)
Upgrade Notes
PLEASE MAKE SURE TO READ AND APPLY THEM!
Available here.
v2.3.3
- [Enhancement] Routing based topics allocation for swarm (Pro)
- [Enhancement] Publish the
-1
shutdown reason status for a non-responding node in swarm. - [Enhancement] Allow for using the
distribution
mode for DataDog listener histogram reporting (Aerdayne). - [Change] Change
internal.swarm.node_report_timeout
to 60 seconds from 30 seconds to compensate for long pollings. - [Fix] Static membership routing evaluation happens too early in swarm.
- [Fix] Close producer in supervisor prior to forking and warmup to prevent invalid memory states.
v2.3.2
- [Feature] Provide swarm capabilities to OSS and Pro.
- [Feature] Provide ability to use complex strategies in DLQ (Pro).
- [Enhancement] Support using
:partition
as the partition key for ActiveJob assignments. - [Enhancement] Expand Logger listener with swarm notifications.
- [Enhancement] Introduce K8s swarm liveness listener.
- [Enhancement] Use
Process.warmup
in Ruby 3.3+ prior to forks (in swarm) and prior to app start. - [Enhancement] Provide
app.before_warmup
event to allow hooking code loading tools prior to final warmup. - [Enhancement] Provide
Consumer#errors_tracker
to be able to get errors that occurred while doing complex recovery. - [Fix] Infinite consecutive error flow with VPs and without DLQ can cause endless offsets accumulation.
- [Fix] Quieting mode causes too early unsubscribe.