v3.2.2

What's Changed

Broker

• [fix][broker] Avoid execute prepareInitPoliciesCacheAsync if namespace is deleted (#22268)
• [fix][broker] Avoid expired unclosed ledgers when checking expired messages by ledger closure time (#22335)
• [fix][broker] Check cursor state before adding it to the waitingCursors (#22191)
• [fix][broker] Close dispatchers stuck due to mismatch between dispatcher.consumerList and dispatcher.consumerSet (#22270)
• [fix][broker] Fix OpReadEntry.skipCondition NPE issue (#22367)
• [fix][broker] Fix ResourceGroup report local usage (#22340)
• [fix][broker] Fix ResourceGroups loading (#21781)
• [fix][broker] Fix issue of field 'topic' is not set when handle GetSchema request (#22377)
• [fix][broker] Fix wrong double-checked locking for readOnActiveConsumerTask in dispatcher (#22279)
• [fix][broker] fix mismatch between dispatcher.consumerList and dispatcher.consumerSet (#22283)
• [fix][ml]Expose ledger timestamp (#22338)
• [fix][offload] Fix Offload readHandle cannot close multi times. (#22162)
• [fix][txn]Fix TopicTransactionBuffer potential thread safety issue (#22149)
• [improve][admin] Fix the createMissingPartitions doesn't response correctly (#22311)
• [improve][broker] Add createTopicIfDoesNotExist option to RawReader constructor (#22264)
• [improve][broker] Add fine-grain authorization to ns/topic management endpoints (#22305)
• [improve][broker] Add missing configuration keys for caching catch-up reads (#22295)
• [improve][broker] Support create RawReader based on configuration (#22280)

Client

• [fix][client] Consumer lost message ack due to race condition in acknowledge with batch message (#22353)
• [fix][client] Do no retrying for error subscription not found when disabled allowAutoSubscriptionCreation (#22164)
• [fix][client] Fix wrong results of hasMessageAvailable and readNext after seeking by timestamp (#22363)
• [fix][client] GenericProtobufNativeSchema not implement getNativeSchema method. (#22204)
• [fix][client] Unclear error message when creating a consumer with two same topics (#22255)
• [fix][client] fix Reader.hasMessageAvailable might return true after seeking to latest (#22201)
• [fix][client]Fixed getting an incorrect maxMessageSize value when accessing multiple clusters in the same process (#22306)

Others

• [fix][misc] Make ConcurrentBitSet thread safe (#22361)
• [improve][misc] Remove the call to sun InetAddressCachePolicy (#22329)
• [fix][ws] Check the validity of config before start websocket service (#22231)

Library updates

• [fix][sec] Upgrade jose4j to 0.9.4 (#22273)
• [fix][sec] Go Functions security updates (#21844)
• [fix][sec] Upgrade Zookeeper to 3.9.2 to address CVE-2024-23944 (#22275)
• [improve][misc] Include native epoll library for Netty for arm64 (#22319)
• [improve][misc] Upgrade Netty version to 4.1.105.Final (#21923)

Tests & CI

• [fix][ci] Enable CI for branch-3.2 (#22287)
• [fix][ci] Tolerate mount option change failing in CI (#22241)
• [fix][test] Fix flaky ManagedLedgerErrorsTest.recoverAfterZnodeVersionError (#22368)
• [fix][test] Fix flaky RGUsageMTAggrWaitForAllMsgsTest (#22252)
• [fix][test] Fix flaky test BrokerServiceAutoSubscriptionCreationTest (#22190)
• [fix][test] Fix flaky test ManagedLedgerTest.testGetNumberOfEntriesInStorage (#22344)
• [fix][test] Fix thread leak in TopicPoliciesAuthZTest (#22257)
• [fix][test][branch-3.2] Fix broken ManagedLedgerTest.testGetNumberOfEntriesInStorage
• [fix][fn] fix broken function-go test (#22260)
• [fix][test] Fix the tests with same namespace name (#22240)
• [fix][build] Fix version in pulsar-bom/pom.xml and fix src/set-project-version.sh
• [improve][test][branch-3.2] Improve ManagedLedgerTest.testGetNumberOfEntriesInStorage
• [improve][misc] Upgrade checkstyle to 10.14.2 (#22291)

For the complete list, check the full changelog.

v3.0.4

What's Changed

Broker

• [fix][broker] Avoid execute prepareInitPoliciesCacheAsync if namespace is deleted (#22268)
• [fix][broker] Avoid expired unclosed ledgers when checking expired messages by ledger closure time (#22335)
• [fix][broker] Check cursor state before adding it to the waitingCursors (#22191)
• [fix][broker] Close dispatchers stuck due to mismatch between dispatcher.consumerList and dispatcher.consumerSet (#22270)
• [fix][broker] Fix OpReadEntry.skipCondition NPE issue (#22367)
• [fix][broker] Fix ResourceGroup report local usage (#22340)
• [fix][broker] Fix ResourceGroups loading (#21781)
• [fix][broker] Fix issue of field 'topic' is not set when handle GetSchema request (#22377)
• [fix][broker] Fix wrong double-checked locking for readOnActiveConsumerTask in dispatcher (#22279)
• [fix][broker] fix mismatch between dispatcher.consumerList and dispatcher.consumerSet (#22283)
• [fix][ml]Expose ledger timestamp (#22338)
• [improve][admin] Fix the createMissingPartitions doesn't response correctly (#22311)
• [improve][broker] Add createTopicIfDoesNotExist option to RawReader constructor (#22264)
• [improve][broker] Add fine-grain authorization to ns/topic management endpoints (#22309)
• [improve][broker] Add missing configuration keys for caching catch-up reads (#22295)
• [improve][broker] Change log level to reduce duplicated logs (#22147)

Client

• [fix][client] Consumer lost message ack due to race condition in acknowledge with batch message (#22353)
• [fix][client] Fix wrong results of hasMessageAvailable and readNext after seeking by timestamp (#22363)
• [fix][client] GenericProtobufNativeSchema not implement getNativeSchema method. (#22204)
• [fix][client] Unclear error message when creating a consumer with two same topics (#22255)
• [fix][client] fix Reader.hasMessageAvailable might return true after seeking to latest (#22201)
• [fix][client]Fixed getting an incorrect maxMessageSize value when accessing multiple clusters in the same process (#22306)
• [improve][client] Add backoff for seek (#20963)

Pulsar IO and Pulsar Functions

• [fix][fn] enable Go function token auth and TLS (#20468)

Others

• [improve][misc] Remove the call to sun InetAddressCachePolicy (#22329)
• [fix][misc] Make ConcurrentBitSet thread safe (#22361)
• [fix][ws] Check the validity of config before start websocket service (#22231)

Library updates

• [fix][sec] Upgrade jose4j to 0.9.4 (#22273)
• [fix][sec] Bump google.golang.org/grpc from 1.38.0 to 1.56.3 in /pulsar-function-go (#21444)
• [fix][sec] Go Functions security updates (#21844)
• [fix][sec] Upgrade Zookeeper to 3.9.2 to address CVE-2024-23944 (#22275)
• [fix][sec] Upgrade prometheus client_golang to v1.12.2 to fix CVE-2022-21698 (#20579)
• [fix][build] Upgrade alluxio version to 2.9.3 to fix CVE-2023-38889 (#21715)
• [improve][misc] Include native epoll library for Netty for arm64 (#22319)
• [improve][misc] Pin Netty version in pulsar-io/alluxio (#21728)
• [improve][misc] Upgrade Netty version to 4.1.105.Final (#21923)

Tests & CI

• [fix][ci] Tolerate mount option change failing in CI (#22241)
• [fix][ci][branch-3.0] Increase Maven's heap size from 1024m to 1500m as it is in master
• [fix][test] Fix flaky ManagedLedgerErrorsTest.recoverAfterZnodeVersionError (#22368)
• [fix][test] Fix flaky RGUsageMTAggrWaitForAllMsgsTest (#22252)
• [fix][test] Fix flaky test BrokerServiceAutoSubscriptionCreationTest (#22190)
• [fix][test] Fix flaky test ManagedLedgerTest.testGetNumberOfEntriesInStorage (#22344)
• [fix][test][branch-3.0] Fix broken ManagedLedgerTest.testGetNumberOfEntriesInStorage
• [fix][test] Fix the tests with same namespace name (#22240)
• [improve][test][branch-3.0] Improve ManagedLedgerTest.testGetNumberOfEntriesInStorage
• [improve][misc] Upgrade checkstyle to 10.14.2 (#22291)

For the complete list, check the full changelog.

v3.0.3

What's Changed

Broker

• [improve][broker] Consistently add fine-grain authorization to REST API 22202

• [improve][broker] Add fine-grain authorization to retention admin API 22163

• [improve][broker] Cache the internal writer when sent to system topic. 22099

• [improve][broker] Add an error log to troubleshoot the failure of starting broker registry. 22065

• [improve][broker] Do not retain the data in the system topic 22022

• [improve] [broker] Do not try to open ML when the topic meta does not exist and do not expect to create a new one. #21995 22004

• [improve] [broker] Do not print an Error log when responding to HTTP-404 when calling Admin API and the topic does not exist. 21995

• [improve][broker] Do not close the socket if lookup failed due to LockBusyException 21993

• [improve][broker] Don't rollover empty ledgers based on inactivity 21893

• [improve][broker] Improve NamespaceUnloadStrategy error message 21880

• [improve][broker] Skip loading the NAR packages if not configured 21867

• [improve][broker] defer the ownership checks if the owner is inactive (ExtensibleLoadManager) 21857

• [improve][broker] defer the ownership checks if the owner is inactive (ExtensibleLoadManager) 21811

• [improve][broker] cleanup the empty subscriptionAuthenticationMap in zk when revoke subscription permission 21696

• [improve] [broker] Let the producer request success at the first time if the previous one is inactive 21220

• [improve][broker] Avoid record inactiveproducers when deduplication is disable. 21193

• [improve][broker] Support not retaining null-key message during topic compaction (#21578) 21662

• [improve][broker] Print recoverBucketSnapshot log if cursorProperties are empty 21651

• [improve][broker][PIP-318] Support not retaining null-key message during topic compaction 21578

• [fix] [broker] print non log when delete partitioned topic failed 22153

• [fix][broker] Fix broker not starting when both transactions and the Extensible Load Manager are enabled 22139

• [fix][broker] Return getOwnerAsync without waiting on source broker upon Assigning and Releasing and handle role change during role init 22112

• [fix] [broker] Enabling batch causes negative unackedMessages due to ack and delivery concurrency 22090

• [fix][broker] Support running docker container with gid != 0 22081

• [fix][broker] Set ServiceUnitStateChannel topic compaction threshold explicitly, improve getOwnerAsync, and fix other bugs 22064

• [fix][broker] Fix hash collision when using a consumer name that ends with a number 22053

• [fix][broker] Sanitize values before logging in apply-config-from-env.py script 22044

• [fix] [broker] Fix can not subscribe partitioned topic with a suffix-matched regexp 22025

• [fix] [broker] Subscription stuck due to called Admin API analyzeSubscriptionBacklog 22019

• [fix][broker][branch-3.1] Avoid PublishRateLimiter use an already closed RateLimiter 22011

• [fix] [broker] [branch-3.0] Fast fix infinite HTTP call createSubscriptions caused by wrong topicName 21997

• [fix][broker]Support setting autoSkipNonRecoverableData dynamically in expiryMon… 21991

• [fix] [broker] add timeout for health check read. 21990

• [fix][broker] Fix schema deletion error when deleting a partitioned topic with many partitions and schema 21977

• [fix] [broker] Fix reader stuck when read from compacted topic with read compact mode disable 21969

• [fix] [broker] Replication stopped due to unload topic failed 21947

• [fix] [broker] Expire messages according to ledger close time to avoid client clock skew 21940

• [fix][broker] Restore the broker id to match the format used in existing Pulsar releases 21937

• [fix][broker] Fix getMessageById throws 500 21919

• [fix] [broker] fix write all compacted out entry into compacted topic 21917

• [fix][broker] Fix deadlock while skip non-recoverable ledgers. 21915

• [fix][broker] Fix leader broker cannot be determined when the advertised address and advertised listeners are configured 21894

• [fix] [broker] Fix break change: could not subscribe partitioned topic with a suffix-matched regexp due to a mistake of PIP-145 21885

• [fix][broker] Fix compaction/replication data loss when expire messages 21865

• [fix][broker]Fix NonPersistentDispatcherMultipleConsumers ArrayIndexOutOfBoundsException 21856

• [fix][broker] Fix returns wrong webServiceUrl when both webServicePort and webServicePortTls are set 21842

• [fix][broker] Fix String wrong format 21829

• [fix] [broker] Update topic policies as much as possible when some ex was thrown 21810

• [fix][broker] Fix TableViewLoadDataStoreImpl NPE 21777

• [fix][broker] Fix the wrong value of BrokerSrevice.maxUnackedMsgsPerDispatcher 21765

• [fix][broker] Fixed the ExtensibleLoadManagerImpl internal system getTopic failure when the leadership changes 21764

• [fix][broker] Delete compacted ledger when topic is deleted 21745

• [fix][broker] Skip topic auto-creation for ExtensibleLoadManager internal topics 21729

• [fix][broker] Fixed ServiceUnitStateChannel monitor to tombstone only inactive bundle states 21721

• [fix][broker] Avoid compaction task stuck when the last message to compact is a marker 21718

• [fix][broker] Fix the issue of topics possibly being deleted. 21704

• [fix][broker] Fix typo in the config key 21690

• [fix] [broker] network package lost if enable haProxyProtocolEnabled 21684

• [fix][broker] Record GeoPersistentReplicator.msgOut before producer#sendAsync 21673

• [fix][broker] Fix memory leak during topic compaction 21647

• [fix][broker] Fix lookupRequestSemaphore leak when topic not found 21646

• [fix][broker] Fix returns wrong webServiceUrl when both webServicePort and webServicePortTls are set 21633

• [fix][broker] Fixed getting incorrect KeyValue schema version 21632
  (#21620)

• [fix] [broker] Delete topic timeout due to NPE 21595

• [fix][broker] Fix incorrect unack count when using shared subscription on non-persistent topic 21592

• [fix][broker] Duplicate LedgerOffloader creation when namespace/topic… 21591

• [fix][broker] Correct schema deletion for partitioned topic 21574

• [fix][broker] Fix setReplicatedSubscriptionStatus incorrect behavior 21510

• [fix][broker] Do not write replicated snapshot marker when the topic which is not enable replication 21495

• [fix][broker] Fix issue with consumer read uncommitted messages from compacted topic 21465

• [fix][broker] Fix resource_quota_zpath 21461

• [fix][broker] Fix MultiRoles token provider NPE when using anonymous clients 21429

• [fix][broker]...

v3.2.1

Broker

• Consistently add fine-grain authorization to REST API #22202
• Add fine-grain authorization to retention admin API #22163
• Print non log when delete partitioned topic failed #22153
• Fix broker not starting when both transactions and the Extensible Load Manager are enabled #22139
• Return getOwnerAsync without waiting on source broker upon Assigning and Releasing and handle role change during role init #22112
• Cache the internal writer when sent to system topic. #22099
• Enabling batch causes negative unackedMessages due to ack and delivery concurrency #22090
• Support running docker container with gid != 0 #22081
• Add an error log to troubleshoot the failure of starting broker registry #22065
• Set ServiceUnitStateChannel topic compaction threshold explicitly, improve getOwnerAsync, and fix other bugs #22064
• Fix hash collision when using a consumer name that ends with a number #22053
• Sanitize values before logging in apply-config-from-env.py script #22044
• Fix can not subscribe partitioned topic with a suffix-matched regexp #22025
• Do not retain the data in the system topic #22022
• Subscription stuck due to called Admin API analyzeSubscriptionBacklog #22019
• Do not try to open ML when the topic meta does not exist and do not expect to create a new one #22004
• Do not print an Error log when responding to HTTP-404 when calling Admin API and the topic does not exist #21995
• Do not close the socket if lookup failed due to LockBusyException #21993
• Support setting autoSkipNonRecoverableData dynamically in PersistentMessageExpiryMonitor #21991
• Add timeout for health check read #21990
• Fix schema deletion error when deleting a partitioned topic with many partitions and schema #21977
• Defer the ownership checks if the owner is inactive (ExtensibleLoadManager) #21811
• Fix compaction/replication data loss when expire messages #21865
• Skip loading the NAR packages if not configured #21867
• Improve NamespaceUnloadStrategy error message #21880
• Don't rollover empty ledgers based on inactivity #21893
• Expire messages according to ledger close time to avoid client clock skew #21940
• Fix reader stuck when read from compacted topic with read compact mode disable #21969

Client

• Fix multi-topics consumer could receive old messages after seek #21945
• Fix ConsumerBuilderImpl#subscribe silent stuck when using pulsar-client:3.0.x with jackson-annotations prior to 2.12.0 #21985

Pulsar IO and Pulsar Functions

• [FN] Add configuration for connector & functions package url sources #22184
• [FN] Add missing "exception" argument to some log.error #22140
• [FN] Prevent putstate uses empty values #22127
• [FN] Optimize Function Worker startup by lazy loading and direct zip/bytecode access #22122
• [FN] Prevent create state table from API calls for non-exists instances #22107
• [FN] Use unified PackageManagement service to download packages #21955

CLI

• [Admin] Expose the offload threshold in seconds to the admin #22101
• [CLI] Fix the bug when set-retention specified size with -T #22150
• [CLI] Error out when --destinationBroker is set when --bundle is empty #21879

Others

• [MISC] Add a check for the input time value #22023
• [Proxy] Add a check for brokerServiceURL that does not support multi uri yet #21972
• [TX] Get previous position by managed ledger #22024
• [ML] Fix retry mechanism of deleting ledgers to invalidate #21869
• [ML] Filter out deleted entries before read entries from ledger #21739

Library updates

• Upgrade Jetty to 9.4.54.v20240208 to address CVE-2024-22201 #22144
• pgrade commons-compress to 1.26.0 #22086
• Upgrade BookKeeper dependency to 4.16.4 #21983

A special thanks to the following contributors who contributed to Pulsar 3.2.1:
mattisonchao, lhotari, poorbarcode, Technoboy-, jiangpengcheng, dragosvictor, freeznet, liangyepianzhou, heesung-sn, zymap, massakam, thetumbled, BewareMyPower, chenhongSZ, Shawyeok, hangc0276, 315157973, coderzc, dao-jun.

v3.1.3

Broker

• [improve][broker] Consistently add fine-grain authorization to REST API #22202 (@mattisonchao)
• [improve][broker] Add fine-grain authorization to retention admin API #22163 (@mattisonchao)
• [improve][broker] Cache the internal writer when sent to system topic. #22099 (@Technoboy-)
• [improve][broker] Add an error log to troubleshoot the failure of starting broker registry. #22065 (@liangyepianzhou)
• [improve][broker] Do not retain the data in the system topic #22022 (@liangyepianzhou)
• [improve][broker] Do not try to open ML when the topic meta does not exist and do not expect to create a new one. #21995 #22004 (@poorbarcode)
• [improve][broker] Do not print an Error log when responding to HTTP-404 when calling Admin API and the topic does not exist. #21995 (@poorbarcode)
• [improve][broker] Do not close the socket if lookup failed due to LockBusyException #21993 (@BewareMyPower)
• [improve][broker] Don't rollover empty ledgers based on inactivity #21893 (@lhotari)
• [improve][broker] Improve NamespaceUnloadStrategy error message #21880 (@dragosvictor)
• [improve][broker] Skip loading the NAR packages if not configured #21867 (@BewareMyPower)
• [improve][broker] defer the ownership checks if the owner is inactive (ExtensibleLoadManager) #21811 (@heesung-sn)
• [improve][broker] cleanup the empty subscriptionAuthenticationMap in zk when revoke subscription permission #21696 (@TakaHiR07)
• [improve][broker] Not close the socket if lookup failed caused by bundle unloading or metadata ex #21211 (@poorbarcode)
• [fix][broker][branch-3.1] Fix broker not starting when both transactions and the Extensible Load Manager are enabled #22194 (@dragosvictor)
• [fix][broker] print non log when delete partitioned topic failed #22153 (@poorbarcode)
• [fix][broker] Return getOwnerAsync without waiting on source broker upon Assigning and Releasing and handle role change during role init #22112 (@heesung-sn)
• [fix][broker] Enabling batch causes negative unackedMessages due to ack and delivery concurrency #22090 (@poorbarcode)
• [fix][broker] Support running docker container with gid != 0 #22081 (@lhotari)
• [fix][broker] Set ServiceUnitStateChannel topic compaction threshold explicitly, improve getOwnerAsync, and fix other bugs #22064 (@heesung-sn)
• [fix][broker] Fix hash collision when using a consumer name that ends with a number #22053 (@lhotari)
• [fix][broker] Sanitize values before logging in apply-config-from-env.py script #22044 (@lhotari)
• [fix][broker] Fix can not subscribe partitioned topic with a suffix-matched regexp #22025 (@poorbarcode)
• [fix][broker] Subscription stuck due to called Admin API analyzeSubscriptionBacklog #22019 (@poorbarcode)
• [fix][broker][branch-3.1] Avoid PublishRateLimiter use an already closed RateLimiter #22011 (@coderzc)
• [fix][broker] Support setting autoSkipNonRecoverableData dynamically in expiryMon… #21991 (@chenhongSZ)
• [fix][broker] add timeout for health check read. #21990 (@thetumbled)
• [fix][broker] Fix schema deletion error when deleting a partitioned topic with many partitions and schema #21977 (@heesung-sn)
• [fix][broker] Fix reader stuck when read from compacted topic with read compact mode disable #21969 (@thetumbled)
• [fix][broker] Replication stopped due to unload topic failed #21947 (@poorbarcode)
• [fix][broker] Expire messages according to ledger close time to avoid client clock skew #21940 (@315157973)
• [fix][broker] Restore the broker id to match the format used in existing Pulsar releases #21937 (@lhotari)
• [fix][broker] Fix getMessageById throws 500 #21919 (@nodece)
• [fix][broker] fix write all compacted out entry into compacted topic #21917 (@thetumbled)
• [fix][broker] Fix deadlock while skip non-recoverable ledgers. #21915 (@hrzzzz)
• [fix][broker] Fix leader broker cannot be determined when the advertised address and advertised listeners are configured #21894 (@lhotari)
• [fix][broker] Fix break change: could not subscribe partitioned topic with a suffix-matched regexp due to a mistake of PIP-145 #21885 (@poorbarcode)
• [fix][broker] Fix compaction/replication data loss when expire messages #21865 (@coderzc)
• [fix][broker] Fix NonPersistentDispatcherMultipleConsumers ArrayIndexOutOfBoundsException #21856 (@Technoboy-)
• [fix][broker] Fix returns wrong webServiceUrl when both webServicePort and webServicePortTls are set #21842 (@Technoboy-)
• [fix][broker] Fix TableViewLoadDataStoreImpl NPE #21777 (@Demogorgon314)
• [fix][broker] Fix the wrong value of BrokerSrevice.maxUnackedMsgsPerDispatcher #21765 (@aloyszhang)
• [fix][broker] Fixed the ExtensibleLoadManagerImpl internal system getTopic failure when the leadership changes #21764 (@heesung-sn)
• [fix][broker] Delete compacted ledger when topic is deleted #21745 (@coderzc)
• [fix][broker] Skip topic auto-creation for ExtensibleLoadManager internal topics #21729 (@heesung-sn)
• [fix][broker] Avoid compaction task stuck when the last message to compact is a marker #21718 (@coderzc)
• [fix][broker] Fix the issue of topics possibly being deleted. #21704 (@crossoverJie)
• [fix][broker] network package lost if enable haProxyProtocolEnabled #21684 (@poorbarcode)
• [fix][broker] Fix memory leak during topic compaction #21647 (@coderzc)
• [fix][broker] Fix lookupRequestSemaphore leak when topic not found #21646 (@pengxiangrui127)
• [fix][broker] Fix returns wrong webServiceUrl when both webServicePort and webServicePortTls are set #21633 (@coderzc)
• [fix][broker] Fixed getting incorrect KeyValue schema version #21632 (@Technoboy-)
• [fix][broker] Fix issue with GetMessageIdByTimestamp can't find match messageId from compacted ledger #21600 (@coderzc)
• [fix][broker] Fix incorrect unack count when using shared subscription on non-persistent topic #21592 (@1Jack2)
• [fix][broker] Correct schema deletion for partitioned topic #21574 (@poorbarcode)
• [fix][broker] Avoid consumers receiving acknowledged messages from compacted topic after reconnection #21187 (@coderzc)
• [fix][broker] Fix PulsarService.getLookupServiceAddress returns wrong port if TLS is enabled #21015 (@Technoboy-)
• [fix][txn] Fix TopicTransactionBuffer potential thread safety issue #22149 (@dao-jun)
• [fix][txn] Get previous position by managed ledger. #22024 (@thetumbled)
• [fix][txn] Fix getting last message ID when there are ongoing transactions #21466 (@liangyepianzhou)
• [fix][offload] Fix Offload readHandle cannot close multi times. #22162 (@dao-jun)
• [improve][ml] Filter out deleted entries before read entries from ledger. #21739 (@dao-jun)
• [fix][ml] Fix retry mechanism of deleting ledgers to invalidate #21869 (@poorbarcode)
• [fix][ml] Make mlOwnershipChecker asynchronous so that it doesn't block/deadlock threads #21333 (@lhotari)

Client

• [improve][client] Prevent reserve memory with a negative memory size to avoid send task stuck #21804 ...

v2.11.4

What's Changed

Broker

• [fix][broker] Fix break change: could not subscribe partitioned topic with a suffix-matched regexp due to a mistake of PIP-145 (#21885)
• [fix][broker] Fix String wrong format (#21829)
• [fix][broker] Enabling batch causes negative unackedMessages due to ack and delivery concurrency (#22090)
• [fix][broker] Fix can not subscribe partitioned topic with a suffix-matched regexp (#22025)
• [fix][broker] Fix lookupRequestSemaphore leak when topic not found (#21646)
• [fix][broker] Fix the issue of topics possibly being deleted. (#21704)
• [fix][broker] Fix write all compacted out entry into compacted topic (#21917)
• [fix][broker] Replication stopped due to unload topic failed (#21947)
• [fix][broker] Sanitize values before logging in apply-config-from-env.py script (#22044)
• [fix][broker] Support running docker container with gid != 0 (#22081)
• [fix][broker] Update topic policies as much as possible when some ex was thrown (#21810)
• [fix][broker] fix the wrong value of BrokerSrevice.maxUnackedMsgsPerDispatcher (#21765)
• [fix][broker]Delete compacted ledger when topic is deleted (#21745) (#21850)
• [fix][broker]Fix NonPersistentDispatcherMultipleConsumers ArrayIndexOutOfBoundsException (#21856)
• [fix][broker][branch-3.1] Avoid PublishRateLimiter use an already closed RateLimiter (#22011)
• [fix][ml] Fix retry mechanism of deleting ledgers to invalidate (#21869)
• [improve][admin] internalGetMessageById shouldn't be allowed on partitioned topic (#19013)
• [improve][broker] Add fine-grain authorization to retention admin API (#22163)
• [improve][broker] Avoid print redirect exception log when get list from bundle (#20846)
• [improve][broker] Consistently add fine-grain authorization to REST API (#22202)
• [improve][broker] Do not print an Error log when responding to HTTP-404 when calling Admin API and the topic does not exist. (#21995)
• [improve][broker] Make get list from bundle Admin API async (#20652)
• [refactor][broker] Suppress error logging when message expiration fails (#19778)

Misc

• [improve][build] Upgrade Apache ZooKeeper to 3.9.1 (#20933)
• [fix][misc] Bump GRPC version to 1.55.3 to fix CVE (#21057)
• [fix][sec] Bump avro version to 1.11.3 for CVE-2023-39410 (#21341)
• [fix][sec] Exclude avro from hadoop-client (#21719)
• [fix][sec] Upgrade Jetty to 9.4.53 to address CVE-2023-44487 (#21395)
• [fix][sec] Upgrade Jetty to 9.4.54.v20240208 to address CVE-2024-22201 (#22144)
• [fix][sec] Upgrade Netty to 4.1.100 to address CVE-2023-44487 (#21397)
• [fix][sec] Upgrade commons-compress to 1.26.0 (#22086)
• [fix][sec] cve: exclude ch.qos.logback in canal.protocol * resolve CVE-2023-6378 (95e1de7)
• [fix][test] Fix PerformanceProducer send count error (#21706)
• [fix][test] Fix test testTransactionBufferMetrics (#22117)
• [fix][test] Make base test class method protected so that it passes ReportUnannotatedMethods validation (#21976)
• [fix][test] testModularLoadManagerRemoveBundleAndLoad (#19710)

Proxy

• [improve][proxy] Fix comment about enableProxyStatsEndpoints (#21757)

Functions

• [fix][fn] Fix Deadlock in Functions Worker LeaderService (#21711)
• [improve][fn] Add configuration for connector & functions package url sources (#22184)
• [improve][fn] Optimize Function Worker startup by lazy loading and direct zip/bytecode access (#22122)

Clients

• [fix][client] fix huge permits if acked a half batched message (#22091)

Full Changelog: v2.11.3...v2.11.4

v2.10.6

Broker

Improvements

• [improve][broker] Consistently add fine-grain authorization to REST API 22202
• [improve] [broker] Do not print an Error log when responding to HTTP-404 when calling Admin API and the topic does not exist. 21995
• [improve][proxy] Fix comment about enableProxyStatsEndpoints 21757
• [improve][broker][PIP-318] Support not retaining null-key message during topic compaction 21578
• [improve] [client] Merge lookup requests for the same topic 21232
• [improve] [broker] Let the producer request success at the first time if the previous one is inactive 21220
• [improve] [proxy] Not close the socket if lookup failed caused by too many requests 21216
• [improve] [broker] Not close the socket if lookup failed caused by bundle unloading or metadata ex 21211
• [improve] [broker] Print warn log if ssl handshake error & print ledger id when switch ledger 21201
• [improve] [broker] improve read entry error log for troubleshooting 21169
• [improve][proxy] Support disabling metrics endpoint 21031
• [improve][broker] Improve cache handling for partitioned topic metadata when doing lookup 21063
• [improve][broker] Avoid print redirect exception log when get list from bundle 20846
• [improve][sql] Fix the wrong format of the logs 20907
• [improve][broker] Add consumer-id into the log when doing subscribe. 20568
• [improve] [broker] Print warn log if compaction failure 19405
• [improve][admin]internalGetMessageById shouldn't be allowed on partitioned topic 19013

Fixes

• [fix][sec] Upgrade Jetty to 9.4.54.v20240208 to address CVE-2024-22201 22144
• [fix][sec] Upgrade commons-compress to 1.26.0 22086
• [fix][broker] Support running docker container with gid != 0 22081
• [fix][broker] Sanitize values before logging in apply-config-from-env.py script 22044
• [fix][broker][branch-3.1] Avoid PublishRateLimiter use an already closed RateLimiter 22011
• [fix] [broker] Replication stopped due to unload topic failed 21947
• [fix] [broker] fix write all compacted out entry into compacted topic 21917
• [fix] [ml] Fix retry mechanism of deleting ledgers to invalidate 21869
• [fix][broker]Fix NonPersistentDispatcherMultipleConsumers ArrayIndexOutOfBoundsException 21856
• [fix][broker] Fix String wrong format 21829
• [fix] [broker] Update topic policies as much as possible when some ex was thrown 21810
• [fix][broker] Fix the wrong value of BrokerSrevice.maxUnackedMsgsPerDispatcher 21765
• [fix][sec] Exclude avro from hadoop-client 21719
• [fix][test] Fix PerformanceProducer send count error 21706
• [fix][broker] Fix the issue of topics possibly being deleted. 21704
• [fix][broker] Fix typo in the config key 21690
• [fix] [broker] network package lost if enable haProxyProtocolEnabled 21684
• [fix][broker] Fix memory leak during topic compaction 21647
• [fix][broker] Duplicate LedgerOffloader creation when namespace/topic… 21591
• [fix][broker] Correct schema deletion for partitioned topic 21574
• [fix][broker] Fix namespace bundle stuck in unloading status (#21445) 21567
• [fix][broker] Fix create topic with different auto creation strategies causes race condition 21545
• [fix] [ml] Fix orphan scheduled task for ledger create timeout check 21542
• [fix] [broker] Fix thousands orphan PersistentTopic caused OOM 21540
• [fix][ml] Fix unfinished callback when deleting managed ledger 21530
• [fix][broker] Fix setReplicatedSubscriptionStatus incorrect behavior 21510
• [fix][broker] Do not write replicated snapshot marker when the topic which is not enable replication 21495
• [branch-2.10][fix][proxy] Move status endpoint out of auth coverage 21494
• [fix][broker] Avoid pass null role in MultiRolesTokenAuthorizationProvider 21486
• [fix][broker] Fix the deadlock when using BookieRackAffinityMapping with rackaware policy 21481
• [fix][broker] Fix namespace bundle stuck in unloading status 21445
• [fix][broker] Fix MultiRoles token provider NPE when using anonymous clients 21429
• [fix][proxy] Move status endpoint out of auth coverage 21428
• [fix][sec] Upgrade Netty to 4.1.100 to address CVE-2023-44487 21397
• [fix][sec] Bump avro version to 1.11.3 for CVE-2023-39410 21341
• [fix][test] Fix flaky test NarUnpackerTest 21328
• [fix] [bk-client] Fix bk client MinNumRacksPerWriteQuorum and EnforceMinNumRacksPerWriteQuorum not work problem. 21327
• [fix][ml] Fix thread safe issue with RangeCache.put and RangeCache.clear 21302
• [fix][sec] Upgrade snappy-java to 1.1.10.5 21280
• [fix][txn] Ack all message ids when ack chunk messages with transaction. 21268
• [fix][broker][branch-2.10] Fix inconsistent topic policy 21258
• [fix] [ml] fix wrong msg backlog of non-durable cursor after trim ledgers 21250
• [fix] [ml] Reader can set read-pos to a deleted ledger 21248
• [fix][broker]Fixed produce and consume when anonymousUserRole enabled 21237
• [fix][broker] Fix inconsistent topic policy 21231
• [fix][broker] Fixed reset for AggregatedNamespaceStats 21225
• [fix][broker] Make the new exclusive consumer instead the inactive one faster 21183
• [fix][broker][branch-2.10] Backport fix UniformLoadShedder selecet wrong overloadbroker and underloadbroker 21182
• [Branch-2.10] [imporve] [bookie] Upgrade BookKeeper dependency to 4.14.8 for branch 2.10 21148
• [fix][client] Fix repeat consume when using n-ack and batched messages 21116
• [fix][client] Avoid ack hole for chunk message 21101
• [branch-2.10] [fix] [broker] Fix isolated group not work problem. 21098
• [fix] [broker] Fix isolated group not work problem. 21096
• [fix][broker] Fix write duplicate entries into the compacted ledger after RawReader reconnects 21081
• [fix][client] Fix consumer can't consume resent chunked messages 21070
• [fix][broker] Make sure all inflight writes have finished before completion of compaction 21067
• [fix][broker] Use MessageDigest.isEqual when comparing digests 21061
• [fix][misc] Bump GRPC version to 1.55.3 to fix CVE 21057
• [fix] [bk] Correctct the bookie info after ZK client is reconnected 21035
• [fix][io] Update test certs for Elasticsearch 21001
• [fix][broker] Fix can't stop phase-two of compaction even though messageId read reaches lastReadId 20988
• [fix][broker] Fix message loss during topic compaction [20980](https://github.com/apache/pulsar/p...

v3.2.0

PIPs

• PIP-323: Complete Backlog Quota Telemetry #21709
• PIP-322: Pulsar Rate Limiting Refactoring #21680
• PIP-320: OpenTelemetry Scaffolding #21635
• PIP-318: Don't retain null-key messages during topic compaction #21541
• PIP-315: Configurable max delay limit for delayed delivery #21490
• PIP-313: Support force unsubscribe using consumer api #21452
• PIP-312:
  • Use StateStoreProvider to manage state in Pulsar Functions endpoints #21597
  • Use StateStoreProvider to manage state in Pulsar Functions endpoints #21438
• PIP-307:
  • Support subscribing multi topics for WebSocket #21390
  • Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager #20748
• PIP-305: Customize DNS servers to use for Pulsar Client #21352
• PIP-303: Add optional parameters for getPartitionedStats #21228
• PIP-302: Introduce refreshAsync API for TableView #21271
• PIP-301: Introduce LoadBalanceResources to unify the load-date CRUD #21129
• PIP-300: Add custom dynamic configuration for plugins #21127
• PIP-299: Stop dispatch messages if the individual acks will be lost in the persistent storage #21118
• PIP-297: Support terminating Function & Connector with the fatal exception #21079
• PIP-293: Delete config disableBrokerInterceptors #20993
• PIP-290: Provide a way to implement WSS E2E encryption and not need to expose the private key to the WebSocket Proxy #20923
• PIP-286: Make the TopicCompactionService to support find entry based on publishTime or index #20867
• PIP-277: Add current option in the Clusters list cmd #20614
• PIP-264: Enhanced OTel-based metric system #21080

Broker

• Replication stopped due to unload topic failed #21947
• Fix getMessageById throws 500 #21919
• Fix write all compacted out entry into compacted topic #21917
• Fix deadlock while skip non-recoverable ledgers #21915
• Fix leader broker cannot be determined when the advertised address and advertised listeners are configured #21894
• Fix DLQ producer name conflicts when multiples consumers send messages to DLQ #21890
• Fix break change: could not subscribe partitioned topic with a suffix-matched regexp due to a mistake of PIP-145 #21885
• Fix NonPersistentDispatcherMultipleConsumers ArrayIndexOutOfBoundsException #21856
• Fix returns wrong webServiceUrl when both webServicePort and webServicePortTls are set #21842
• Fix the wrong value of BrokerSrevice.maxUnackedMsgsPerDispatcher #21765
• Fix the ExtensibleLoadManagerImpl internal system getTopic failure when the leadership changes #21764
• Fix closing of AbstractDispatcherSingleActiveConsumer and reduce flakiness of test #21736
• Skip topic auto-creation for ExtensibleLoadManager internal topics #21729
• Fix ServiceUnitStateChannel monitor to tombstone only inactive bundle states #21721
• Avoid compaction task stuck when the last message to compact is a marker #21718
• Fix the issue of topics possibly being deleted #21704
• CheckTopicExists supports checking partitioned topic without index #21701
• Remove internalGetListAsync and validatePartitionTopicUpdateAsync #21699
• Cleanup the empty subscriptionAuthenticationMap in zk when revoke subscription permission #21696
• Fix typo in the config key #21690
• network package lost if enable haProxyProtocolEnabled #21684
• Pulsar Rate Limiting Refactoring changes (PIP-322) #21681
• Record GeoPersistentReplicator.msgOut before producersendAsync #21673
• Print recoverBucketSnapshot log if cursorProperties are empty #21651
• Fix memory leak during topic compaction #21647
• Fix lookupRequestSemaphore leak when topic not found #21646
• Fix returns wrong webServiceUrl when both webServicePort and webServicePortTls are set #21633
• Fix getting incorrect KeyValue schema version #21632
• Add optional parameters for getPartitionedStats #21611
• PIP-301 Part-3: Add QuotaResources #21596
• Delete topic timeout due to NPE #21595
• Fix incorrect unack count when using shared subscription on non-persistent topic #21592
• Duplicate LedgerOffloader creation when namespace/topic… #21591
• Support not retaining null-key message during topic compaction #21578
• Correct schema deletion for partitioned topic #21574
• Fix create topic with different auto creation strategies causes race condition #21545
• Fix thousands orphan PersistentTopic caused OOM #21540
• Fix setReplicatedSubscriptionStatus incorrect behavior #21510
• Fix failure while creating non-durable cursor with inactive managed-ledger #21508
• Fix PulsarService/BrokerService shutdown when brokerShutdownTimeoutMs=0 #21496
• Do not write replicated snapshot marker when the topic which is not enable replication #21495
• Avoid pass null role in MultiRolesTokenAuthorizationProvider #21486
• Fix the deadlock when using BookieRackAffinityMapping with rackaware policy #21481
• Fix issue with consumer read uncommitted messages from compacted topic #21465
• Delete methods marked Lombok @getter #21462
• Fix resource_quota_zpath #21461
• Fix namespace bundle stuck in unloading status #21445
• Ignore individual acknowledgment for CompactorSubscription when an entry has been filtered #21434
• Fix MultiRoles token provider NPE when using anonymous clients #21429
• Allow broker deployment in heterogeneous hw config cluster without restricting nic speed detection #21409
• Fix avoid creating new topic after migration is started #21368
• Fix race condition of replication cluster connection during migration topic #21364
• Optimize and clean up aggregation of topic stats #21361
• Fix heartbeat namespace create event topic and cannot delete heartbeat topic #21360
• PIP-188 Fix cluster migration state store into local metadatastore #21359
• PIP-188 Add support to auto create topic resources into green cluster before migration [part-3] #21354
• Fix heartbeat namespace create transaction internal topic #21348
• Fix unload operation stuck when use ExtensibleLoadManager #21332
• Fix typos in Subscription interface #21321
• Use ConcurrentHashMap in ServiceUnitStateChannel and avoid recursive update error #21282
• Cleanup deprecated method of metadata store etcd...

v2.11.3

Broker

• [fix][broker] Fix typo in the config key #21690
• [fix] [broker] network package lost if enable haProxyProtocolEnabled #21684
• [fix][broker] Fix memory leak during topic compaction #21647
• [fix][broker] Fix incorrect unack count when using shared subscription on non-persistent topic #21592
• [fix][broker] Duplicate LedgerOffloader creation when namespace/topic… #21591
• [improve][broker][PIP-318] Support not retaining null-key message during topic compaction #21578
• [fix][broker] Correct schema deletion for partitioned topic #21574
• [fix][broker] Fix namespace bundle stuck in unloading status (#21445) #21566
• [fix][broker] Fix create topic with different auto creation strategies causes race condition #21545
• [fix] [ml] Fix orphan scheduled task for ledger create timeout check #21542
• [fix] [broker] Fix thousands orphan PersistentTopic caused OOM #21540
• [fix][ml] Fix unfinished callback when deleting managed ledger #21530
• [fix][broker] Fix setReplicatedSubscriptionStatus incorrect behavior #21510
• [fix][txn] OpRequestSend reuse problem cause tbClient commitTxnOnTopic timeout unexpectedly #21505
• [fix][broker] Avoid pass null role in MultiRolesTokenAuthorizationProvider #21486
• [fix][broker] Fix the deadlock when using BookieRackAffinityMapping with rackaware policy #21481
• [fix][broker] Fix issue with consumer read uncommitted messages from compacted topic #21465
• [fix][broker] Fix resource_quota_zpath #21461
• [fix][broker] Fix namespace bundle stuck in unloading status #21445
• [fix][broker] Fix MultiRoles token provider NPE when using anonymous clients #21429
• [fix][ci] Fix docker image building by releasing more disk space before building #21365
• [fix][broker] Fix heartbeat namespace create event topic and cannot delete heartbeat topic #21360
• [fix][broker] Fix heartbeat namespace create transaction internal topic #21348
• [fix] [bk-client] Fix bk client MinNumRacksPerWriteQuorum and EnforceMinNumRacksPerWriteQuorum not work problem. #21327
• [fix] [auto-recovery] [branch-2.11] Fix pulsar ledger auditor dead lock problem. #21319
• [fix] [auto-recovery] [branch-2.11] Fix PulsarLedgerUnderreplicationManager notify problem. #21318
• [fix][auto-recovery] [branch-2.11] Fix metadata store deadlock due to BookkeeperInternalCallbacks.Processor #21315
• [fix][ml] Fix thread safe issue with RangeCache.put and RangeCache.clear #21302
• [fix][txn] Ack all message ids when ack chunk messages with transaction. #21268
• [fix] [ml] fix wrong msg backlog of non-durable cursor after trim ledgers #21250
• [fix] [ml] Reader can set read-pos to a deleted ledger #21248
• [fix][broker]Fixed produce and consume when anonymousUserRole enabled #21237
• [fix][broker] Fix inconsistent topic policy #21231
• [fix][broker] Fixed reset for AggregatedNamespaceStats #21225
• [improve] [broker] Let the producer request success at the first time if the previous one is inactive #21220
• [improve] [proxy] Not close the socket if lookup failed caused by too many requests #21216
• [improve] [broker] Not close the socket if lookup failed caused by bundle unloading or metadata ex #21211
• [fix][broker] replicator leak when removeReplicator in NonPersistentTopic #21205
• [improve] [broker] Print warn log if ssl handshake error & print ledger id when switch ledger #21201
• [fix] [broker] Make the new exclusive consumer instead the inactive one faster #21183
• [fix] [auto-recovery] Fix pulsar ledger auditor dead lock problem. #21181
• [fix][broker] Backport fix UniformLoadShedder selecet wrong overloadbroker and underloadbroker #21178
• [improve] [broker] improve read entry error log for troubleshooting #21169
• [branch-2.11] Fix compatibility issues #21163
• [fix] [auto-recovery] Fix PulsarLedgerUnderreplicationManager notify problem. #21161
• [fix][auto-recovery] Fix metadata store deadlock due to BookkeeperInternalCallbacks.Processor #21159
• [fix] [bookie] Fix RocksDB configuration #21157
• [fix] [broker] Make specified producer could override the previous one #21155
• [improve] [broker] Improve logs for troubleshooting #21141
• [fix][broker] Fix unack count when mixing non batch index and batch index acks #21126
• [fix] [broker] Fix isolated group not work problem. #21096
• [fix][broker] Fix write duplicate entries into the compacted ledger after RawReader reconnects #21081
• [fix] [broker] remove bundle-data in local metadata store. #21078
• [fix][proxy] Fix Proxy 502 gateway error when it is configured with Keystore TLS and admin API is called #21077
• [fix][broker] Make sure all inflight writes have finished before completion of compaction #21067
• [improve][broker] Make read compacted entries support maxReadSizeBytes limitation #21065
• [improve] [broker] Improve cache handling for partitioned topic metadata when doing lookup #21063
• [fix][broker] Use MessageDigest.isEqual when comparing digests #21061
• [fix] [auto-recovery] Improve to the ReplicaitonWorker performance by deleting invalid underreplication nodes #21059
• [fix][broker] Fix potential case cause retention policy not working on topic level #21041
• [fix][fn] Fix ProducerConfig cannot update error #21037
• [fix] [bk] Correctct the bookie info after ZK client is reconnected #21035
• [fix][broker] fix UniformLoadShedder selecet wrong overloadbroker and underloadbroker #21025
• [fix][fn] Fix the --batch-builder not working error for functions #21023
• [fix][meta] Fix deadlock in AutoRecovery. #21010
• [fix][broker] Fix incorrect unack msk count when dup ack a message #20990
• [fix][broker] Fix can't stop phase-two of compaction even though messageId read reaches lastReadId #20988
• [fix][broker] Fix compaction subscription delete by inactive subscription check. #20983
• [fix][broker] Fix message loss during topic compaction #20980
• [fix][broker] Fix incorrect number of read compacted entries #20978
• [fix] [admin] Fix get topic stats fail if a subscription catch up concurrently #20971
• [branch-2.11] [fix] [broker] fix multiple versions of bouncy-castle #20960
• [fix] [broker] Producer is blocked on creation because backlog exceeded on topic, when dedup is enabled and no producer is there #20951
• [fix][broker]Fix chunked messages will be filtered by duplicating #20948
• [fix][broker] rackaware policy is ineffective when delete zk rack info after bkclient initialize #20944
• [fix][broker]Check that the super user role is in the MultiRolesTokenAuthorizationProvider plugin #20939
• [fix][broker] fix MessageDeduplication throw NPE when enable broker dedup and set namespace disable deduplication. #20905
• [fix] [ml] fix discontinuous ledger deletion #20898
• [fix][broker] In replication scenario, remote consumer could not be registered if there has no message was sent #20888
• [improve][txn]Add log when can't send TcClientConnectRequest to broker. #20880
• [fix][broker] Fix inconsensus namespace policies by getPoliciesIfCached #20855
• [fix][broker] Inconsistent behaviour for topic auto_creation #20843
• [improve] [ws] add cryptoKeyReaderFactoryClassName into the file websocket.conf #20840
• [fix][build] Upgrade PyYaml version to 6.0.1 #20831
• [fix][broker] Avoid infinite bundle unloading #20822
• [improve][offload] Extend the offload policies to allow specifying more conf #20804
• [improve][offload] Create offload resources lazily #20775
• [fix] [broker] Can not receive any messages after switch to standby cluster #20767
• [fix][broker] Fix get topic policies as null during clean cache #20763
• [fix][broker] Avoid throwing RestException in BrokerService #20761
• [improve][offload] Support to configure more offload driver #20736
• [fix][broker] Gracefully shutdown does not work with admin cli in standalone #20709
• [fix][broker] Fix ModularLoadManagerImpl always delete active bundle-data. sec ver. #20620
• [improve] [broker] Add consumer-id into the log when doing subscribe. #20568
• [fix][broker] Allow Access to System Topic Metadata for Reader Creation Post-Namespace Deletion #20304
• [improve][broker] Replace ScheduledExecutorService to ExecutorService in ModularLoadManagerImpl #19656
• [fix][broker] Fix namespace deletion if __change_events topic has not been created yet #18804
• [improve] Introduce the sync() API to ensure consistency on reads during critical metadata operation paths #18518
• [fix][broker] Fix namespace not found will cause request timeout #18512
• [fix][broker] fix bug caused by optimistic locking #18390
• [fix][broker] fix broker unackmessages become a negative number #17003

Clients

• [fix][client] Fix print error log 'Auto getting partitions failed' when expend partition. #21485
• [fix] [client] fix reader.hasMessageAvailable return false when incoming queue is not empty #21259
• [improve] [client] Merge lookup requests for the same topic #21232
• [fix] [client] fix same producer/consumer use more than one connection per broker #21144
• [fix][client] Fix repeat consume when using n-ack and batched messages #21116
• [fix][client] Fix logging problem in pulsar client #21094
• [fix][client] Fix subscribing pattern topics through Proxy not working #20739
• [fix][client] Fix producer could send timeout when enable batching #19191
• [fix][client] Fix perf-producer get OOM with high publish latency #20914
• [fix][client] Fix RawReader hasMessageAvailable returns true when no messages #21032
• [fix][client] Fix cannot retry chunk messages and send to DLQ #21048
• [fix][client] Fix consumer can't consume resent chunked messages #21070

Pulsar IO and Pulsar Functions

• [fix] [io] elastic-search sink connector not support JSON.String schema. #20741
• [improve][sql] Fix the wrong format of the logs #20907
• [fix][io] Fix --retain[-key]-ordering not working error for sink #21060
• [fix][io] Update test certs for Elasticsearch #21001
• [fix][io][branch-2.11] Not restart instance when kafka source pol...

v3.1.2

PIPs

• PIP-318: Support not retaining null-key message during topic compaction by @coderzc in #21578

Broker

• Fixed ServiceUnitStateChannel monitor to tombstone only inactive bundle states by @heesung-sn in #21721
• Fix typo in the config key by @coderzc in #21690
• Record GeoPersistentReplicator.msgOut before producer#sendAsync by @nodece in #21673
• Print recoverBucketSnapshot log if cursorProperties are empty by @coderzc in #21651
• Support not retaining null-key message during topic compaction by @coderzc in #21601
• Delete topic timeout due to NPE by @poorbarcode in #21595
• Duplicate LedgerOffloader creation when namespace/topic policies are updated by @shibd in #21591
• Fix issue with consumer read uncommitted messages from compacted topic by @coderzc in #21571
• Fix create topic with different auto creation strategies causes race condition by @mattisonchao in #21545
• Fix orphan scheduled task for ledger create timeout check by @poorbarcode in #21542
• Fix thousands orphan PersistentTopic caused OOM by @poorbarcode in #21540
• Fix unfinished callback when deleting managed ledger by @mattisonchao in #21530
• Fix setReplicatedSubscriptionStatus incorrect behavior by @liudezhi2098 in #21510
• Fix failure while creating non-durable cursor with inactive managed-ledger by @rdhabalia in #21508
• Fix PulsarService/BrokerService shutdown when brokerShutdownTimeoutMs=0 by @lhotari in #21496
• Do not write replicated snapshot marker when the topic which is not enable replication by @liangyepianzhou in #21495
• Avoid pass null role in MultiRolesTokenAuthorizationProvider by @mattisonchao in #21486
• Fix the deadlock when using BookieRackAffinityMapping with rackaware policy by @erobot in #21481
• Fix issue with consumer read uncommitted messages from compacted topic by @coderzc in #21465
• Fix resource_quota_zpath by @AnonHxy in #21461
• Fix namespace bundle stuck in unloading status by @mattisonchao in #21445
• Ignore individual acknowledgment for CompactorSubscription when an entry has been filtered. by @coderzc in #21434
• Fix MultiRoles token provider NPE when using anonymous clients by @Technoboy- in #21429
• Fix heartbeat namespace create event topic and cannot delete heartbeat topic by @TakaHiR07 in #21360
• Fix heartbeat namespace create transaction internal topic by @TakaHiR07 in #21348
• Fix unload operation stuck when use ExtensibleLoadManager by @Demogorgon314 in #21332
• Fix lookup heartbeat and sla namespace bundle when using extensible load manager(#21213) by @Demogorgon314 in #21314
• Fix thread safe issue with RangeCache.put and RangeCache.clear by @lhotari in #21302
• use ConcurrentHashMap in ServiceUnitStateChannel and avoid recursive update error by @heesung-sn in #21282
• Fix inconsistent topic policy by @mattisonchao in #21255
• Let the producer request success at the first time if the previous one is inactive by @poorbarcode in #21220
• Fix lookup heartbeat and sla namespace bundle when using extensible load manager by @Demogorgon314 in #21213
• Avoid record inactiveproducers when deduplication is disable by @lifepuzzlefun in #21193
• Make specified producer could override the previous one by @poorbarcode in #21155
• Rackaware policy is ineffective when delete zk rack info after bkclient initialize by @TakaHiR07 in #20944
• Check that the super user role is in the MultiRolesTokenAuthorizationProvider plugin by @tuteng in #20939

Client

• Fix inconsistent API annotations of getTopicName by @liangyepianzhou in #21620
• Fix print error log 'Auto getting partitions failed' when expend partition by @hanmz in #21485
• Avert extensive time consumption during table view construction by @liangyepianzhou in #21270

Transaction

• OpRequestSend reuse problem cause tbClient commitTxnOnTopic timeout unexpectedly by @TakaHiR07 in #21505
• Ack all message ids when ack chunk messages with transaction by @liangyepianzhou in #21268

CI and Test

• Fix LocalBookkeeperEnsemble resource leak in tests by @lhotari in #21407
• Add new CI unit test group "Broker Group 4" with cluster migration tests by @lhotari in #21391
• Fix resource leaks with Pulsar Functions tests by @lhotari in #21378
• Fix some resource leaks in compaction tests by @lhotari in #21374
• Fix a resource leak in ClusterMigrationTest by @lhotari in #21366
• Fix docker image building by releasing more disk space before building by @lhotari in #21365
• Fix flaky CompactionTest.testDispatcherMaxReadSizeBytes by @lhotari in #21329
• Fix flaky test NarUnpackerTest by @lhotari in #21328
• Fix zookeeper related flacky test by @horizonzy in #21310
• fix flaky test PatternTopicsConsumerImplTest by @poorbarcode in #21222

Security

• Upgrade rabbitmq client to address CVE-2023-46120 by @liangyepianzhou in #21619
• Upgrade Zookeeper to 3.8.3 to address CVE-2023-44981 by @lhotari in #21398
• Upgrade Netty to 4.1.100 to address CVE-2023-44487 by @lhotari in #21397
• Upgrade Jetty to 9.4.53 to address CVE-2023-44487 by @lhotari in #21395
• Fix MultiRoles token provider when using anonymous clients by @merlimat in #21338
• Add OWASP Dependency Check suppressions by @lhotari in #21281
• Upgrade snappy-java to 1.1.10.5 by @lhotari in #21280

Build

• Fix Stage Docker images fail on M1 Mac by @AnonHxy in #21659
• Fix apt download issue in building the docker image by @lhotari in #21489
• Rename schema_example.conf to schema_example.json by @poorbarcode in #21447
• Upgrade Lombok to 1.18.30 to support compiling with JDK21 by @lhotari in #21278

Others

• [offload] Don't cleanup data when offload met MetaStore exception by @zymap in #21686
• [proxy] Move status endpoint out of auth coverage by @mattisonchao in #21428
• [auto-recovery] Migrate the replication testing from BookKeeper to Pulsar by @horizonzy in #21340
• [bk-client] Fix bk client MinNumRacksPerWriteQuorum and EnforceMinNumRacksPerWriteQuorum not work problem by @horizonzy in #21327
• [sql] Support UUID for json and avro by @liangyepianzhou in #21267
• [log] fix the vague response if topic not found by @poorbarcode in #20932