Release v0.8.0

Apache Uniffle (Incubating) Release v0.8.0

Highlight

• Support TEZ
• Introduce Netty for shuffle data transmission
• Use off heap memory to store shuffle data.
• Introduce REST API for cluster management.
• Introduce command line for cluster management.

ChangeLog

• Change license owner to ASF by @kaijchen in #5
• Trivial code improvements by @wForget in #7
• [Minor] Store shuffleId int to be consistent with other data structure by @zuston in #10
• Introduce the asList method in ConfigOptions by @zuston in #9
• Rename package by @jerqi in #6
• Minimize apache-rat excluded files by @kaijchen in #11
• Update module names by @kaijchen in #12
• Covert PartitionAssignmentInfo to static inner class by @pan3793 in #15
• [Followup] Migrate to Junit5 by @zuston in #14
• [Bug] Fix NPE problem when process the event if application was cleared already by @colinmjj in #16
• [CI] Enable codecov report by @kaijchen in #17
• Correct the config description and fix typo by @zuston in #19
• Add CI and Codecov badges in README by @kaijchen in #20
• [Followup] Use asList method in some existing configOptions by @zuston in #18
• Move rss-integration-spark-common-test module package by @wForget in #23
• [INFRA] Improve asf.yaml to reduce the notifications by @jerryshao in #25
• [TEST] Improve code coverage in rss-common by @kaijchen in #26
• Remove redundant package by @wForget in #27
• [CI] Switch to temurin JDK by @kaijchen in #24
• [INFRA] Improve asf.yaml to reduce the notifications (another-try) by @jerryshao in #33
• Bump commons-lang3 from 3.5 to 3.10 by @wForget in #28
• Fix the log of incorrectly bound class by @wForget in #35
• [TYPO] Fix misspelled word "integration" by @kaijchen in #34
• Fix some hyperlink in README.md by @daugraph in #32
• Upgrade gRPC to support Apple Silicon by @pan3793 in #13
• Allow to specify custom tags to decide the assignment of servers by @zuston in #30
• Optimize the bash script by @zuston in #29
• [Improvement] reduce compiler warnings by @advancedxy in #46
• [Chore]: document update and build time optimize by @advancedxy in #45
• Supplement doc about assignment tags by @zuston in #47
• [Bug] Fix skip() api maybe skip unexpected bytes which makes inconsistent data by @colinmjj in #40
• [improvement] Remove experimental feature with ShuffleUploader by @colinmjj in #51
• [Improvement] Provides utility classes for creating thread factories by @smallzhongfeng in #49
• Enable spotbugs and fix high priority bugs by @kaijchen in #38
• [CI] Change default checkstyle severity to error by @kaijchen in #57
• [Style] Check indentation by @kaijchen in #56
• [Experimental Feature] MR Supports Remote Spill by @frankliee in #55
• [Improvement] Log indicate the shuffle server host:port when doing re… by @zuston in #58
• Send commit concurrently in client side by @zuston in #59
• Explicitly set the constructor with AccessManager when extending AccessChecker by @zuston in #43
• [DOC] Replace Firestorm with Uniffle by @jerqi in #60
• Introduce the extraProperties to support user-defined pluggable accessCheckers by @zuston in #42
• Log enhancement: Merge multiple logs into oneline and add more description by @zuston in #62
• [TEST] Add more unit tests in rss-common by @kaijchen in #63
• [MINOR] Comments of PartitionBalanceAssignmentStrategy miss byte units by @smallzhongfeng in #68
• [Minor] Make config keys and default values finalized by @kaijchen in #70
• [Log Improvment] Add more detailed debug info for MR client by @frankliee in #84
• [Improvement] Shutdown the grpc executors pool when closing by @zuston in #83
• Log enhancement: return error message when getting assignment servers and log exception when initializing by @zuston in #64
• [ISSUE-48] [Feature] Init Kubernetes operator directory by @jerqi in #75
• [Improvement] No need to use synchronized lock of the method scope when getting client by @zuston in #82
• [DOC] Remove Wechat group in README by @jerqi in #88
• [Performance Optimization] Improve the speed of writing index file in shuffle server by @zuston in #91
• [DOC] Update title and description in README by @kaijchen in #94
• [Improvement] ShuffleBlock should be release when finished reading by @xianjingfeng in #74
• [IMPROVEMENT][COMMON] Fix common module code style by @jerqi in #99
• [Improvement]LocalStorage init use multi thread #71 by @xianjingfeng in #72
• [Improvement] Use OR operation instead of serialization for cloning BitMaps by @kaijchen in #103
• [Improvement] Ignore partial failure on initializing local storage in shuffle server side by @zuston in #102
• [CI] Test compile in Java 11 and Java 17 by @kaijchen in #105
• Sleep less time but try more times when stopping by @xianjingfeng in #112
• [Improvement] Use ConfigBuilder to rewrite the class RssSparkConfig by @smallzhongfeng in #104
• [Improvement] Introduce config to customize assignment server numbers in client side by @zuston in #100
• Assign partition again if registerShuffleServers failed by @xianjingfeng in #115
• [ISSUE-106][IMPROVEMENT] Set rpc timeout for all rpc interface by @xianjingfeng in #113
• [MINOR][IMPROVEMENT] Avoid CoordinatorServer#initialization multiple new Configuration() by @zwangsheng in #118
• [Improve] Remove useless server id from StorageManagerFactory#createStorageManager by @zwangsheng in #119
• [MINOR][IMPROVEMENT][COORD] Fix coordinator module code style by @jerqi in #122
• [Improvement] Set heartBeatExecutorService as daemon thread by @smallzhongfeng in #121
• [JUnit] Introduce the property of trimStackTrace to show error stacktrace in mvn-test by @zuston in #126
• Make the conf of rss.storage.basePath as list by @zuston in #130
• [MINOR][IMPROVEMENT][STORAGE] Fix storage module code style by @jerqi in #131
• [Improvement] Add timeout reconnection when DelegationRssShuffleManager send the request of AccessCluster by @smallzhongfeng in #139
• [MINOR] Fix flaky test testGetHostIp by @izchen in #141
• [Improvement] Add the number of unhealthy nodes in CoordinatorMetrics by @smallzhongfeng in #147
• [ISSUE-48][FEATURE] Add Uniffle Dockerfile by @wangao1236 in #132
• [BUGFIX] Fix memory leak which cause oom by @summaryzb in #145
• [Log Improvement] Output the register...

Release v0.7.1

Apache Uniffle (Incubating) Release v0.7.1

Highlight

• Improvements

  • Refresh application when reading memory data to prevent application from being expired.

• Major bug fixes

  • Cache proxy user ugi to avoid memory leak.
  • Make metric reporter usable by fixing some logic errors.
  • Close ShuffleWriteClient after task was completed to exit container process.
  • Make sure 'finishShuffle' invoked after all shuffle data sent.
  • Update local storage metadata for all related events instead of just the first event.
  • Correct a wrong metric, grpc_server_connection_number.

• Minor bug fixes

  • Avoid returning null in defaultUserApps when quote file doesn't config user.
  • Fix LocalStorageManager divide by zero exception.

ChangeLog

• [ISSUE-669] improvement: refresh application when reading memory data (#741)
• Bump project version to 0.7.1-SNAPSHOT
• [ISSUE-772] fix(kerberos): cache proxy user ugi to avoid memory leak (#773)
• Revert "[ISSUE-772] fix(kerberos): cache proxy user ugi to avoid memory leak (#773)
• [ISSUE-796][0.7] bug: Fix the issues of MetricReporter (#821)
• [MINOR][0.7] Avoid returning null in defaultUserApps when quota file doesn't config user (#822)
• [ISSUE-772] fix(kerberos): cache proxy user ugi to avoid memory leak (#773)
• Revert "[ISSUE-772] fix(kerberos): cache proxy user ugi to avoid memory leak (#773)
• [ISSUE-772][0.7] fix(kerberos): cache proxy user ugi to avoid memory leak (#773)
• [ISSUE-715] fix(mr): The container does not exit because shuffleclient is not closed (#882)
• [ISSUE-886] fix(mr): MR Client may lose data or throw exception when rss.storage.type without MEMORY (#887)
• Revert "[ISSUE-886] fix(mr): MR Client may lose data or throw exception when rss.storage.type without MEMORY (#887)
• [MINOR] fix: Fix LocalStorageManager divide by zero exception (#900)
• [ISSUE-881][0.7] fix(followup): Ensure LocalStorageMeta disk size is correctly updated when events are processed for 0.7.0 (#914)
• [ISSUE-933][0.7] fix: incorrect metric grpc_server_connection_number (#941)
• change version to 0.7.1 release

Release v0.7.0

Apache Uniffle (Incubating) Release v0.7.0

Highlight

• Better support for Spark AQE
  • Leveraging the LOCAL_ORDER data distribution to improve performance
  • Estimating assignment number of shuffle servers to improve user experience
  • Better assignment policy of assigning adjacent partitions to the same shuffle server to improve performance
• Optimization of huge partition to improve stability of shuffle servers
• More bug fixes and usability improvements of K8S operator
• Add support of user quota management and more compression algorithms
• Add support of spark data eviction mechanism of stage level
• Some improvement of stability and performance

Changelog

• Add more badges in README by @kaijchen in #219
• Fix incorrect log format strings by @kaijchen in #220
• Change total lines badge url to sloc.xyz in README by @kaijchen in #222
• [MINOR] Fix warnings reported by lgtm by @kaijchen in #223
• [MINOR] Simplify creating buffer logic by @zuston in #227
• Support cancelling previous ci actions by @zuston in #225
• Use the conf of shuffleNodesNumber from jobs to be as checking factor by @zuston in #208
• Output the stderr and stdout to output file in startup script by @zuston in #226
• [ISSUE-48][FEATURE][FOLLOW UP] Add controller component by @wangao1236 in #214
• Add more metrics about requiring read memory by @zuston in #231
• Adjust the memory required times to match grpc max deadline conf by @zuston in #218
• [MINOR] Fix flaky test by @jerqi in #238
• [ISSUE-48][FEATURE][FOLLOW UP] Add yaml of components and crd exampes by @wangao1236 in #236
• Fix Flaky test GetShuffleReportForMultiPartTest by @leixm in #241
• Set the default disk capacity to the total space by @zuston in #237
• Add issue template by @jerqi in #8
• [MINOR] Fix inefficient map iteration by @kaijchen in #245
• Support deploy multiple shuffle servers in a single node by @xianjingfeng in #166
• Fast fail when reading failed in ComposedClientReadHandler by @zuston in #213
• Fix startup shell problem by @jerqi in #251
• New version 0.7.0-snapshot by @jerqi in #252
• [ISSUE-196] Fix flaky test about kerberos by @zuston in #250
• [ISSUE-48][FEATURE][FOLLOW UP] add unit test for validating rss objects by @wangao1236 in #248
• [Improvement] Add hdfs path health check to AppBalanceSelectStorageStrategy by @smallzhongfeng in #210
• [TYPO] Replace Chinese colon by ASCII colon by @kaijchen in #255
• Introduce startup-silent-period mechanism to avoid partial assignments by @zuston in #247
• Replace DISCLAIMER with DISCLAIMER-WIP by @jerqi in #258
• [ISSUE-244] Fix flaky test of CoordinatorGrpcTest.rpcMetricsTest by @zuston in #256
• Fix flaky test of ClientConfManagerTest by @smallzhongfeng in #260
• [Refactor] Optimize creating shuffle handlers by @zuston in #259
• Introduce data cleanup mechanism on stage level by @zuston in #249
• [ISSUE-48][FEATURE][FOLLOW UP] add docs for operator by @wangao1236 in #261
• Fix potenial missing reads of exclude nodes by @zuston in #269
• [ISSUE-257] RssMRUtils#getBlockId change the partitionId of int type to long by @fpkgithub in #266
• [ISSUE-273][BUG] Get shuffle result failed caused by concurrent calls to registerShuffle by @leixm in #274
• Add enum type test about case insensitive by @zuston in #280
• Support ZSTD by @zuston in #254
• [ISSUE-239][BUG] RssUtils#transIndexDataToSegments should consider the length of the data file by @leixm in #275
• Remove code quality badge and add release badge by @kaijchen in #284
• [ISSUE-163][FEATURE] Write to hdfs when local disk can't be write by @xianjingfeng in #235
• Upgrade Github actions for Node.js 16 by @kaijchen in #292
• Fix NPE in WriteBufferManager.addRecord by @wForget in #296
• Fix AbstractStorage#containsWriteHandler by @xianjingfeng in #281
• Add more test cases on LocalStorageManager.selectStorage by @zuston in #298
• [ISSUE-137][Improvement][AQE] Sort MapId before the data are flushed by @zuston in #293
• [ISSUE-283][FEATURE] Support snappy compression/decompression by @amaliujia in #304
• [ISSUE-290] Make RpcNodePort and HttpNodePort optional by @amaliujia in #305
• [ISSUE-301][Subtask][Improvement][AQE] Merge continuous ShuffleDataSegment into single one by @zuston in #303
• Cleanup RuntimeException and fetchRemoteStorage logic in ClientUtils by @kaijchen in #295
• [ISSUE-135][FOLLOWUP][Improvement][AQE] Assign adjacent partitions to the same ShuffleServer by @leixm in #307
• Correct the contributing guide link in pull-request template by @zuston in #314
• Fix bug of "Comparison method violates its general contract" by @zuston in #315
• [AQE][LocalOrder] Fix potenial bug when merging continuous segments by @zuston in #318
• [AQE][LocalOrder] Fix wrong param of expectedTaskIds in LocalOrderSegmentSplit by @zuston in #319
• [Feature] Support the estimated number of ShuffleServers required. by @leixm in #322
• [Bug] Fix potenial bug when the index reading offset is greater than data length by @zuston in #320
• [ISSUE-154][Improvement] Support Empty assignment to Shuffle Server by @rhh777 in #325
• [Bug] Fix invalid owner of host path volumes by @wangao1236 in #330
• [ISSUE-309][FEATURE] Support ShuffleServer latency metrics. by @leixm in #327
• [ISSUE-329]Catch NPE in ShuffleTaskManager#addFinishedBlockIds by @xianjingfeng in #331
• [BUG] Fix wrong method name by @leixm in #335
• [ISSUE-328] Cleanup unused shuffle servers after stage completed by @xianjingfeng in #334
• [MINOR] Migrate RankValue to the package of the common class by @smallzhongfeng in #265
• [BUG] Fix incorrect spark metrics by @zuston in #324
• [Improvement][LocalOrder] Add tests about keeping consistent with FixedSize when no skew optimization by @zuston in #336
• [INFRA] Add k8s pipeline by @jerqi in #340
• Remove unused class of RssShuffleUtils by @zuston in #345
• [ISSUE-342][Improvement] Check Spark Serializer type by @chong0929 in #344
• [Feature] Support user's app quota level limit by @smallzhongfeng in #311
• [BUG][AQE][LocalOrder] Fix the bug of missed data due to block sorting by @zuston in #347
• [ISSUE-364] Fix indexWriter don't close if exception thrown when close dataWriter by @xianjingfeng in #349
• [BUG] Fix flaky test of AQESkewedJoinWithLocalOrderTest by @zuston in #350
• Add collaborators by @jerqi in #351
• [BUG][FOLLOWUP] Fix flaky test of AQESkewedJoinWithLocalOrderTest by @zuston in https://github.com/...

Release v0.6.1

Apache Uniffle (Incubating) Release v0.6.1

Highlight

• Major bug fixes

  • Partition cannot be accessed in MapReduce when the reduce task number exceeds 1024.
  • Get shuffle result failure caused by concurrent calls to registerShuffle.
  • Inconsistent blocks caused by missing length in RssUtils#transIndexDataToSegments.
  • Handle NPE in WriteBufferManager#addRecord in the same way as Spark.
  • AbstractStorage#containsWriteHandler is checking the wrong Map.
  • indexWriter isn't closed if exception is thrown when closing dataWriter.
  • Incorrect dependency of protobuf-java at compile time.
  • Potential memory leak when encountering disk unhealthy.

• Minor bug fixes

  • Potenial missing reads of exclude nodes.
  • Incorrect contributing link in pull-request template.
  • Incorrect spark metrics.

ChangeLog

• [ISSUE-257] RssMRUtils#getBlockId change the partitionId of int type to long (#266)
• [ISSUE-273][BUG] Get shuffle result failed caused by concurrent calls to registerShuffle (#274)
• Fix potenial missing reads of exclude nodes (#269)
• [ISSUE-239][BUG] RssUtils#transIndexDataToSegments should consider the length of the data file (#275)
• Fix NPE in WriteBufferManager.addRecord (#296)
• Fix AbstractStorage#containsWriteHandler (#281)
• Correct the pull-request contributing link in template (#314)
• [BUG] Fix incorrect spark metrics (#324)
• [ISSUE-364] Fix indexWriter don't close if exception thrown when close dataWriter (#349)
• [ISSUE-228] Fix the problem of protobuf-java incorrect dependency at compile time (#362)
• Bump project version to 0.6.1
• [BUG] Potenial memory leak when encountering disk unhealthy (#370)

Release v0.6.0

Apache Uniffle (Incubating) Release v0.6.0

Highlight

• Optimize the assignment strategy

• Some improvement of stability and performance

• Add a plugin mechanism of SelectStorageStrategy

• Add LowestIOSampleCostSelectStorageStrategy

• Support Kerberos HDFS

ChangeLog

• Change license owner to ASF by @kaijchen in #5
• Trivial code improvements by @wForget in #7
• [Minor] Store shuffleId int to be consistent with other data structure by @zuston in #10
• Introduce the asList method in ConfigOptions by @zuston in #9
• Rename package by @jerqi in #6
• Minimize apache-rat excluded files by @kaijchen in #11
• Update module names by @kaijchen in #12
• Covert PartitionAssignmentInfo to static inner class by @pan3793 in #15
• [Followup] Migrate to Junit5 by @zuston in #14
• [Bug] Fix NPE problem when process the event if application was cleared already by @colinmjj in #16
• [CI] Enable codecov report by @kaijchen in #17
• Correct the config description and fix typo by @zuston in #19
• Add CI and Codecov badges in README by @kaijchen in #20
• [Followup] Use asList method in some existing configOptions by @zuston in #18
• Move rss-integration-spark-common-test module package by @wForget in #23
• [INFRA] Improve asf.yaml to reduce the notifications by @jerryshao in #25
• [TEST] Improve code coverage in rss-common by @kaijchen in #26
• Remove redundant package by @wForget in #27
• [CI] Switch to temurin JDK by @kaijchen in #24
• [INFRA] Improve asf.yaml to reduce the notifications (another-try) by @jerryshao in #33
• Bump commons-lang3 from 3.5 to 3.10 by @wForget in #28
• Fix the log of incorrectly bound class by @wForget in #35
• [TYPO] Fix misspelled word "integration" by @kaijchen in #34
• Fix some hyperlink in README.md by @daugraph in #32
• Upgrade gRPC to support Apple Silicon by @pan3793 in #13
• Allow to specify custom tags to decide the assignment of servers by @zuston in #30
• Optimize the bash script by @zuston in #29
• [Improvement] reduce compiler warnings by @advancedxy in #46
• [Chore]: document update and build time optimize by @advancedxy in #45
• Supplement doc about assignment tags by @zuston in #47
• [Bug] Fix skip() api maybe skip unexpected bytes which makes inconsistent data by @colinmjj in #40
• [improvement] Remove experimental feature with ShuffleUploader by @colinmjj in #51
• [Improvement] Provides utility classes for creating thread factories by @smallzhongfeng in #49
• Enable spotbugs and fix high priority bugs by @kaijchen in #38
• [CI] Change default checkstyle severity to error by @kaijchen in #57
• [Style] Check indentation by @kaijchen in #56
• [Experimental Feature] MR Supports Remote Spill by @frankliee in #55
• [Improvement] Log indicate the shuffle server host:port when doing re… by @zuston in #58
• Send commit concurrently in client side by @zuston in #59
• Explicitly set the constructor with AccessManager when extending AccessChecker by @zuston in #43
• [DOC] Replace Firestorm with Uniffle by @jerqi in #60
• Introduce the extraProperties to support user-defined pluggable accessCheckers by @zuston in #42
• Log enhancement: Merge multiple logs into oneline and add more description by @zuston in #62
• [TEST] Add more unit tests in rss-common by @kaijchen in #63
• [MINOR] Comments of PartitionBalanceAssignmentStrategy miss byte units by @smallzhongfeng in #68
• [Minor] Make config keys and default values finalized by @kaijchen in #70
• [Log Improvment] Add more detailed debug info for MR client by @frankliee in #84
• [Improvement] Shutdown the grpc executors pool when closing by @zuston in #83
• Log enhancement: return error message when getting assignment servers and log exception when initializing by @zuston in #64
• [ISSUE-48] [Feature] Init Kubernetes operator directory by @jerqi in #75
• [Improvement] No need to use synchronized lock of the method scope when getting client by @zuston in #82
• [DOC] Remove Wechat group in README by @jerqi in #88
• [Performance Optimization] Improve the speed of writing index file in shuffle server by @zuston in #91
• [DOC] Update title and description in README by @kaijchen in #94
• [Improvement] ShuffleBlock should be release when finished reading by @xianjingfeng in #74
• [IMPROVEMENT][COMMON] Fix common module code style by @jerqi in #99
• [Improvement]LocalStorage init use multi thread #71 by @xianjingfeng in #72
• [Improvement] Use OR operation instead of serialization for cloning BitMaps by @kaijchen in #103
• [Improvement] Ignore partial failure on initializing local storage in shuffle server side by @zuston in #102
• [CI] Test compile in Java 11 and Java 17 by @kaijchen in #105
• Sleep less time but try more times when stopping by @xianjingfeng in #112
• [Improvement] Use ConfigBuilder to rewrite the class RssSparkConfig by @smallzhongfeng in #104
• [Improvement] Introduce config to customize assignment server numbers in client side by @zuston in #100
• Assign partition again if registerShuffleServers failed by @xianjingfeng in #115
• [ISSUE-106][IMPROVEMENT] Set rpc timeout for all rpc interface by @xianjingfeng in #113
• [MINOR][IMPROVEMENT] Avoid CoordinatorServer#initialization multiple new Configuration() by @zwangsheng in #118
• [Improve] Remove useless server id from StorageManagerFactory#createStorageManager by @zwangsheng in #119
• [MINOR][IMPROVEMENT][COORD] Fix coordinator module code style by @jerqi in #122
• [Improvement] Set heartBeatExecutorService as daemon thread by @smallzhongfeng in #121
• [JUnit] Introduce the property of trimStackTrace to show error stacktrace in mvn-test by @zuston in #126
• Make the conf of rss.storage.basePath as list by @zuston in #130
• [MINOR][IMPROVEMENT][STORAGE] Fix storage module code style by @jerqi in #131
• [Improvement] Add timeout reconnection when DelegationRssShuffleManager send the request of AccessCluster by @smallzhongfeng in #139
• [MINOR] Fix flaky test testGetHostIp by @izchen in #141
• [Improvement] Add the number of unhealthy nodes in CoordinatorMetrics by @smallzhongfeng in #147
• [ISSUE-48][FEATURE] Add Uniffle Dockerfile by @wangao1236 in #132
• [BUGFIX] Fix memory leak which cause oom by @summaryzb in #145
• [Log Improvement] Output the re...