Tags merge optimization

[mstyura]

I've observed unexpectedly significant CPU time spent in Tags merge function in one of the production services which rely on micrometer:

I've slightly refactored internals of Tags class to take advantage of tags set is being already sorted array, which could be then combined with other sorted set in linear time.

I've also added microbenchmark for tags concat operation to see the performance gain before I can test this change in production.

Below I'm providing results from Apple M2 Pro + JDK 21 machine:

Before:

Benchmark             Mode  Cnt    Score   Error  Units
TagsBenchmark.dotAnd  avgt    2  204.243          ns/op
TagsBenchmark.of      avgt    2  104.397          ns/op

Benchmark                     Mode  Cnt     Score   Error  Units
TagsMergeBenchmark.mergeTags  avgt    2  2244.210          ns/op

Full JMH output (before)

# JMH version: 1.37
# VM version: JDK 21.0.1, OpenJDK 64-Bit Server VM, 21.0.1+12-LTS
# VM invoker: /Library/Java/JavaVirtualMachines/amazon-corretto-21.jdk/Contents/Home/bin/java
# VM options: -Dfile.encoding=UTF-8 -Duser.country=PL -Duser.language=en -Duser.variant
# Blackhole mode: compiler (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)
# Warmup: 2 iterations, 10 s each
# Measurement: 2 iterations, 10 s each
# Timeout: 10 min per iteration
# Threads: 16 threads, will synchronize iterations
# Benchmark mode: Average time, time/op
# Benchmark: io.micrometer.benchmark.core.TagsMergeBenchmark.mergeTags

# Run progress: 0.00% complete, ETA 00:00:40
# Fork: 1 of 1
# Warmup Iteration   1: 646.455 ±(99.9%) 5.904 ns/op
# Warmup Iteration   2: 2216.230 ±(99.9%) 20.156 ns/op
Iteration   1: 2243.049 ±(99.9%) 33.435 ns/op
Iteration   2: 2245.372 ±(99.9%) 27.381 ns/op


Result "io.micrometer.benchmark.core.TagsMergeBenchmark.mergeTags":
  2244.210 ns/op


# Run complete. Total time: 00:00:40

REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
experiments, perform baseline and negative tests that provide experimental control, make sure
the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
Do not assume the numbers tell you what you want them to tell.

NOTE: Current JVM experimentally supports Compiler Blackholes, and they are in use. Please exercise
extra caution when trusting the results, look into the generated code to check the benchmark still
works, and factor in a small probability of new VM bugs. Additionally, while comparisons between
different JVMs are already problematic, the performance difference caused by different Blackhole
modes can be very significant. Please make sure you use the consistent Blackhole mode for comparisons.

Benchmark                     Mode  Cnt     Score   Error  Units
TagsMergeBenchmark.mergeTags  avgt    2  2244.210          ns/op


# JMH version: 1.37
# VM version: JDK 21.0.1, OpenJDK 64-Bit Server VM, 21.0.1+12-LTS
# VM invoker: /Library/Java/JavaVirtualMachines/amazon-corretto-21.jdk/Contents/Home/bin/java
# VM options: -Dfile.encoding=UTF-8 -Duser.country=PL -Duser.language=en -Duser.variant
# Blackhole mode: compiler (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)
# Warmup: 2 iterations, 10 s each
# Measurement: 2 iterations, 10 s each
# Timeout: 10 min per iteration
# Threads: 16 threads, will synchronize iterations
# Benchmark mode: Average time, time/op
# Benchmark: io.micrometer.benchmark.core.TagsBenchmark.dotAnd

# Run progress: 0.00% complete, ETA 00:01:20
# Fork: 1 of 1
# Warmup Iteration   1: 184.006 ±(99.9%) 2.288 ns/op
# Warmup Iteration   2: 202.977 ±(99.9%) 1.666 ns/op
Iteration   1: 202.850 ±(99.9%) 2.525 ns/op
Iteration   2: 205.637 ±(99.9%) 2.360 ns/op


Result "io.micrometer.benchmark.core.TagsBenchmark.dotAnd":
  204.243 ns/op


# JMH version: 1.37
# VM version: JDK 21.0.1, OpenJDK 64-Bit Server VM, 21.0.1+12-LTS
# VM invoker: /Library/Java/JavaVirtualMachines/amazon-corretto-21.jdk/Contents/Home/bin/java
# VM options: -Dfile.encoding=UTF-8 -Duser.country=PL -Duser.language=en -Duser.variant
# Blackhole mode: compiler (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)
# Warmup: 2 iterations, 10 s each
# Measurement: 2 iterations, 10 s each
# Timeout: 10 min per iteration
# Threads: 16 threads, will synchronize iterations
# Benchmark mode: Average time, time/op
# Benchmark: io.micrometer.benchmark.core.TagsBenchmark.of

# Run progress: 50.00% complete, ETA 00:00:40
# Fork: 1 of 1
# Warmup Iteration   1: 116.358 ±(99.9%) 1.819 ns/op
# Warmup Iteration   2: 107.735 ±(99.9%) 1.373 ns/op
Iteration   1: 104.108 ±(99.9%) 1.247 ns/op
Iteration   2: 104.686 ±(99.9%) 1.438 ns/op


Result "io.micrometer.benchmark.core.TagsBenchmark.of":
  104.397 ns/op


# Run complete. Total time: 00:01:20

REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
experiments, perform baseline and negative tests that provide experimental control, make sure
the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
Do not assume the numbers tell you what you want them to tell.

NOTE: Current JVM experimentally supports Compiler Blackholes, and they are in use. Please exercise
extra caution when trusting the results, look into the generated code to check the benchmark still
works, and factor in a small probability of new VM bugs. Additionally, while comparisons between
different JVMs are already problematic, the performance difference caused by different Blackhole
modes can be very significant. Please make sure you use the consistent Blackhole mode for comparisons.

Benchmark             Mode  Cnt    Score   Error  Units
TagsBenchmark.dotAnd  avgt    2  204.243          ns/op
TagsBenchmark.of      avgt    2  104.397          ns/op

After:

Benchmark             Mode  Cnt    Score   Error  Units
TagsBenchmark.dotAnd  avgt    2  118.698          ns/op
TagsBenchmark.of      avgt    2  100.275          ns/op

Benchmark                     Mode  Cnt    Score   Error  Units
TagsMergeBenchmark.mergeTags  avgt    2  180.412          ns/op

Full JMH output (after)

# JMH version: 1.37
# VM version: JDK 21.0.1, OpenJDK 64-Bit Server VM, 21.0.1+12-LTS
# VM invoker: /Library/Java/JavaVirtualMachines/amazon-corretto-21.jdk/Contents/Home/bin/java
# VM options: -Dfile.encoding=UTF-8 -Duser.country=PL -Duser.language=en -Duser.variant
# Blackhole mode: compiler (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)
# Warmup: 2 iterations, 10 s each
# Measurement: 2 iterations, 10 s each
# Timeout: 10 min per iteration
# Threads: 16 threads, will synchronize iterations
# Benchmark mode: Average time, time/op
# Benchmark: io.micrometer.benchmark.core.TagsMergeBenchmark.mergeTags

# Run progress: 0.00% complete, ETA 00:00:40
# Fork: 1 of 1
# Warmup Iteration   1: 183.716 ±(99.9%) 2.371 ns/op
# Warmup Iteration   2: 188.851 ±(99.9%) 1.677 ns/op
Iteration   1: 180.315 ±(99.9%) 1.748 ns/op
Iteration   2: 180.508 ±(99.9%) 1.816 ns/op


Result "io.micrometer.benchmark.core.TagsMergeBenchmark.mergeTags":
  180.412 ns/op


# Run complete. Total time: 00:00:40

REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
experiments, perform baseline and negative tests that provide experimental control, make sure
the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
Do not assume the numbers tell you what you want them to tell.

NOTE: Current JVM experimentally supports Compiler Blackholes, and they are in use. Please exercise
extra caution when trusting the results, look into the generated code to check the benchmark still
works, and factor in a small probability of new VM bugs. Additionally, while comparisons between
different JVMs are already problematic, the performance difference caused by different Blackhole
modes can be very significant. Please make sure you use the consistent Blackhole mode for comparisons.

Benchmark                     Mode  Cnt    Score   Error  Units
TagsMergeBenchmark.mergeTags  avgt    2  180.412          ns/op


# JMH version: 1.37
# VM version: JDK 21.0.1, OpenJDK 64-Bit Server VM, 21.0.1+12-LTS
# VM invoker: /Library/Java/JavaVirtualMachines/amazon-corretto-21.jdk/Contents/Home/bin/java
# VM options: -Dfile.encoding=UTF-8 -Duser.country=PL -Duser.language=en -Duser.variant
# Blackhole mode: compiler (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)
# Warmup: 2 iterations, 10 s each
# Measurement: 2 iterations, 10 s each
# Timeout: 10 min per iteration
# Threads: 16 threads, will synchronize iterations
# Benchmark mode: Average time, time/op
# Benchmark: io.micrometer.benchmark.core.TagsBenchmark.dotAnd

# Run progress: 0.00% complete, ETA 00:01:20
# Fork: 1 of 1
# Warmup Iteration   1: 124.894 ±(99.9%) 2.132 ns/op
# Warmup Iteration   2: 116.994 ±(99.9%) 1.534 ns/op
Iteration   1: 117.101 ±(99.9%) 2.191 ns/op
Iteration   2: 120.295 ±(99.9%) 1.594 ns/op


Result "io.micrometer.benchmark.core.TagsBenchmark.dotAnd":
  118.698 ns/op


# JMH version: 1.37
# VM version: JDK 21.0.1, OpenJDK 64-Bit Server VM, 21.0.1+12-LTS
# VM invoker: /Library/Java/JavaVirtualMachines/amazon-corretto-21.jdk/Contents/Home/bin/java
# VM options: -Dfile.encoding=UTF-8 -Duser.country=PL -Duser.language=en -Duser.variant
# Blackhole mode: compiler (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)
# Warmup: 2 iterations, 10 s each
# Measurement: 2 iterations, 10 s each
# Timeout: 10 min per iteration
# Threads: 16 threads, will synchronize iterations
# Benchmark mode: Average time, time/op
# Benchmark: io.micrometer.benchmark.core.TagsBenchmark.of

# Run progress: 50.00% complete, ETA 00:00:40
# Fork: 1 of 1
# Warmup Iteration   1: 106.194 ±(99.9%) 0.890 ns/op
# Warmup Iteration   2: 101.206 ±(99.9%) 1.222 ns/op
Iteration   1: 101.113 ±(99.9%) 1.249 ns/op
Iteration   2: 99.436 ±(99.9%) 1.208 ns/op


Result "io.micrometer.benchmark.core.TagsBenchmark.of":
  100.275 ns/op


# Run complete. Total time: 00:01:20

REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
experiments, perform baseline and negative tests that provide experimental control, make sure
the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
Do not assume the numbers tell you what you want them to tell.

NOTE: Current JVM experimentally supports Compiler Blackholes, and they are in use. Please exercise
extra caution when trusting the results, look into the generated code to check the benchmark still
works, and factor in a small probability of new VM bugs. Additionally, while comparisons between
different JVMs are already problematic, the performance difference caused by different Blackhole
modes can be very significant. Please make sure you use the consistent Blackhole mode for comparisons.

Benchmark             Mode  Cnt    Score   Error  Units
TagsBenchmark.dotAnd  avgt    2  118.698          ns/op
TagsBenchmark.of      avgt    2  100.275          ns/op

[mstyura]

Hello @jonatan-ivanov! Could you please review this MR? Thanks a lot in advance!

[mstyura]

Hello @shakuzen, could you please review this merge request when you have a chance? Thank you in advance!

[shakuzen]

Thanks for the pull request. It's on our list of things to review.

[mstyura]

Thanks! Is there anything I can do to facilitate review, like maybe separating MR into 2 smaller independent MRs: one with benchmark and follow up with refactoring? So it may be easier to review?