Renaissance 0.15.0

This release is primarily aimed at improving compatibility with Java 21 (LTS) and Java 22 (EA), but it also marks the departure from Java 8 by requiring at least Java 11 to build the suite and to run the benchmarks. Several benchmarks now use Scala 3 and a memory leak was plugged in one of the benchmarks.

To improve compatibility with Java 21, the Apache Spark framework used by the apache-spark benchmarks was updated to version 3.5.0 and the Neo4j framework used by the neo4j-analytics benchmark was updated to version 5.12.0 (the benchmark now requires Java 17 to run). The database libraries used in the db-shootout benchmark were updated to more recent versions, which makes the benchmark compatible with Java 18 (while still far from Java 21, it is an improvement over Java 11, where the benchmark was stuck for some time).

The Scala 2.12 and 2.13 versions used by various benchmarks were updated to versions 2.12.18 and 2.13.12, respectively. The akka-uct, dotty, and scala-kmeans are now using Scala 3.3.1, and the neo4j-analytics benchmark now uses Scala 2.13. This leaves only the philosophers, reactors and stala-stm-bench7 benchmarks in the Scala 2.12 camp.

Even though the source code of most benchmarks remains unchanged (apart from bug fixes and changes due to library APIs), the actual code executed at runtime may be affected by the dependency updates.

Other changes include a fix for a memory leak in the reactors benchmark, and two fixes that improve compatibility with JMH.

The actual changes are spread over various pull requests, with the most important categorized below.

Dependency updates

• Update apache-spark benchmarks to use Spark 3.5.0 (#399) and Hadoop 3.3.6 (#425)
  • Makes the apache-spark benchmarks compatible with Java 21 (up from Java 20)
  • The Hadoop update improves compatibility with the (open) IBM Semeru OpenJ9-based JVM builds
• Update rx-scrabble benchmark to use RxJava 3.1.8 and migrate to rxjava3 API (#414)
• Update db-shootout benchmark to use H2 MVStore version 2.2.224, MapDB version 3.0.10, and Chronicle Map version 3.22.9 (#419, #412)
  • Makes the benchmark compatible with Java 18 (up from Java 11)
• Update dotty benchmark to use scala3-compiler 3.3.1 and migrate to Scala 3.3 (#407)
• Update akka-uct benchmark to use Akka Actors 2.6.32 and migrate to Scala 3.3 (#406)
• Migrate scala-stdlib to Scala 3.3 (#405)
• Update twitter-finagle benchmarks to use Finagle 22.12.0 (#404)
• Update neo4j-analytics benchmark to use Neo4j 5.12.0 and migrate to Scala 2.13 (#419, #422)
  • Makes the benchmark compatible with Java 21 (up from Java 15, but requires at least Java 17)
• Update Scala 2 benchmarks to use Scala 2.12.8 and 2.13.12 (#411)
• Update various shared dependencies (#409, #419, #423)
  • Most notably JNA 5.13.0, Netty 4.1.99, Guava 32.1.2-jre, ASM 9.6, Eclipse Collections 11.1.0, Jackson 2.15.2, and Jersey 2.40

Workload changes/fixes

• Prevent memory leaks in reactors benchmark (#389)
• Ensure that forked tasks get executed by threads from the fork-join pool in fj-kmeans (#420)
• Ensure that JMH invokes benchmark methods with correct context class loader (#421)

Build system and CI changes

• Target Java 11 as the default minimal JVM for benchmarks (#427)
• Update SBT to version 1.9.6 and JMH to version 1.37 (#398)
• Streamline CI to focus on Java LTS versions and the latest JDK only (#394, #426)

Renaissance 0.14.2

This is a maintenance release featuring minor updates to the underlying libraries to achieve the following two goals:

1. Avoid packaging vulnerable versions of the log4j framework. Even though we believe that the Renaissance suite itself (and the way it is used) is not susceptible to the vulnerabilities, the presence of JAR files with vulnerable classes can trigger alarms from various security scanners in some environments. (#366, #367)
2. Improve compatibility with some processor architectures (loongarch64) by using an updated version of the JNA library. (#371)

There have been no changes to the benchmark code.

See also the release announcement.

Renaissance 0.14.1

This release fixes race conditions that were discovered in the finagle-chirper and reactors benchmarks:

• finagle-chirper: avoid using iterator that can be modified during iteration (#356, #357)
• reactors: avoid incorrect usage of partially-initialized objects (#360)

Apart from the bug fixes, there have been no other changes to the benchmark code or the underlying libraries.

See also the release announcement.

Renaissance 0.14.0

This release contains internal cleanups and benchmark dependency updates to improve compatibility and to support execution of benchmarks using a single classloader, which makes the runtime less convoluted when using the benchmark in ahead-of-time compilation scenarios.

The source code of the benchmarks remains unchanged, but several dependencies were updated, potentially affecting the code executed by the benchmarks at runtime. The most significant change is the update of Apache Spark to version 3.2.0 and the associated migration of the apache-spark benchmarks from Scala 2.12 to Scala 2.13.

See also the release announcement.

The actual changes are spread roughly over 15 pull requests, with the most important categorized below.

Workload changes

• Update apache-spark benchmarks to use Spark 3.2.0 and Scala 2.13 (#327)
• Update Scala 2.13 benchmarks to use Scala 2.13.8 (#344)
• Update akka-uct benchmark to use Akka Actors 2.6.18 (#344)
• Update twitter-finagle benchmarks to use Finagle 21.12.0 (#344)
• Update neo4j-analytics benchmark to use Neo4j 4.4.2 (#344)
• Update future-genetic benchmark to use Jenetics 5.2.0 (#342)
• Update rx-scrabble benchmark to use RX Java 1.3.8 (#342)
• Update dotty benchmark to use Scala3 compiler 3.0.2 (#342)
• Update and commonize various dependencies (#342, #344)
  • Use common JNA version (5.10.0) to support more architectures and fix architecture mapping on SAP JVM, resolving issues with db-shootout on non-Intel architectures.
  • Use common Netty version (4.1.72) to avoid bundling several different versions and avoid component version mismatch in standalone (single classloader) benchmark execution.
  • Use common versions of several other components (see PR for details).

Workload fixes

• Expand 'Add-Opens' to prevent use of 'NilJvm' interface in Finagle (#330)

Harness changes

• Add --benchmark-metadata CLI option to override the location benchmark metadata (#341)
• Add --standalone CLI option to disable module loading (#341)
• Add Renaissance-Use-Modules manifest attribute to disable module loading (#341)
• Add plugin for collecting basic memory information (#335)
• Tolerate missing implementation of JMX system info methods (#302)

Build system and CI changes

• Update SBT to version 1.6.1 and JMH to version 1.34 (#344)
• Generate metadata-only jars that allow running benchmarks with a single class loader (#341)
  • Include renaissance-harness package for both Scala 2.12 and Scala 2.13.
• Refactor SBT project definition to speed up compilation and packaging (#336)
  • Compile benchmarks in parallel, avoid bundling duplicated jars, avoid extra filesystem copies.
  • Use JMH directly when generating benchmark JMH wrappers.
• Split testing into multiple workflows, keeping the important platforms in the main workflow (#325)

Renaissance 0.13.0

This release is primarily a JDK compatibility update, introducing support for JDK17. As a result, this version of the Renaissance suite supports JDK8, JDK11, and JDK17 on Linux, MacOS X, and Windows (on x86_64 architectures).

The changes in this release are minimal. The most important one (#297) just brings Scala 2.12 benchmarks to Scala 2.12.15, which now properly supports JDK17 and we just pass these improvements on. No benchmark code has been changed, only project documentation (#294) and parts of the harness responsible for plugin loading (#292, #293).

See also the release announcement.

Renaissance 0.12.0

Apart from a number of internal cleanups, this release focuses on improving compatibility with modern platforms, and on moving away from Scala 2.11 and Spark 2 towards Scala 2.12, Scala 2.13, and Spark 3. Please see the release announcement for a high-level overview and more details.

The changes in this release are spread over more than 30 pull requests, some of which were fairly extensive. Because it is difficult to neatly attribute individual changes to individual PRs, we only list some of the more important PRs for reference.

Workload changes

• Update benchmarks to use Scala 2.12 or 2.13 (#242)
• Clean up apache-spark benchmarks (#248)
• Update scala-dotty to use scalap sources and do hash-based validation (#263)
• Remove executor control and explicit repartitioning (and few other cleanups) (#274)
• Set benchmark-specific thread limits for the Spark local executor (#284)

Harness changes

• Remove old temporary directory API from benchmark context (#246)
• Allow overriding benchmark parameters (#262)

Build system and CI changes

• Switch to git-based versioning and rework the CI to follow suit (#253)
• Enable GitHub Actions (#271)

Renaissance 0.11.0

Workload changes:

• Avoid unnecessary I/O in page-rank (#197)
• Avoid creating response handlers per request in finagle-http (#215)
• Fix RDD caching issues and add checks in Spark workloads (#228)
• Fix resource path issues in finagle-chirper on Windows (#222)
• Fix classpath issues in dotty on Windows (#183)
• Update the Neo4J benchmark (#224)

Harness changes:

• Use single shot time mode in JMH wrappers (#185)
• Hardcode locale to improve output stability (#198)
• Force GC between iterations by default (#196)
• Add hardware counter collection plugin (#195)
• Add compilation time collection plugin (#218)
• Support named benchmark configurations (#191)
• Support benchmark parameters (#182)
• Record benchmark failure in JSON (#178)
• Record revision information in JSON (#177)
• Do not include dummy benchmarks in CLI list (#229)
• Add environment information to JSON (#216)

Renaissance 0.10.0

The following changes are part of the Renaissance release 0.10.0 :

• Introduced the scala-doku benchmark, a Sudoku solver written in Scala (#173 )
• Increased parallelism of finagle-http (#149, #148 )
• finagle-http and finagle-chirper bug fixes (#161, #164 )
• Fixed parallelism of movie-lens and als to 4 executor threads to prevent contention (#145 )
• Provide support for bundling a JMH jar (#120 )
• Support of JDK8 and JDK11 (#73, #127 )
• Support PowerPC infrastructure (#153 )

Renaissance 0.9.0

v0.9.0

Renaissance 0.9.0