actions/setup-java@v2 - Support different distributions

[maxim-lobanov]

Description:
This pull-request reworks installer approach and create abstract base that can be used for adding new Java distributions. Also this pull-request adds support for Azul and Adoptium (AdoptOpenJDK) distributions.
These changes are going to be released as actions/setup-java@v2 so breaking changes are expected.

Note: we have decided to split changes to a few pull-requests to simplify review process:

• Installers: (this pull-request) (workflows are temporary removed, they are added back in next PR)
• Tests: actions/setup-java@v2 - Test coverage #133
• Docs: actions/setup-java@v2 - Documentation #134
• Related ADR: V2 setup-java ADR #97

A few technical notes:

1. Added distribution input without default value.
2. Added support for two distributions: zulu (Azul OpenJDK), adoptium (Adopt OpenJDK)
3. The approach for searching Azul Java versions was reworked from scratch. Previously, actions/setup-java@v1 parsed html page https://static.azul.com/zulu/bin/ and use regex to find Java versions.
   We have reworked it to use Azul API: https://app.swaggerhub.com/apis-docs/azul/zulu-download-community/1.0. Potentially, it could cause some differences in version resolving but we have tested all major use-cases and didn't find any issues.
4. Use Adopt API V3 https://api.adoptopenjdk.net/swagger-ui/ to get the list of versions. Unfortunately, API has very hard limitation (20 items per page) so need to paginate pages.
5. Use the following tool naming in toolcache: Java_${distribution}_${packageType} to make sure that we can cache different vendors side by side.
   Also, adds -ea postfix for version in toolcache to distinguish stable and unstable versions (in V1 - it could cause issues)
6. Bumped all dependencies to latest versions (no risk since it is major release)
7. We have decided to make distribution and java-package properties optional to simplify use-cases when task is used for setting up Maven publishing (settings.xml), customers don't really need to spend time on Java setup. So if distribution and java-version properties are specified - Java will be installed. Otherwise - only settings.xml will be created. Also both distribution and java-package should be specified together (if only one is specified - error will be raised)
8. Java versions are stored in tool-cache with 3 digits like (11.0.3) despite the fact that Java versioning is 4 digits (the fourth is build). The same behavior was in V1. It could cause some confusing but neither V1 nor V2 support specifying versions with 4 digits in input - so it shouldn't be an issue. Also, looks like toolkit/toolcache doesn't support 4 digits versions because of semver.clean invocation.
9. Minor refactoring of auth / gpg code: moving methods to the separate files, replace console.log with core.info, apply styling. No major changes there.
10. Remove version input in favor of java-version (V1 supported both for backward compatibility)

[konradpabjan]

Looking great! Did an initial sweep, just a few things

[giltene]

FYI, the Disco API (https://github.com/foojay2020/discoapi) is coming along nicely, with multiple tools starting to use it. It provides a sweeping solution for the "find a JDK" task across multiple distros, and an actions/setup-java PR is coming soon for using it directly, replacing the current distro-specific code, but keeping all existing functionality. This will likely overlap with this PR.

See some FOSDEM discussion from last month: https://fosdem.org/2021/schedule/event/discoapi/

[joschi]

FYI, the Disco API (https://github.com/foojay2020/discoapi) is coming along nicely, with multiple tools starting to use it.

Where might I have seen that before?
#13 (comment)

[konradpabjan]

Down the road we can consider utilizing the Disco API or any other tools to expand support for more distributions. With the initial v2 preview and release we're primarily focusing on just expanding support for Adoptium and ironing out any bugs that might come up since that was the biggest ASK from users.

[maxim-lobanov]

@maxim : On the caching subject: can we discuss this in some more detail? [and is there a separate thread to do that on?] E.g. will caching check that the cached versions are actually identical (e.g. have the same checksum or hash) as the thing that would have been downloaded had a cached version not been found? There are many update scenarios (e.g. whenever quarterly OpenJDK updates are rolled out, and whenever out-of-cycle critical updates are released, both of which are frequent enough to not ignore) where stale cache decisions could be very problematic, and such decisions an be avoided if the cache logic verifies the checksums of what is available in the cache against the checksums of what would be downloaded.

@giltene , caching logic will work the same way like it works for other tools (go, node, python, haskell) - compare version only, not checksum.
Images will contain latest version of every LTS and will be updated with 1-week delay.
First, action tries to find version that satisfies input locally. If it is not found locally, it will download and install version in runtime.
If you specify exact version like 11.0.10+9 but image contains only 11.0.10+8, then 11.0.10+9 will be downloaded in runtime.

[maxim-lobanov]

@sormuras , we have validated actions/setup-java@v2 with your repository: https://github.com/dmitry-shibanov/download-jdk/actions/runs/645961061 - it works as expected

[ibauersachs]

@maxim-lobanov I'm a somewhat heavy user of setup-java and cannot think of a single reason not to specify java-version. The Maven/Gradle setup is a convencience (that I like), but if that was the only thing I'd need, I would definitely not use setup-java. Please, make the java-version and distribution properties mandatory!

[maxim-lobanov]

Thank you for providing your input, I have pushed PR update to mark java-version and distribution properties as required.

[giltene]

,,,Images will contain latest version of every LTS and will be updated with 1-week delay.
First, action tries to find version that satisfies input locally. If it is not found locally, it will download and install version in runtime.
If you specify exact version like 11.0.10+9 but image contains only 11.0.10+8, then 11.0.10+9 will be downloaded in runtime.

@maxim-lobanov I think this will cause a lot of issues. Bringing in a later-than-cached-version only if it is explicitly specified is both counter-intuitive and will likely break many of the commonly relied on behaviors of setup-java.

For example, a common [best] practice is to run tests both against a statically and explicitly selected update version (e.g. "11.0.7") and the latest update version (e.g. "11") of the same java versions, for any version that is currently maintained and has new updates posted multiple times per year. The simple reason for this is to help triage and identify whether tests break because of a change in the jdk or a change in the repo. It is [unfortunately] common for new updates to break things. E.g. just in the recent few months, OpenJDK 8u262, 8u272, and 11.0.9 all had significant real-world breakages found post release, leading to the release of the out of cycle 8u265, 8u275, and 11.0.9.1 within weeks. Things that had such breakages revealed in post-release tests included e.g. hadoop and solr.

A one week delay in testing new updates due to caching of e.g. LTS updates would create wide ranging negative impacts, both for the tested repos (which will find out a week late that stuff is newly broken that they may need to be aware of work around, or that new out of cycle stuff has a fix that needs to be urgently used), and for OpenJDK in general: the wide ranging testing of new updates in the hours and days immediately after release can help drive the feedback that identify these things. With 60K+ Github workflows currently testing openjdk updates as they appear, this quick post-release exposure is very helpful, and delaying it by a week would be sad.

I'd like to suggest that, at the very least, the lookup and resolution of the semver specified versions to URIs be done on each run with no attempt to cache that logic, but that the download URI itself be used as a key for caching. This way updates will show up immediately in tests that run after they appear. A preferred solution would cache the hash or checksum to make sure that even replaced binaries at the same URI are treated properly, but URI based caching is probably the next best thing.

[maxim-lobanov]

By default, action should always check cache firstly. The most of users prefer to use cached version (speed up their builds by 10-30 seconds and more reliable since there is no any change to fail randomly via connection issue) but get up-to-date version with a few days delay. The same way like actions/setup-go, actions/setup-python, actions/setup-node work.

The use-case that you have described makes sense but it shouldn't be a default behavior. For Node.JS, it is achieved via separate flag check-latest so probably we should consider adding something similar to cover this use-case.

[maxim-lobanov]

Merged these changes to v2-preview branch. Feel free to create an issue for any suggestions / found bugs.
The v2-preview version can be consumed via

- name: setup-java
  uses: actions/setup-java@v2-preview
  with:
    distribution: 'adopt'
    java-version: '11'

[ibauersachs]

I have to admit, I don't know where the cache duration is handled. But I have to concur with @giltene's reasoning.

By default, action should always check cache firstly. The most of users prefer to use cached version (speed up their builds by 10-30 seconds and more reliable since there is no any change to fail randomly via connection issue) but get up-to-date version with a few days delay.

Caching, yes, please. But definitely not for a week! If I specify java-version: 11 I expect to get the newest release within hours of it's availability.

Also, you have discarded my comment about downloading the entire list of available JDKs and the possible connection failures. Now you're using the exact same argument for an insanely long cache timeout.
You can always fall back to using the cache (if available) on connection failures and report a warning.

The same way like actions/setup-go, actions/setup-python, actions/setup-node work.
The use-case that you have described makes sense but it shouldn't be a default behavior. For Node.JS, it is achieved via separate flag check-latest so probably we should consider adding something similar to cover this use-case.

A flag/timeout option is fine with me, but it should default to getting new releases fast.

[giltene]

Caching the contents of a chosen JDK download makes perfect sense, and can save 20-30 seconds and a bunch of bandwidth. Caching the decision of which JDK to choose to download does not. It would only save a small fraction of a second and practically no bandwidth, and lead to operational changes compared to the current behavior seen by the many users of 1.x.

In addition, inconsistent functional behavior that depends on under-the-hood choice to selectively cache JDKs (only LTS? Only some distros?) in ways that cause widely varying behavior in terms of how up to date the JDK is (compared to what is currently available for that version spec and distro) will add “surprising” outcomes. Java has an industry-aligned cadence for quarterly updates (across all coordinating distros), so varying this behavior by version spec and distro will lead to some weird stuff.

is there any reason not to just reactively cache all resolved JDK URIs for some reasonable amount of time? Or maybe cache the few hundred recent or hotly used ones if the number or space is a concern?

[ibauersachs]

@maxim-lobanov would you please get back to the caching issue? Just not answerig anymore isn't a solution. Or did I miss a separate thread?

[maxim-lobanov]

As for the caching timeline - Hosted images are updated on weekly basis so the image cache can't be updated more often.
As for the caching behavior:

Context: we have an image with cached Java 11.0.7+3. Yesterday, Java 11.0.7+4 was released.
User specifies task input java-version: 11

We have two different behavior options:

1. Java 11.0.7+3 will be chosen without delay and without checking remote source since it is in cache and satisfies user's input. User will be able to specify exact version 11.0.7+4 as input to get the new version earlier.
2. Action will check the latest version and find that 11.0.7+4 is available. Then action will download and install 11.0.7+4 in runtime with some time delay.

@konradpabjan , The first option will satisfy users who more interested in getting fast and lower-cost builds and don't care much about 1-week delay with updates. We use this approach with other tools and it works perfectly.
The both cases make sense for me but in my understanding, the default task behavior should be the 1st option to use image cache as much as possible and reduce customers' build time like other 1st-party actions do.

In setup-node, we resolved this case with additional flag like check-latest that forces action to use the 2nd option and keeping 1st option by default.

[ibauersachs]

As @giltene already mentioned, the delay of case 2) is only there if there is actually a newer version. In the majority of builds, there is no newer version and the only delay is the API call, not the actual JRE/JDK download.
If you're so concerned about API call timeouts and delays, make the behavior configurable with check-latest and reduce the amount of API calls necessary to retrieve the wanted version.

[konradpabjan]

The first option will satisfy users who more interested in getting fast and lower-cost builds and don't care much about 1-week delay with updates. We use this approach with other tools and it works perfectly.

I really like the approach that setup-node takes. I think most users would prefer to have their workflows just run fast and use whatever is cached on the image by default. In the scenario where a very recent version is needed that is not cached then an extra input such as check-latest makes sense.

@maxim-lobanov let's try to squeeze this into v2-preview 😃

Thanks everyone for the feedback and discussion!

[dmitry-shibanov]

Hello everyone. We prepared pull request with adding check-latest flag. At the moment the tests are failing but it will be fixed by this pull request.