Rework release CI/CD for efficiency #409
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Brooks Townsend <brooks@cosmonic.com> added Cross.toml Signed-off-by: Brooks Townsend <brooks@cosmonic.com> corrected the artifact name Signed-off-by: Brooks Townsend <brooks@cosmonic.com>
Signed-off-by: Brooks Townsend <brooks@cosmonic.com> debug check paths Signed-off-by: Brooks Townsend <brooks@cosmonic.com> dll not exe Signed-off-by: Brooks Townsend <brooks@cosmonic.com> WINDOWSS Signed-off-by: Brooks Townsend <brooks@cosmonic.com> giving a full build a try Signed-off-by: Brooks Townsend <brooks@cosmonic.com> renamed needs sections Signed-off-by: Brooks Townsend <brooks@cosmonic.com> fix small mv error, test docker build too Signed-off-by: Brooks Townsend <brooks@cosmonic.com> correct windows target, copy into dockerfile Signed-off-by: Brooks Townsend <brooks@cosmonic.com>
Signed-off-by: Brooks Townsend <brooks@cosmonic.com> redid some paths Signed-off-by: Brooks Townsend <brooks@cosmonic.com> added built Signed-off-by: Brooks Townsend <brooks@cosmonic.com> updated mix env to release_prod Signed-off-by: Brooks Townsend <brooks@cosmonic.com> added release prod file Signed-off-by: Brooks Townsend <brooks@cosmonic.com> removed executable for windows, ls for debugging Signed-off-by: Brooks Townsend <brooks@cosmonic.com> MIX ENVIRONMENT WINDOWS GAH Signed-off-by: Brooks Townsend <brooks@cosmonic.com> corrected docker path Signed-off-by: Brooks Townsend <brooks@cosmonic.com> see if anything else is broken Signed-off-by: Brooks Townsend <brooks@cosmonic.com> compile esbuild in release_prod, correct mix env in docker Signed-off-by: Brooks Townsend <brooks@cosmonic.com> added file debug Signed-off-by: Brooks Townsend <brooks@cosmonic.com> updated deps, use macos-11 Signed-off-by: Brooks Townsend <brooks@cosmonic.com> use release prod for docker Signed-off-by: Brooks Townsend <brooks@cosmonic.com> create the directory first just in case Signed-off-by: Brooks Townsend <brooks@cosmonic.com> create dir Signed-off-by: Brooks Townsend <brooks@cosmonic.com> install specific otp/elixir version Signed-off-by: Brooks Townsend <brooks@cosmonic.com> updated dart saas, create RELEASES Signed-off-by: Brooks Townsend <brooks@cosmonic.com> remove unused targets for Rust Signed-off-by: Brooks Townsend <brooks@cosmonic.com> reintroduce aarch64 linux Signed-off-by: Brooks Townsend <brooks@cosmonic.com> don't tar up the release Signed-off-by: Brooks Townsend <brooks@cosmonic.com> upload correct file Signed-off-by: Brooks Townsend <brooks@cosmonic.com> updated distillery Signed-off-by: Brooks Townsend <brooks@cosmonic.com> updated makefiles Signed-off-by: Brooks Townsend <brooks@cosmonic.com> corrected filename Signed-off-by: Brooks Townsend <brooks@cosmonic.com>
brooksmtownsend
force-pushed
the
ci/precompiled-nifs
branch
from
9cb8b28
to
35013f9
Compare
Signed-off-by: Brooks Townsend <brooks@cosmonic.com>
Signed-off-by: Brooks Townsend <brooks@cosmonic.com>
Signed-off-by: Brooks Townsend <brooks@cosmonic.com>
Signed-off-by: Brooks Townsend <brooks@cosmonic.com>
Signed-off-by: Brooks Townsend <brooks@cosmonic.com>
Signed-off-by: Brooks Townsend <brooks@cosmonic.com>
Signed-off-by: Brooks Townsend <brooks@cosmonic.com>
Signed-off-by: Brooks Townsend <brooks@cosmonic.com>
Signed-off-by: Brooks Townsend <brooks@cosmonic.com>
Signed-off-by: Brooks Townsend <brooks@cosmonic.com>
Signed-off-by: Brooks Townsend <brooks@cosmonic.com>
Signed-off-by: Brooks Townsend <brooks@cosmonic.com>
Signed-off-by: Brooks Townsend <brooks@cosmonic.com>
Signed-off-by: Brooks Townsend <brooks@cosmonic.com>
Signed-off-by: Brooks Townsend <brooks@cosmonic.com>
Signed-off-by: Brooks Townsend <brooks@cosmonic.com>
brooksmtownsend
requested review from
autodidaddict,
thomastaylor312 and
stevelr
as code owners
Signed-off-by: Brooks Townsend <brooks@cosmonic.com>
thomastaylor312
approved these changes
Jun 8, 2022
| target: | ||
| - x86_64-unknown-linux-gnu | ||
| - aarch64-unknown-linux-gnu | ||
| - x86_64-apple-darwin |
There was a problem hiding this comment.
You can build for an m1 on an x86_mac. Should we have a separate step that runs-on mac and builds the nif for m1 macs?
There was a problem hiding this comment.
NOTE: I don't know if we could build the rest of the elixir stuff as well, but I know we can at least get the nif
There was a problem hiding this comment.
I can definitely build the NIF for m1 mac, even with cross here but I don't have a way to get the erlang runtime system for the m1 mac (I tested and verified that x86-macos.tar.gz that we normally publish doesn't work even with rosetta but that was a quick test)
There was a problem hiding this comment.
We could possibly get the elixir stuff by running the erlang docker image with --platform linux/arm64/v8 and downloading the erts folder... but it's a stretch and would require rearranging all of this to not package the erts with the release
(This is something I really want to do, but I'm not sure if it's best done now since this PR has been in progress for a while)
There was a problem hiding this comment.
Yeah no need to block on this. It is just a nit
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR reworks the release CI/CD for our host_core and wasmcloud_host releases to use precompiled NIFs while still preserving the development experience of using a local cargo install. This is under the
release_prodprofile to avoid confusion. Additionally, as you can see in a mostly successful run on https://github.com/wasmCloud/wasmcloud-otp/actions/runs/2457366553, this cuts the total release time down to around 40 minutes. From tag push to releasing new tarballs is around 15 minutes, and tag push to new docker images (yes, even the aarch64 one) is around 30 minutes.I'm blocked here as there is a specific combination of OTP/Elixir versions that we're forced to use with homebrew (currently investigating ways to manually select a homebrew version) that Distillery isn't quite updated for yet. See issue here: bitwalker/distillery#744. For now, I'm resorting to using ASDF on macos to install erlang/elixir specific versions as my homebrew tap hack didn't end up working.I started this PR with the notion that this would be a good way to upgrade to the self-extracting tarball that distillery offers, but it isn't supported on Windows, and with the general unmaintained feel of the distillery package I would suggest we move to mix release, the officially reported release mechanism, instead. mix release also doesn't support self-extracting tarballs, which is unfortunate, but at the very least changing to the executable tarball format now would require a breaking change if we moved to mix releases.