Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: remove qq and parallelize build/pack/promote #1035

Merged
merged 18 commits into from Dec 5, 2022
Merged

feat: remove qq and parallelize build/pack/promote #1035

merged 18 commits into from Dec 5, 2022

Conversation

mshanemc
Copy link
Member

@mshanemc mshanemc commented Nov 22, 2022

we had to drop the --parallel flag on builds because of a bug.

remove qqjs and its qq.cd that's causing the problem with parallelization.

make sure each build process has its own directory isolation

parallelize all the things

@W-12107459@

side effects

fix #347
test: restores integration testing for Debian builds (comment said they weren't runnable on CircleCI. Works on gha)
test: runs integration tests in parallel on different OSes to test both mac and deb packing


perf metrics [mac, 2.4 GHz 8-Core Intel Core i9]

  1. clear all the cache files rm -rf dist test/tmp tmp
  2. yarn test:integration
test this PR (ms) main (ms)
macos 86,942 109,495
tarballs 281,566 696,853
win 97,470 136,077

on gha builds (2 core machine). [mac tests can't run because they only run on macs]

test this PR (ms) main (ms)
tarballs 357,739 594,614
win 67,872 101,885

@mshanemc mshanemc changed the title feat: remove qq and parallelize build/promote feat: remove qq and parallelize build/pack/promote Nov 23, 2022
package.json Outdated Show resolved Hide resolved
@@ -18,7 +17,6 @@ This can be used to create oclif CLIs that use the system node or that come prel
}

async run(): Promise<void> {
const prevCwd = qq.cwd()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mshanemc why is managing cwd no longer necessary?

Copy link
Member Author

@mshanemc mshanemc Nov 23, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all the paths are now either absolute or relative to the project.
Use of process.exec instead of qq sometimes passes in a cwd (still absolute or relative to the project).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nothing ever changes the cwd, which is what was breaking the previous attempt at parallelization

src/tarballs/config.ts Outdated Show resolved Hide resolved
* test: integration tests in parallel
* test: restore debian integration tests
Copy link
Contributor

@RodEsp RodEsp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.DS_Store shouldn't be checked in. Please remove and add to .gitignore.

@mshanemc mshanemc requested a review from RodEsp November 29, 2022 21:07
Copy link
Contributor

@RodEsp RodEsp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you do any testing on the promote command? I know integration tests don't exist for it currently but might be nice to add some with all the changes going in.

src/util.ts Show resolved Hide resolved
src/commands/pack/deb.ts Show resolved Hide resolved
src/commands/pack/win.ts Show resolved Hide resolved
src/commands/promote.ts Outdated Show resolved Hide resolved
src/commands/upload/deb.ts Outdated Show resolved Hide resolved
.gitignore Outdated Show resolved Hide resolved
mshanemc and others added 3 commits November 29, 2022 21:02
Co-authored-by: Rodrigo Espinosa de los Monteros <1084688+RodEsp@users.noreply.github.com>
@mshanemc
Copy link
Member Author

Did you do any testing on the promote command? I know integration tests don't exist for it currently but might be nice to add some with all the changes going in.

just manually. I checked it again with the new changes via.

../oclif-repos/oclif-oclif/bin/dev promote --channel qqqa --version 7.179.0 --sha 4383be1

I'm looking into the integration test options

RodEsp
RodEsp previously approved these changes Nov 30, 2022
Copy link
Contributor

@RodEsp RodEsp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Just one small comment but doesn't need to be changed to merge.

src/commands/promote.ts Show resolved Hide resolved
src/commands/promote.ts Show resolved Hide resolved
@RodEsp RodEsp dismissed peternhale’s stale review November 30, 2022 16:15

Comments were addressed. Re-review please.

package.json Outdated Show resolved Hide resolved
test/integration/publish.test.ts Show resolved Hide resolved
test/integration/publish.test.ts Show resolved Hide resolved
RodEsp
RodEsp previously approved these changes Nov 30, 2022
RodEsp
RodEsp previously approved these changes Nov 30, 2022
@RodEsp RodEsp merged commit db2a858 into main Dec 5, 2022
@RodEsp RodEsp deleted the sm/no-qq branch December 5, 2022 15:05
mattwebbio pushed a commit to appfolio/oclif that referenced this pull request Dec 29, 2022
* feat: remove qq and parallelize build/promote

* chore: reformat pjson

* refactor: remove redundant objects

* refactor: pr feedback

* feat: publishes debian packages with qq

* test: integration tests in parallel
* test: restore debian integration tests

* chore: remove ds_store

* chore: ignore dsStore

* chore: ignore dist

* Update src/commands/upload/deb.ts

Co-authored-by: Rodrigo Espinosa de los Monteros <1084688+RodEsp@users.noreply.github.com>

* fix: cwd on pretarball

* refactor: pr feedback

* test: integration test for tarball promote

* test: revert unnecessary  trims

* perf: optional xz hashes in parallel

* chore: fix pjson format

* refactor: remove unused conditional

* fix: osslsign command requires "sign"

Co-authored-by: Rodrigo Espinosa de los Monteros <1084688+RodEsp@users.noreply.github.com>
// this workaround puts the code in both places that the redirect was doing
// with this, the docs are correct. The copies are all done in parallel so it shouldn't be too costly.
const uploadWorkaround = (file: string) => {
const cloudKey = `${cloudKeyBase}/apt/./${file}`
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing the ./ fixes the issue here

#1074

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Cannot install .deb package: "The repository '/apt ./ Release' does not have a Release file."
4 participants