Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: use os.availableParallelism if available #13738

Merged
merged 6 commits into from Jan 6, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
3 changes: 2 additions & 1 deletion CHANGELOG.md
Expand Up @@ -2,8 +2,9 @@

### Features

- `[@jest/globals, jest-mock]` Add `jest.replaceProperty()` that replaces property value ([#13496](https://github.com/facebook/jest/pull/13496))
- `[expect, @jest/expect-utils]` Support custom equality testers ([#13654](https://github.com/facebook/jest/pull/13654))
- `[jest-config, jest-worker]` Use `os.availableParallelism` if available to calculate number of workers to spawn ([#13738](https://github.com/facebook/jest/pull/13738))
- `[@jest/globals, jest-mock]` Add `jest.replaceProperty()` that replaces property value ([#13496](https://github.com/facebook/jest/pull/13496))
- `[jest-haste-map]` ignore Sapling vcs directories (`.sl/`) ([#13674](https://github.com/facebook/jest/pull/13674))
- `[jest-resolve]` Support subpath imports ([#13705](https://github.com/facebook/jest/pull/13705), [#13723](https://github.com/facebook/jest/pull/13723))
- `[jest-runtime]` Add `jest.isolateModulesAsync` for scoped module initialization of asynchronous functions ([#13680](https://github.com/facebook/jest/pull/13680))
Expand Down
4 changes: 4 additions & 0 deletions packages/jest-config/src/__mocks__/os.js
Expand Up @@ -17,4 +17,8 @@ function __setCpus(newCpus) {
os.__setCpus = __setCpus;
os.cpus = jest.fn(() => cpus);

if (typeof os.availableParallelism === 'function') {
os.availableParallelism = jest.fn(() => cpus?.length ?? 0);
}

module.exports = os;
17 changes: 13 additions & 4 deletions packages/jest-config/src/getMaxWorkers.ts
Expand Up @@ -5,9 +5,19 @@
* LICENSE file in the root directory of this source tree.
*/

import {cpus} from 'os';
import {
// @ts-expect-error - added in Node 19.4.0
availableParallelism,
cpus,
} from 'os';
import type {Config} from '@jest/types';

function getNumCpus(): number {
return typeof availableParallelism === 'function'
? availableParallelism()
: cpus()?.length ?? 1;
}

export default function getMaxWorkers(
argv: Partial<
Pick<Config.Argv, 'maxWorkers' | 'runInBand' | 'watch' | 'watchAll'>
Expand All @@ -22,8 +32,7 @@ export default function getMaxWorkers(
return parseWorkers(defaultOptions.maxWorkers);
} else {
// In watch mode, Jest should be unobtrusive and not use all available CPUs.
const cpusInfo = cpus();
const numCpus = cpusInfo?.length ?? 1;
const numCpus = getNumCpus();
const isWatchModeEnabled = argv.watch || argv.watchAll;
return Math.max(
isWatchModeEnabled ? Math.floor(numCpus / 2) : numCpus - 1,
Expand All @@ -41,7 +50,7 @@ const parseWorkers = (maxWorkers: string | number): number => {
parsed > 0 &&
parsed <= 100
) {
const numCpus = cpus().length;
const numCpus = getNumCpus();
const workers = Math.floor((parsed / 100) * numCpus);
return Math.max(workers, 1);
}
Expand Down
13 changes: 11 additions & 2 deletions packages/jest-repl/src/cli/runtime-cli.ts
Expand Up @@ -5,7 +5,11 @@
* LICENSE file in the root directory of this source tree.
*/

import {cpus} from 'os';
import {
// @ts-expect-error - added in Node 19.4.0
availableParallelism,
cpus,
} from 'os';
import * as path from 'path';
import * as util from 'util';
import chalk = require('chalk');
Expand Down Expand Up @@ -70,8 +74,13 @@ export async function run(
};

try {
const numCpus: number =
typeof availableParallelism === 'function'
? availableParallelism()
: cpus().length;

const hasteMap = await Runtime.createContext(projectConfig, {
maxWorkers: Math.max(cpus().length - 1, 1),
maxWorkers: Math.max(numCpus - 1, 1),
watchman: globalConfig.watchman,
});

Expand Down
15 changes: 13 additions & 2 deletions packages/jest-worker/src/index.ts
Expand Up @@ -5,7 +5,11 @@
* LICENSE file in the root directory of this source tree.
*/

import {cpus} from 'os';
import {
// @ts-expect-error - added in Node 19.4.0
availableParallelism,
cpus,
} from 'os';
import {isAbsolute} from 'path';
import Farm from './Farm';
import WorkerPool from './WorkerPool';
Expand Down Expand Up @@ -54,6 +58,12 @@ function getExposedMethods(
return exposedMethods;
}

function getNumberOfCpus(): number {
return typeof availableParallelism === 'function'
? availableParallelism()
: cpus().length;
}

/**
* The Jest farm (publicly called "Worker") is a class that allows you to queue
* methods across multiple child processes, in order to parallelize work. This
Expand Down Expand Up @@ -98,7 +108,8 @@ export class Worker {
forkOptions: this._options.forkOptions ?? {},
idleMemoryLimit: this._options.idleMemoryLimit,
maxRetries: this._options.maxRetries ?? 3,
numWorkers: this._options.numWorkers ?? Math.max(cpus().length - 1, 1),
numWorkers:
this._options.numWorkers ?? Math.max(getNumberOfCpus() - 1, 1),
resourceLimits: this._options.resourceLimits ?? {},
setupArgs: this._options.setupArgs ?? [],
};
Expand Down
7 changes: 6 additions & 1 deletion scripts/buildTs.mjs
Expand Up @@ -110,7 +110,12 @@ try {
console.log(chalk.inverse(' Validating TypeScript definition files '));

// we want to limit the number of processes we spawn
const cpus = Math.max(1, os.cpus().length - 1);
const cpus = Math.max(
1,
(typeof os.availableParallelism === 'function'
? os.availableParallelism()
: os.cpus().length) - 1,
);

const typesReferenceDirective = '/// <reference types';
const typesNodeReferenceDirective = `${typesReferenceDirective}="node" />`;
Expand Down
8 changes: 7 additions & 1 deletion scripts/lintTs.mjs
Expand Up @@ -14,7 +14,13 @@ import pLimit from 'p-limit';
import {getPackagesWithTsConfig} from './buildUtils.mjs';

// we want to limit the number of processes we spawn
const cpus = Math.max(1, os.cpus().length - 1);
const cpus = Math.max(
1,
(typeof os.availableParallelism === 'function'
? os.availableParallelism()
: os.cpus().length) - 1,
);

const mutex = pLimit(cpus);

const fix = process.argv.slice(2).some(arg => arg === '--fix');
Expand Down