Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(next/swc): allow to run custom turbopack binary #42656

Merged
merged 7 commits into from Nov 15, 2022
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 2 additions & 0 deletions packages/next/build/swc/index.d.ts
Expand Up @@ -9,3 +9,5 @@ export function initCustomTraceSubscriber(traceFileName?: string): void
export function teardownTraceSubscriber(): void
export function teardownCrashReporter(): void
export function loadBindings(): Promise<void>

export function __isCustomTurbopackBinary(): Promise<boolean>
84 changes: 81 additions & 3 deletions packages/next/build/swc/index.js
Expand Up @@ -13,6 +13,30 @@ const ArchName = arch()
const PlatformName = platform()
const triples = platformArchTriples[PlatformName][ArchName] || []

// Allow to specify an absolute path to the custom turbopack binary to load.
// If one of env variables is set, `loadNative` will try to use any turbo-* interfaces from specified
// binary instead. This will not affect existing swc's transform, or other interfaces. This is thin,
// naive interface - `loadBindings` will not validate neither path nor the binary.
//
// Note these are internal flag: there's no stability, feature gaurentee.
const __INTERNAL_CUSTOM_TURBOPACK_BINARY =
process.env.__INTERNAL_CUSTOM_TURBOPACK_BINARY
const __INTERNAL_CUSTOM_TURBOPACK_BINDINGS =
process.env.__INTERNAL_CUSTOM_TURBOPACK_BINDINGS
export const __isCustomTurbopackBinary = async () => {
if (
!!__INTERNAL_CUSTOM_TURBOPACK_BINARY &&
!!__INTERNAL_CUSTOM_TURBOPACK_BINDINGS
) {
throw new Error('Cannot use TURBOPACK_BINARY and TURBOPACK_BINDINGS both')
}

return (
!!__INTERNAL_CUSTOM_TURBOPACK_BINARY ||
!!__INTERNAL_CUSTOM_TURBOPACK_BINDINGS
)
}

// These are the platforms we'll try to load wasm bindings first,
// only try to load native bindings if loading wasm binding somehow fails.
// Fallback to native binding is for migration period only,
Expand All @@ -39,6 +63,7 @@ export async function loadBindings() {
if (pendingBindings) {
return pendingBindings
}
const isCustomTurbopack = await __isCustomTurbopackBinary()
pendingBindings = new Promise(async (resolve, reject) => {
if (!lockfilePatchPromise.cur) {
// always run lockfile check once so that it gets patched
Expand All @@ -62,7 +87,7 @@ export async function loadBindings() {
}

try {
return resolve(loadNative())
return resolve(loadNative(isCustomTurbopack))
} catch (a) {
attempts = attempts.concat(a)
}
Expand Down Expand Up @@ -238,7 +263,7 @@ async function loadWasm(importPath = '') {
throw attempts
}

function loadNative() {
function loadNative(isCustomTurbopack = false) {
if (nativeBindings) {
return nativeBindings
}
Expand Down Expand Up @@ -351,7 +376,60 @@ function loadNative() {
...options,
noOpen: options.noOpen ?? true,
}
bindings.startTurboDev(toBuffer(devOptions))

if (!isCustomTurbopack) {
bindings.startTurboDev(toBuffer(devOptions))
} else if (!!__INTERNAL_CUSTOM_TURBOPACK_BINARY) {
console.warn(
`Loading custom turbopack binary from ${__INTERNAL_CUSTOM_TURBOPACK_BINARY}`
)

return new Promise((resolve, reject) => {
const spawn = require('next/dist/compiled/cross-spawn')
const args = []

Object.entries(devOptions).forEach(([key, value]) => {
let cli_key = `--${key.replace(
/[A-Z]/g,
(m) => '-' + m.toLowerCase()
)}`
if (key === 'dir') {
args.push(value)
} else if (typeof value === 'boolean' && value === true) {
args.push(cli_key)
} else if (typeof value !== 'boolean' && !!value) {
args.push(cli_key, value)
}
})

console.warn(`Running turbopack with args: [${args.join(' ')}]`)

const child = spawn(__INTERNAL_CUSTOM_TURBOPACK_BINARY, args, {
stdio: 'inherit',
env: {
...process.env,
},
})
child.on('close', (code) => {
if (code !== 0) {
reject({
command: `${__INTERNAL_CUSTOM_TURBOPACK_BINARY} ${args.join(
' '
)}`,
})
return
}
resolve(0)
})
})
} else if (!!__INTERNAL_CUSTOM_TURBOPACK_BINDINGS) {
console.warn(
`Loading custom turbopack bindings from ${__INTERNAL_CUSTOM_TURBOPACK_BINARY}`
)
console.warn(`Running turbopack with args: `, devOptions)

require(__INTERNAL_CUSTOM_TURBOPACK_BINDINGS).startDev(devOptions)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
require(__INTERNAL_CUSTOM_TURBOPACK_BINDINGS).startDev(devOptions)
require(__INTERNAL_CUSTOM_TURBOPACK_BINDINGS).startTurboDev(toBuffer(devOptions))

Is this meant to be a custom swc/index.js that wraps loading the bindings again or the napi build itself?

Copy link
Contributor Author

@kwonoj kwonoj Nov 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Expected usecase is __INTERNAL_CUSTOM_TURBOPACK_BINDINGS points to the native napi bindings directly (.node), reason this does not passes Buffer for the options. It is up to caller's responsibility to prepare bindings to work with suggested interface, either if it's custom wrapped .js or direct call to .node binary.

Copy link
Contributor Author

@kwonoj kwonoj Nov 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I honestly did not throughly consider possible edge cases for this path, as turbopack itself currently dose not produce napi bindings binaries yet. Potentially there'll be a couple of iteration based on turbopack side changes.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah can be updated in follow-ups as needed from testing

}
},
startTrace: (options = {}) =>
bindings.runTurboTracing(toBuffer({ exact: true, ...options })),
Expand Down