Add entrypoint tracing #25538

ijjk · 2021-05-28T04:20:22Z

This adds tracing entrypoints directly after they have have been transpiled to allow us to trace before the webpack runtime has been added to the modules. This should allow for more accurate tracing of entrypoints and allow the trace step to be cached.

Bug

Related issues linked using fixes #number
Integration tests added

x-ref: #24700
x-ref: #26200
x-ref: #23894
x-ref: #25431
x-ref: #8251

packages/next/build/webpack/plugins/next-trace-entrypoints-plugin.ts

sokra · 2021-07-01T12:16:33Z

packages/next/build/webpack/plugins/next-trace-entrypoints-plugin.ts

+              }
+
+              try {
+                return compilation.inputFileSystem.readFileSync(path)


Reading the files directly from fs is probably not the best idea.

A loader could have modified the source code after filesystem. Or it could be transpiled from a language nft doesn't understand.

There is a NormalModule.originalSource() function which gives you the source code for a module after loader processing. That's in a language webpack understands, so either javascript, or something else you can ignore. Best check Module.type for javascript/* to only process JS code.

Note that some modules might not end up at all in the output, so best use the list of chunks from createTraceAssets and grab all modules from there, de-duplicate them, analyse them with nft for more references, merge and cache the results per chunk and add them to the analysis data while iterating over the chunks.

It looks like we can't use the list of files from createTraceAssets during this hook since we need it to be immediately after transpiling with babel and compilation.entrypoints.values() seems to be empty at this stage. I added some additional tests and gathering the files from the entry's module dependencies seems to gather everything we need currently and allows us to use the dependencies source if available as well.

sokra · 2021-07-01T12:16:59Z

packages/next/build/webpack/plugins/next-trace-entrypoints-plugin.ts

+            }
+            const readlink = (path: string) => {
+              try {
+                return compilation.inputFileSystem.readlinkSync(path)


sync fs calls might have a performance influence...

It seems node-file-trace currently relies on sync fs calls, ideally we could get these from the webpack cache so that it's not actually hitting the filesystem for most of these calls

packages/next/taskfile.js

changes applied

ijjk · 2021-08-16T19:09:10Z

Stats from current PR

Default Build (Increase detected ⚠️)

General Overall increase ⚠️

	vercel/next.js canary	ijjk/next.js add/nft	Change
buildDuration	13.6s	13.4s	-160ms
buildDurationCached	3.4s	3.4s	⚠️ +59ms
nodeModulesSize	49.1 MB	49.7 MB	⚠️ +591 kB

Page Load Tests Overall increase ✓

	vercel/next.js canary	ijjk/next.js add/nft	Change
/ failed reqs	0	0	✓
/ total time (seconds)	2.676	2.625	-0.05
/ avg req/sec	934.19	952.54	+18.35
/error-in-render failed reqs	0	0	✓
/error-in-render total time (seconds)	1.436	1.435	0
/error-in-render avg req/sec	1741.03	1742.56	+1.53

Client Bundles (main, webpack, commons)

	vercel/next.js canary	ijjk/next.js add/nft	Change
745.HASH.js gzip	179 B	179 B	✓
framework-HASH.js gzip	42.2 kB	42.2 kB	✓
main-HASH.js gzip	23.1 kB	23.1 kB	✓
webpack-HASH.js gzip	1.44 kB	1.44 kB	✓
Overall change	67 kB	67 kB	✓

Legacy Client Bundles (polyfills)

	vercel/next.js canary	ijjk/next.js add/nft	Change
polyfills-a4..dd70.js gzip	31 kB	31 kB	✓
Overall change	31 kB	31 kB	✓

Client Pages

	vercel/next.js canary	ijjk/next.js add/nft	Change
_app-HASH.js gzip	980 B	980 B	✓
_error-HASH.js gzip	194 B	194 B	✓
amp-HASH.js gzip	312 B	312 B	✓
css-HASH.js gzip	329 B	329 B	✓
dynamic-HASH.js gzip	2.64 kB	2.64 kB	✓
head-HASH.js gzip	350 B	350 B	✓
hooks-HASH.js gzip	904 B	904 B	✓
image-HASH.js gzip	4.13 kB	4.13 kB	✓
index-HASH.js gzip	261 B	261 B	✓
link-HASH.js gzip	1.66 kB	1.66 kB	✓
routerDirect..HASH.js gzip	319 B	319 B	✓
script-HASH.js gzip	387 B	387 B	✓
withRouter-HASH.js gzip	320 B	320 B	✓
bb14e60e810b..30f.css gzip	125 B	125 B	✓
Overall change	12.9 kB	12.9 kB	✓

Client Build Manifests

	vercel/next.js canary	ijjk/next.js add/nft	Change
_buildManifest.js gzip	491 B	491 B	✓
Overall change	491 B	491 B	✓

Rendered Page Sizes

	vercel/next.js canary	ijjk/next.js add/nft	Change
index.html gzip	540 B	540 B	✓
link.html gzip	551 B	551 B	✓
withRouter.html gzip	532 B	532 B	✓
Overall change	1.62 kB	1.62 kB	✓

Webpack 4 Mode (Increase detected ⚠️)

General Overall increase ⚠️

	vercel/next.js canary	ijjk/next.js add/nft	Change
buildDuration	10.9s	11s	⚠️ +49ms
buildDurationCached	4.5s	4.5s	⚠️ +19ms
nodeModulesSize	49.1 MB	49.7 MB	⚠️ +591 kB

Page Load Tests Overall increase ✓

	vercel/next.js canary	ijjk/next.js add/nft	Change
/ failed reqs	0	0	✓
/ total time (seconds)	2.657	2.669	⚠️ +0.01
/ avg req/sec	940.79	936.76	⚠️ -4.03
/error-in-render failed reqs	0	0	✓
/error-in-render total time (seconds)	1.439	1.427	-0.01
/error-in-render avg req/sec	1737.53	1751.64	+14.11

Client Bundles (main, webpack, commons)

	vercel/next.js canary	ijjk/next.js add/nft	Change
16.HASH.js gzip	186 B	186 B	✓
677f882d2ed8..HASH.js gzip	14 kB	14 kB	✓
framework.HASH.js gzip	41.9 kB	41.9 kB	✓
main-HASH.js gzip	10.6 kB	10.6 kB	✓
webpack-HASH.js gzip	1.19 kB	1.19 kB	✓
Overall change	67.9 kB	67.9 kB	✓

Legacy Client Bundles (polyfills)

	vercel/next.js canary	ijjk/next.js add/nft	Change
polyfills-a4..dd70.js gzip	31 kB	31 kB	✓
Overall change	31 kB	31 kB	✓

Client Pages

	vercel/next.js canary	ijjk/next.js add/nft	Change
_app-HASH.js gzip	965 B	965 B	✓
_error-HASH.js gzip	3.71 kB	3.71 kB	✓
amp-HASH.js gzip	552 B	552 B	✓
css-HASH.js gzip	333 B	333 B	✓
dynamic-HASH.js gzip	2.83 kB	2.83 kB	✓
head-HASH.js gzip	2.97 kB	2.97 kB	✓
hooks-HASH.js gzip	911 B	911 B	✓
index-HASH.js gzip	231 B	231 B	✓
link-HASH.js gzip	1.64 kB	1.64 kB	✓
routerDirect..HASH.js gzip	298 B	298 B	✓
script-HASH.js gzip	2.95 kB	2.95 kB	✓
withRouter-HASH.js gzip	294 B	294 B	✓
e025d2764813..52f.css gzip	125 B	125 B	✓
Overall change	17.8 kB	17.8 kB	✓

Client Build Manifests

	vercel/next.js canary	ijjk/next.js add/nft	Change
_buildManifest.js gzip	498 B	498 B	✓
Overall change	498 B	498 B	✓

Rendered Page Sizes

	vercel/next.js canary	ijjk/next.js add/nft	Change
index.html gzip	584 B	584 B	✓
link.html gzip	596 B	596 B	✓
withRouter.html gzip	578 B	578 B	✓
Overall change	1.76 kB	1.76 kB	✓

Commit: 2997ee2

pomber · 2021-10-05T12:24:18Z

When I turn nftTrancing on, I'm having this error while SSRing a Next.js page on Vercel: Error: Cannot find module './dist/client/router'

Full error

[GET] /simple-editor/material-palenight
13:59:05:35
2021-10-05T11:59:05.516Z	ea8ad694-cdcb-42b0-8039-525927f4c0ba	ERROR	Error: Cannot find module './dist/client/router'
Require stack:
- /var/task/node_modules/next/router.js
- /var/task/.next/server/pages/[demo]/[theme].js
- /var/task/node_modules/next/dist/server/require.js
- /var/task/node_modules/next/dist/server/load-components.js
- /var/task/node_modules/next/dist/server/api-utils.js
- /var/task/node_modules/next/dist/server/next-server.js
- /var/task/___next_launcher.js
- /var/runtime/UserFunction.js
- /var/runtime/index.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:889:15)
    at Function.Module._load (internal/modules/cjs/loader.js:745:27)
    at Module.require (internal/modules/cjs/loader.js:961:19)
    at require (internal/modules/cjs/helpers.js:92:18)
    at Object.<anonymous> (/var/task/node_modules/next/router.js:1:18)
    at Module._compile (internal/modules/cjs/loader.js:1072:14)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1101:10)
    at Module.load (internal/modules/cjs/loader.js:937:32)
    at Function.Module._load (internal/modules/cjs/loader.js:778:12)
    at Module.require (internal/modules/cjs/loader.js:961:19) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/var/task/node_modules/next/router.js',
    '/var/task/.next/server/pages/[demo]/[theme].js',
    '/var/task/node_modules/next/dist/server/require.js',
    '/var/task/node_modules/next/dist/server/load-components.js',
    '/var/task/node_modules/next/dist/server/api-utils.js',
    '/var/task/node_modules/next/dist/server/next-server.js',
    '/var/task/___next_launcher.js',
    '/var/runtime/UserFunction.js',
    '/var/runtime/index.js'
  ],
  page: '/simple-editor/material-palenight'
}
RequestId: ea8ad694-cdcb-42b0-8039-525927f4c0ba Error: Runtime exited with error: exit status 1
Runtime.ExitError

Here are the links to the Vercel project I'm testing with nftTrancing and without nftTrancing (not sure how to make those links public). I can try to create a minimal open source repro if you need it.

ijjk · 2021-10-05T14:00:42Z

@pomber this has been updated in the latest version of Next.js v11.1.3-canary.43 please update and give it a try! Note: this config has also been renamed to experimental.outputFileTracing in the latest version.

pomber · 2021-10-05T15:26:18Z

hmm, tried with "next": "11.1.3-canary.43" and same error.

config:

module.exports = {
  target: "serverless",
  experimental: {
    esmExternals: true,
    outputFileTracing: true,
  },
};

ijjk · 2021-10-05T18:12:42Z

@pomber I see, after further investigation this does appear to be a bug in the latest version, I opened a PR here to fix this #29649

ijjk · 2021-10-05T20:14:17Z

The above fix has been released in v11.1.3-canary.45 of Next.js, please update and give it a try!

pomber · 2021-10-06T12:01:40Z

The above fix has been released in v11.1.3-canary.45 of Next.js, please update and give it a try!

New error: Cannot find module './utils/resolve-rewrites'

[GET] /section/material-palenight
14:00:11:90
2021-10-06T12:00:12.081Z	933c771f-4622-4cf1-b251-0029a0b07e5c	ERROR	Error: Cannot find module './utils/resolve-rewrites'
Require stack:
- /var/task/node_modules/next/dist/shared/lib/router/router.js
- /var/task/node_modules/next/dist/client/router.js
- /var/task/node_modules/next/router.js
- /var/task/.next/server/pages/[demo]/[theme].js
- /var/task/node_modules/next/dist/server/require.js
- /var/task/node_modules/next/dist/server/load-components.js
- /var/task/node_modules/next/dist/server/api-utils.js
- /var/task/node_modules/next/dist/server/next-server.js
- /var/task/___next_launcher.js
- /var/runtime/UserFunction.js
- /var/runtime/index.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:889:15)
    at Function.Module._load (internal/modules/cjs/loader.js:745:27)
    at Module.require (internal/modules/cjs/loader.js:961:19)
    at require (internal/modules/cjs/helpers.js:92:18)
    at Object.<anonymous> (/var/task/node_modules/next/dist/shared/lib/router/router.js:25:47)
    at Module._compile (internal/modules/cjs/loader.js:1072:14)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1101:10)
    at Module.load (internal/modules/cjs/loader.js:937:32)
    at Function.Module._load (internal/modules/cjs/loader.js:778:12)
    at Module.require (internal/modules/cjs/loader.js:961:19) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/var/task/node_modules/next/dist/shared/lib/router/router.js',
    '/var/task/node_modules/next/dist/client/router.js',
    '/var/task/node_modules/next/router.js',
    '/var/task/.next/server/pages/[demo]/[theme].js',
    '/var/task/node_modules/next/dist/server/require.js',
    '/var/task/node_modules/next/dist/server/load-components.js',
    '/var/task/node_modules/next/dist/server/api-utils.js',
    '/var/task/node_modules/next/dist/server/next-server.js',
    '/var/task/___next_launcher.js',
    '/var/runtime/UserFunction.js',
    '/var/runtime/index.js'
  ],
  page: '/section/material-palenight'
}
RequestId: 933c771f-4622-4cf1-b251-0029a0b07e5c Error: Runtime exited with error: exit status 1
Runtime.ExitError

ijjk · 2021-10-06T15:38:51Z

@pomber Ah that error is from rewrites not being used causing us to not trace that file, I opened an additional fix for that case here #29673. In the mean time you can add a random rewrite e.g. { source: '/no-op', destination: '/no-op' } to fix that. Will update here when the above patch is landed.

This ensures we don't add the no-op resolve alias when rewrites aren't used for the server resolving since it is still needed while tracing server files. ## Bug - [x] Related issues linked using `fixes #number` - [x] Integration tests added - [x] Errors have helpful link attached, see `contributing.md` Fixes: #25538 (comment)

ijjk · 2021-10-06T19:00:22Z

The above is now available in v11.1.3-canary.48 of Next.js

This ensures we don't add the no-op resolve alias when rewrites aren't used for the server resolving since it is still needed while tracing server files. ## Bug - [x] Related issues linked using `fixes #number` - [x] Integration tests added - [x] Errors have helpful link attached, see `contributing.md` Fixes: vercel#25538 (comment)

ijjk added 4 commits May 27, 2021 15:08

Add ncc'd nft

5ddd237

Add initial page tracing plugin

7645c51

Include dynamic chunks and add outputting traces

2c8df87

Merge remote-tracking branch 'upstream/canary' into add/nft

963a210

ijjk added the type: next label May 28, 2021

This comment has been minimized.

Sign in to view

ijjk added 2 commits May 27, 2021 23:27

add check

6d4bd43

check name

5b1ed01

This comment has been minimized.

Sign in to view

ijjk added 2 commits May 27, 2021 23:58

Merge remote-tracking branch 'upstream/canary' into add/nft

11d1c38

remove todo and clean up ignores

61fb021

This comment has been minimized.

Sign in to view

ijjk marked this pull request as ready for review June 4, 2021 17:03

ijjk requested review from divmain, lfades, shuding and timneutkens as code owners June 4, 2021 17:03

ijjk added 4 commits June 16, 2021 14:45

Merge remote-tracking branch 'upstream/canary' into add/nft

03f1235

Merge remote-tracking branch 'upstream/canary' into add/nft

3b5c026

update compiled

9376530

Add initial test for traces

20aafbc

This comment has been minimized.

Sign in to view

Update test for webpack 4

8b69270

This comment has been minimized.

Sign in to view

sokra requested changes Jul 1, 2021

View reviewed changes

ijjk added 2 commits July 6, 2021 23:22

Merge remote-tracking branch 'upstream/canary' into add/nft

7419cd7

apply suggestions from review

3ab3cd9

ijjk requested review from huozhi, padmaia and styfle as code owners July 7, 2021 05:02

styfle previously approved these changes Aug 16, 2021

View reviewed changes

ijjk added 2 commits August 16, 2021 12:45

Merge remote-tracking branch 'upstream/canary' into add/nft

5cfee37

Add excludes to nft ignore

9823923

ijjk dismissed styfle’s stale review via 9823923 August 16, 2021 17:50

ijjk requested a review from styfle August 16, 2021 17:50

styfle approved these changes Aug 16, 2021

View reviewed changes

This comment has been minimized.

Sign in to view

Merge branch 'canary' into add/nft

2997ee2

kodiakhq bot merged commit 24b09ad into vercel:canary Aug 16, 2021

ijjk deleted the add/nft branch August 16, 2021 19:37

ijjk mentioned this pull request Aug 18, 2021

Update generating next-server dependencies #28223

Merged

timneutkens mentioned this pull request Aug 18, 2021

Fix for __dirname #8334

Closed

This was referenced Sep 1, 2021

chore(deps): bump next from 11.1.0 to 11.1.1 filecoin-project/slate#916

Closed

chore(deps): bump next from 11.1.0 to 11.1.2 filecoin-project/slate#917

Closed

ijjk mentioned this pull request Oct 5, 2021

Fix missing trace items and migrate required files test #29649

Merged

3 tasks

ijjk mentioned this pull request Oct 6, 2021

Ensure conditional resolve alias does not affect server #29673

Merged

3 tasks

ofhouse mentioned this pull request Oct 30, 2021

Runtime: Use server target milliHQ/terraform-aws-next-js#86

Open

vercel locked as resolved and limited conversation to collaborators Jan 27, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add entrypoint tracing #25538

Add entrypoint tracing #25538

ijjk commented May 28, 2021 •

edited

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

sokra Jul 1, 2021

ijjk Jul 12, 2021

sokra Jul 1, 2021

ijjk Jul 7, 2021

This comment has been minimized.

ijjk commented Aug 16, 2021

pomber commented Oct 5, 2021 •

edited

ijjk commented Oct 5, 2021

pomber commented Oct 5, 2021

ijjk commented Oct 5, 2021

ijjk commented Oct 5, 2021

pomber commented Oct 6, 2021

ijjk commented Oct 6, 2021

ijjk commented Oct 6, 2021

Add entrypoint tracing #25538

Add entrypoint tracing #25538

Conversation

ijjk commented May 28, 2021 • edited

Bug

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

sokra Jul 1, 2021

Choose a reason for hiding this comment

ijjk Jul 12, 2021

Choose a reason for hiding this comment

sokra Jul 1, 2021

Choose a reason for hiding this comment

ijjk Jul 7, 2021

Choose a reason for hiding this comment

This comment has been minimized.

ijjk commented Aug 16, 2021

Stats from current PR

pomber commented Oct 5, 2021 • edited

ijjk commented Oct 5, 2021

pomber commented Oct 5, 2021

ijjk commented Oct 5, 2021

ijjk commented Oct 5, 2021

pomber commented Oct 6, 2021

ijjk commented Oct 6, 2021

ijjk commented Oct 6, 2021

ijjk commented May 28, 2021 •

edited

pomber commented Oct 5, 2021 •

edited