Allow devs to supply a dispatcher to the Node implementation of fetch

[steveluscher]

Summary

The current version of web3.js allows folks to supply what's called an ‘HTTP Agent.’ This is a class that mediates how network requests should be fetched. One example is to establish a connection pool for a given URL, and keep connections open for a time so that TLS/SSL negotiation doesn't have to be done on every request.

The new web3.js up until this point has punted on how to reintroduce this concept. In this PR we bring it back in the form of allowing callers to create transports with custom Undici Dispatchers.

It only works in Node. If you supply the dispatcher_NODE_ONLY property in non-Node environments you'll get a console warning that your config has been ignored.

Alternate designs considered

I desperately wanted to avoid the situation in this PR, which was to create a config property on transports that's only usable in Node.

Alternate approaches considered:

1. Create a special transport for use in Node. Callers would have to createDefaultNodeRpcTransport() and/or solanaRpcFactoryForNode() which I thought was a bridge too far.
2. Encourage people to inject a global dispatcher. This is a terrible idea for all of the reasons that global state is terrible:
   • Applies to all connections in the process
   • Your global dispatcher can be unset by someone else
   • There might already be a global dispatcher installed (eg. a ProxyAgent) and you might unknowingly replace it
3. Export a Dispatcher setter that lets you install a dispatcher accessible only to @solana/fetch-impl. Besides having most of the bad features of #​2, this would not work. We inline @solana/fetch-impl in the build, meaning there's no longer anywhere to export such a method from.

Test Plan

cd packages/fetch-impl/
pnpm test:unit:node
cd ../rpc-transport-http/
pnpm test:unit:node

See benchmark tests in next PR.

Closes #2126.

[steveluscher]

Current dependencies on/for this PR:

• Add README for @solana/rpc-transport-http #2199
• Create benchmarks for tuning the Node transport #2179
• Allow devs to supply a dispatcher to the Node implementation of fetch #2178 👈
• Ignore any test files ending with -test.node.ts or -test.browser.ts in other environments #2177
• Remove node-fetch external from build script #2176
• master

This stack of pull requests is managed by Graphite.

[socket-security]

👍 Dependency issues cleared. Learn more about Socket for GitHub ↗︎

This PR previously contained dependency changes with security issues that have been resolved, removed, or ignored.

View full report↗︎

[steveluscher]

I made the decision to deliver a default dispatcher to everyone. This is what the current web3.js does. I really hate this, because everyone's workload is different and requires tuning, but it's also true that having no dispatching strategy has pretty awful performance.

[lorisleiva]

Is there no way to bring this Dispatcher API to the browser?

[steveluscher]

There is not. The browser doesn't let you tune its networking stack.

[socket-security]

New and removed dependencies detected. Learn more about Socket for GitHub ↗︎

Package	New capabilities	Transitives	Size	Publisher
npm/node-abort-controller@3.0.1	None	0	14.3 kB	southpolesteve
npm/node-fetch@2.7.0	network	+3	493 kB	node-fetch-bot
npm/tslib@2.6.2	None	0	84 kB	typescript-bot
npm/typescript@5.2.2	None	0	40.6 MB	typescript-bot

🚮 Removed packages: npm/bn.js@5.2.1, npm/borsh@0.7.0, npm/browserslist-to-esbuild@1.2.0, npm/bs58@4.0.1, npm/chai-as-promised@7.1.1, npm/chai@4.3.7, npm/chalk@5.3.0, npm/commander@11.1.0, npm/commitlint@18.6.1, npm/cross-env@7.0.3, npm/eslint-config-prettier@9.1.0, npm/eslint-config-turbo@1.10.13, npm/eslint-plugin-import@2.29.0, npm/eslint-plugin-jest@27.4.2, npm/eslint-plugin-mocha@10.1.0, npm/eslint-plugin-prettier@5.0.1, npm/eslint-plugin-sort-keys-fix@1.1.2, npm/eslint@8.45.0, npm/eslint@8.51.0, npm/esm@3.2.25, npm/fast-stable-stringify@1.0.0, npm/fastestsmallesttextencoderdecoder@1.0.22, npm/graphql@16.8.1, npm/jayson@4.1.0, npm/jest-dev-server@10.0.0, npm/jest-environment-jsdom@29.7.0, npm/jest-runner-eslint@2.1.2, npm/jest-runner-prettier@1.0.0, npm/jest-watch-master@1.0.0, npm/jest-watch-select-projects@2.0.0, npm/jest-watch-typeahead@2.2.2, npm/jest-websocket-mock@2.5.0, npm/jest@29.7.0

View full report↗︎

[steveluscher]

I'm starting to regret this in light of my latest benchmarks in #2179.

[lorisleiva]

If we're not creating a more performant Agent by providing these opinionated default values then that's good right? At least we don't need to be opinionated here.

[steveluscher]

Yeah, I think I might tear this default Agent out until we have evidence that undici with no dispatcher is insufficient. The getLatestBlockhash benchmarks would indicate that it does a fine job of scheduling requests without one.

[lorisleiva]

If we're not creating a more performant Agent by providing these opinionated default values then that's good right? At least we don't need to be opinionated here.

[steveluscher]

Merge activity

• Feb 27, 10:58 AM PST: @steveluscher started a stack merge that includes this pull request via Graphite.
• Feb 27, 10:58 AM PST: @steveluscher merged this pull request with Graphite.

[github-actions]

🎉 This PR is included in version 1.90.1 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀

[github-actions]

Because there has been no activity on this PR for 14 days since it was merged, it has been automatically locked. Please open a new issue if it requires a follow up.