feat: add require.context support

[EvanBacon]

Continued work for adding require.context to Metro, which started in #821. The overall design is discussed in microsoft/rnx-kit#1515. The user-facing API is closely modelled on Webpack's require.context.

This feature is experimental and unsupported. We will document and announce this for general consumption once it's stable.

How it works

When we encounter a dependency that has contextParams (see #821), we will now generate a corresponding virtual module in the dependency graph, with dependencies on every file in the file map that matches the context params.

┌─────────┐  require.context('./ctx', ...)   ┌−−−−−−−−−−−−−−−−−−┐     ┌──────────┐
│ /bundle │ ───────────────────────────────▶ ╎ <virtual module> ╎ ──▶ │ /ctx/bar │
└─────────┘                                  └−−−−−−−−−−−−−−−−−−┘     └──────────┘
                                               │
                                               │
                                               ▼
                                             ┌──────────────────┐
                                             │     /ctx/foo     │
                                             └──────────────────┘

Crucially, we keep this context module up-to-date as file change events come in, so that HMR / Fast Refresh continues to work reliably and transparently. This accounts for the bulk of the complexity of the implementation and tests.

Main pieces of the implementation

• [collectDependencies] Done in feat: add support for capturing require.context in dependency collector #821: Extract require.context calls as dependencies with added metadata. (Behind a flag)
• [metro-file-map] HasteFS: API for querying the file map for the set of files that match a particular resolved context.
• [DeltaBundler] DeltaCalculator: Logic to mark a previously-generated context module as dirty when the set of files it matches has changed.
• [DeltaBundler] graphOperations: Logic to "resolve" context params to a virtual module stored in the graph object, and forward the resolved params to the transformer to generate the module's body and dependencies.
• [DeltaBundler] graphOperations: API for querying the graph for the set of currently active contexts that match a particular file. (Used by DeltaCalculator)
• [metro] transformHelpers, contextModuleTemplates: Logic to generate the contents of a virtual context module by querying the file map. Includes various templates to implement the different require.context modes.
• [metro-runtime] require(): A stub for require.context that helps us throw a meaningful error if the feature is not enabled.

Tests:

• require-context-test: a new integration test suite that builds and executes various bundles that use require.context. In particular this covers the subset of the require.context runtime API that we support.
• DeltaCalculator-context-test: a new test suite covering the changes to DeltaCalculator that are specific to require.context support.
• traverseDependencies-test: expanded and refactored from its previous state. Covers the changes to graphOperations.

Future work

At the moment, every time we invalidate a context module, we scan the entire file map to find the up-to-date matches and then regenerate the module from scratch (including passing it through the transformer).

Two open areas of investigation are:

1. Optimising the initial scan over the file map - e.g. representing it as a path tree to drastically limit the number of files we need to match against.
2. Optimising the incremental case - e.g. directly using the file addition/deletion events we receive from the file map to update the generated module in-place.

At least (2) is essential before we treat this feature as stable.

There's also room to generalise the "virtual modules" concept/infrastructure here to support other use cases. For now everything is very much coupled to the require.context use case.

@EvanBacon's original PR summary follows.

---

Summary

• Continued work for adding require.context to Metro feat: add support for capturing require.context in dependency collector #821
• This PR adds the ability to match files given a "require context". This is similar to the existing method for matching against a regex, but this enables users to search upwards in a directory, search shallow, and match a regex filter.
• Adds ability to pass a Buffer to transformFile as a sort of virtual file mechanism. This accounts for most the changes in packages/metro/src/DeltaBundler/Transformer.js, packages/metro/src/DeltaBundler/Worker.flow.js, packages/metro/src/DeltaBundler/WorkerFarm.js.
• Since we collapse require.context to require I've also added a convenience function in dev mode which warns users if they attempt to access require.context without enabling the feature flag.
• Made DeltaCalculator aware of files being added.
• graphOperations has two notable changes:
  1. When resolving dependencies with context, we attach a query to the absolute path (which is used for indexing), this query has a hex hash of the context -- used hex instead of b64 for filename safety.
  2. We pass the require context to processModule and inside we transform the dependency different based on the existence of a context module.
• When collecting the delta in _getChangedDependencies we now iterate over added and deleted files to see if they match any context modules. This is only enabled when the feature flag is on.
• In packages/metro/src/lib/contextModule.js we handle a number of common tasks related to context handling.
• In packages/metro/src/lib/contextModuleTemplates.js we generate the virtual modules based on the context. There are a number of different modules that can be created based on the ContextMode. I attempted to keep the functionality here similar to Webpack so users could interop bundlers. The most notable missing feature is context.resolve which returns the string path to a module, this doesn't appear to be supported in Metro. This mechansim is used for ensuring a module must be explicitly loaded by the user. Instead I wrapped the require values in getters to achieve the same effect.
• We implement the lazy mode as well but this requires the user to have async bundles already setup, otherwise the module will throw a runtime error.

Notice

I've withheld certain optimizations in order to keep this PR simple but still functional. We will want to follow up with a delta file check to require context so we aren't iterating over every file on every update. This feature can be seen in my test branch.

In my test branch, I built the feature on top of #835 so I know it works, should only require minor changes to graphOperations to get them in sync.

Test plan

• Unit tests -- pending @motiz88 approving the implementation.

Local testing

metro.config.js

const { getDefaultConfig } = require("expo/metro-config");

const config = getDefaultConfig(__dirname);
const path = require("path");

config.watchFolders = [
  path.resolve(__dirname),
  path.resolve(__dirname, "../path/to/metro"),
];

config.transformer = {
  // `require.context` support
  unstable_allowRequireContext: true,
};

module.exports = config;

index.js

console.log(require.context("./").keys());

Start a dev server, when changes to the file system occur, they should be reflected as automatic updates to the context. This does lead to the issue of having multiple contexts triggering multiple sequential reloads, I don't think this is a blocking issue though.

Also tested with async loading enabled, and the context: require.context("./", undefined, undefined, 'lazy').

Behavior

Notable features brought over to ensure this require.context functions as close to the original implementation as possible:

• require.context does not respect platform extensions, it will return every file matched inside of its context.
• require.context can match itself.
• A custom 'empty' module will be generated when no files are matched, this improves the user experience.
• All methods in require.context are named to improve the debugging.
• We always match against a ./ prefix. This prefix is also returned in the keys.
• Modules must be loaded by invoking the context as a function, e.g. context('./module.js')

[motiz88]

I would fold this into the upcoming PR which uses this to implement require.context, to make sure any edge cases are ironed out & tested - I don't want to have to ship breaking changes to DependencyGraph at that point or integration bugs to slip through. Overall looks great though.

[motiz88]

@EvanBacon this seems broadly in line with what we've discussed. Here's a round of comments. I would love to take another look once there are tests for the new functionality and CI is passing.

[motiz88]

My guess is require, but the deciding factor is what Webpack does.

[motiz88]

Incomplete pass of feedback, will make more time next week.

[motiz88]

Mutating contextParams here is a code smell - let's treat the input to this function as read-only.

Also, is dep.name guaranteed to be a relative path? Can you add a comment to this effect?

[EvanBacon]

In the default case we resolve dep.name as a path component, this assumption builds on how the existing functionality works.

[motiz88]

I've come to believe we should merge the existing DeltaCalculator + traverseDependencies test suites. It can be done fairly mechanically by rewriting the traverseDependencies tests we have in terms of a DeltaCalculator instance, but it's a chance to also rethink the way we mock out the transformer/filesystem and hopefully simplify things some more. Anyway, I won't block this PR on that.

[facebook-github-bot]

@motiz88 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

@motiz88 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[robhogan]

(Imported PRs need internal review by someone other than the importer, sharing feedback here for transparency)

Left a few nits and minor comments around path handling that need explanation if not changes, but substantively this looks good to me. Thanks for working on this - must be one of the biggest feature contributions we've ever had!

[robhogan]

Comment:

filePath.startsWith('..') assumes normalisation, which seems to be safe in practice but I can't see it explicitly mentioned in the Node JS docs for relative (as it is for join, for example).

I wondered about the necessity of the isAbsolute check too, given we've obtained this path using relative(), but I guess that's for Windows, eg for two files on different drives?

[EvanBacon]

isAbsolute check appears to be extraneous now.

[EvanBacon]

I'm not sure normalization is required for the the .. check to work as this paradigm is cross-platform.

[robhogan]

I'm not sure normalization is required for the the .. check to work as this paradigm is cross-platform.

If it's not normalised it might not appear at the start of the path - eg foo/../../bar is valid but normalises to ../bar

[robhogan]

isAbsolute check appears to be extraneous now.

I think it's still necessary for Windows tbh, looks like we get absolute paths back when there's no valid relative path between locations:

node -e "console.log(path.win32.relative('C:/foo', 'D:/bar'))";
D:\bar

[facebook-github-bot]

@motiz88 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[kraenhansen]

I've noticed this PR says

This feature is experimental and unsupported. We will document and announce this for general consumption once it's stable.

Are there plans to stabilise this? I've opened facebook/react-native#41421 in case it's already there.

[robhogan]

The main thing holding us back was that the pre-symlink filesystem implementation didn't support enumerating a subtree, so require.context was an O(project) transform. Now we have subtree scans, and Expo/@EvanBacon have battle-tested the implementation, I don't see any reason not to promote this to stable (and always-on).

[Harukisatoh]

I've been experimenting with this feature recently, and it's fantastic! However, I believe I've come across a potential issue. To be honest, I'm not entirely sure if it's an actual problem or if I'm simply using it incorrectly.

My goal is to dynamically import images. In my application, I'm displaying a list of cards, and within each card, there's an area where I'm supposed to render a pre-defined image. I have a collection of images stored locally, and for each card, I intend to render one of these images.

The require.context method functions well when I only have images for a single screen density. The problem arises when I introduce images for multiple screen densities in my 'images' folder, leading the bundler to display an 'Unable to resolve module' error.

Do you think I might be misunderstanding something?

[karlhorky]

For anyone looking for a copy+paste metroRequire.d.ts for TypeScript types for Metro context.require() to add to your own project until facebook/react-native#41421 is merged, @EvanBacon has one here:

• https://github.com/EvanBacon/Metro-Context-Modules-Demo/blob/3ad84a12b40c51060ee50d0b0065d02fd8ce3b43/types/metroRequire.d.ts