Option to treat all node modules as external

[brianmhunt]

Unless I'm missing something, when bundling for Node.js it appears as though one has to use --external for every external module imported, even if those are packages that the compiled bundle could otherwise require/import at run-time.

Without using externals our bundled files end up being in the hundreds of megabytes vs 400k when everything is properly externalized.

So in Makefile I'm doing something like this:

--external-files 	:= fs zlib uuid commander source-map-support sanitize-filename date-fns pako
--externals 		:= $(--external-files:%=--external:%)
ESBUILD     :=  ./node_modules/.bin/esbuild --platform=node --bundle $(--externals)

This is leading to problems when we add/remove packages and their references i.e. additional overhead and developer cycles remembering/identifying/fixing this step.

I've toyed with using e.g. jq to read the packages.json or an ESBuild plugin, etc., but it feels like the responsibility for this is probably with the bundler proper.

I think what is desirable for this proposed option would be if ESBuild treated every package in node_modules/ as an external (or perhaps alternatively, everything in package.json).

This is not a blocker, and there's probably something more general here that would work (e.g. an "--external-path" that matches a glob against the file-system path), but in any case I hope it's something easy/fun.

[remorses]

I made a plugin to use the resolve package to resolve packages and add more customization options, to make node modules external you can do this

import NodeResolve from '@esbuild-plugins/node-resolve'
import { build } from 'esbuild'
build({
    plugins: [
        NodeResolve({
            extensions: ['.ts', '.js'],
            onResolved: (resolved) => {
                if (resolved.includes('node_modules')) {
                    return {
                        external: true,
                    }
                }
                return resolved
            },
        }),
    ],
})

You can find other plugins here

[evanw]

I'm intending for plugins to be used to solve custom use cases like this one instead of having this behavior built in. I'm going to close this since the original poster gave a 👍 on the previous post about using a plugin.

FWIW you don't even need to resolve the path to do this assuming all non-node_modules paths are relative or absolute paths, which is the case for node's module resolution algorithm. Something like this might be sufficient:

let makeAllPackagesExternalPlugin = {
  name: 'make-all-packages-external',
  setup(build) {
    let filter = /^[^.\/]|^\.[^.\/]|^\.\.[^\/]/ // Must not start with "/" or "./" or "../"
    build.onResolve({ filter }, args => ({ path: args.path, external: true }))
  },
}

[brianmhunt]

@evanw This is probably a sane delineation for what should be in esbuild.

I added a thumbs-up on the plugins comment because plugins may solve the problem for others and the 👍 would draw attention to it for future readers, but for myself we're trying to stick to the command-line for now (until we have the cycles to dedicate to adding esbuild to our golang server).

In the interim we're using jq for something like this in our Makefile:

--external-imports := $(shell jq '.dependencies|keys[]' package.json)

If a glob were available for esbuild we'd probably use it, but as you can see we've worked around it and there are plugins so unless it's trivial there are probably better problems to dedicate time to.

[meteorlxy]

@evanw Seems that currently we cannot use plugins in synchronous API calls, so the solution above is limited.

[OnurGvnc]

https://github.com/egoist/tsup/blob/master/src/esbuild/external.ts

[rattrayalex]

FWIW, this issue caused me not to use esbuild. When I ran into this in a node context and saw that "The plugin API is new and still experimental" and doesn't seem to be something I can run from the command line, I gave up on using esbuild for compiling my TS Node project – esbuild went from "plug & play" to "takes some work to setup".

Just sharing in case it's helpful feedback – I'm not frustrated and it may make sense for this to not be in core (though I do have a hard time understanding why node_modules aren't handled out of the box). I'm sure I could use esbuild if I really wanted to.

[eric-burel]

Hi guys, I am trying to get a better grasp at building, but basically when I am bundling some TS code for Node (eg React components that should render server side), I'd expect this to be the default. I am still not sure what to do with the answers here.

[jpike88]

I'm intending for plugins to be used to solve custom use cases like this one instead of having this behavior built in.

This isn't some edge case, it's the only sane way to build a node.js project by default. If esbuild supports the 'node' platform, which marks internal node libs as 'external', it goes to reason that node_modules should be a part of that configuration. Having to pick competing plugins just to keep node_modules separate from my bundle is a clear shortcoming of esbuild, and is causing me to consider using something else.

[jpike88]

After much pain I can't go vanilla esbuild, it's just not mature enough for bundling nodejs applications. moving to vite.

[hardfist]

This isn't some edge case, it's the only sane way to build a node.js project by default.

of course not, It's much sane to bundle node_module, which could reduce cold start time and save lots of node_modules space for user

[jpike88]

save lots of node_modules space for user

How is it saving node_modules space, if you need to have a copy of node_modules present in order to build it in the first place? What about when you're deploying to a server environment that's got a different OS, architecture to your own? What about when you don't want to blow up your deployment package to 80mb when it could just be < 2mb, so it's easily to open and inspect directly if needed? Esbuild doesn't even do vendor bundle splitting yet, so at least I can separate all the junk from my actual code.

I have been developing in nodejs for over 8 years, and not once has the desire to slam my entire node_modules dir into our app bundle ever made sense. Esbuild should account for such a common use case, and it doesn't.