How to reliably override npm dependencies

[samselikoff]

tl;dr – Use yarn resolutions to ignore the requirements of transitive dependencies.

There's an issue that's been following me around my open-source work lately, and it's causing maintenance burdens for me and bugs for my users.

The problem is this:

How can a user of my library reliably override packages that my library itself depends on?

Let's look at an example of how this can come about.

---

Say we're the maintainer of awesome-datepicker, a JavaScript date picker library. During our development of this library, we installed moment as a dependency and used it throughout our code. Suppose that when we did this, the latest version of moment was 1.6.0.

Our library's package.json would have been updated with our new dependency:

  "dependencies": {
+  "moment": "^1.6.0"
  }

So far, so good.

6 months later, Sally the JavaScript developer comes along, finds our library and adds it to her project. Because moment is in the dependencies key of awesome-datepicker's package.json file, Sally will get moment in her own project's dependency graph when she installs awesome-datepicker (using either npm or yarn).

Here's the twist: the latest version of moment is now 2.16.0.

So, what version of moment does Sally end up with?

After she runs yarn install, she can use yarn list to find out:

yarn list --pattern moment
└─ moment@1.7.2

So, yarn (or npm) has bumped the version from the original 1.6.0 to 1.7.2. This is because 1.7.2 is semver-compatible with the version specified in awesome-datepicker's package.json file. However, the latest version of moment, 2.16.0, is not.

So, yarn's doing its best to give Sally any patch updates and new features she might want to take advantage of, while not breaking awesome-datepicker's usage of moment.

---

Fast forward another month. Sally is still working with awesome-datepicker, which takes advantage of Moment's locales. She realizes she needs access to Moment's new Dutch locale, which was added in Moment v2.16.0.

So, she adds the latest version of moment to her project:

yarn add -D moment

Her project's package.json is updated with the new dependency:

  "devDependencies": {
+  "moment": "^2.16.0"
  }

and she gets back to her app.

To her surprise, she tries using the new Dutch locale within her awesome-datepicker code, but alas – it does not work. It seems like the locale is not available, even though she just installed the latest version of Moment. What gives?

She goes back to the terminal to check:

yarn list --pattern moment
├─ awesome-datepicker@1.0.0
│    └── moment@1.7.2
└─ moment@2.16.0

So – it looks like she's ended up with two versions of moment in her project. Even though her app did install 2.16.0, it's not getting used. awesome-datepicker is still installing 1.7.2 and going about its merry way.

Sally had no idea this happened. Sally is sad. 😭

How duplications occur

To summarize what happened, moment moved from being an implementation detail of our library to a user-level concern of Sally's application.

When moment was used strictly inside of awesome-datepicker, Sally (or any other user) didn't really care about which version of moment was being used. From her perspective, she was only concerned with the public APIs of awesome-datepicker.

But once she started caring about Moment's APIs, that's where the mess began.

Because moment was depended upon by both awesome-datepicker as well as her application, the possibility for version conflicts arose.

Speaking more generally, we could refer to moment as both a top-level dependency as well as a transitive dependency of Sally's application. (A transitive dependency is really just a fancy way of saying a dependency of a dependency.)

The presence of non-semver-overlapping versions of top-level dependencies and transitive dependencies is what gives rise to this problem.

Getting unstuck

So, what can Sally do to solve this?

First, she could open a PR to awesome-datepicker to bump its version of moment, and make sure everything works. This is a viable path forward, but maintainers can be slow to respond or libraries can be abandoned. So, she could get stuck going down this path.

Another approach is that she could use yarn's resolutions feature to guarantee that only one version of moment ends up in her dependency graph. She does this by adding a new resolutions key to her package.json, and specifying the version of moment she wants to "win" as yarn installs her app's packages:

resolutions: {
  "moment": "2.16.0"
}

After doing this, when she goes to yarn install, she may see a warning:

Resolution field "moment@2.16.0" is incompatible with requested version "moment@^1.7.2"

This is yarn's way of telling her she might run into a problem. But, if it turns out that awesome-datepicker's usage of Moment is fully functional on Moment version 2, she's in the clear.

How can she know? Unfortunately, the only way is to run her app & tests, and see if anything breaks.

However, using yarn resolutions does provide her with a mechanism to get herself unstuck, so she can use the new locale, and be sure she has only one copy of moment in her application bundle.

Early detection

In this story, Sally was explicitly trying to use a new API, and it was clear she had an issue with her bundle. This led her to investigate immediately, and find out she had a problematic dependency graph.

However, these sorts of problems can often occur much more surreptitiously.

Going forward, how can Sally be proactive and avoid unwanted duplicates of dependencies in her bundle?

In the case of Ember Addons, we haveEmber CLI Dependency Lint, which can notify Sally at build-time (and with a failing test) if any of her addons are duplicated. This is especially helpful, since the typical Ember Addon ships code to the client, where duplicate packages are more likely to cause issues than in node.

For the rest of her node modules, tools like yarn list and yarn install --flat are her best defense.

Unfortunately at this point in the story, there are no bulletproof solutions in the tooling layer that will warn Sally whenever a potential problem crops up.

Being a good citizen as a maintainer

What can we as maintainers of awesome-datepicker do to help future travelers who might find themselves in Sally's shoes?

First, we can be be liberal in the versions we specify for our library's dependencies.

If awesome-datepicker was tested against Moment versions 1 and 2, and we were confident it worked with both, we could update our package.json file to reflect this:

"dependencies": {
  "moment": "^1 || ^2"
}

This would help tools like yarn do their best to deduplicate dependencies if they end up installing packages for a project similar to Sally's.

What about peer dependencies? In theory, moment has become a peerDependency of awesome-datepicker, and if tooling support were there, this feature of package.json could be a path forward. However, npm changed its behavior with respect to peer dependencies in v3, and the wider tooling ecosystem also handles them inconsistently. At best, developers will see a warning at install time that some peer dependency has not been met.

It's also worth noting that peerDependencies are not currently used in the large by most major projects. For example, every Ember Addon technically has a peer dependency on ember-source, though none of them specify it.

Given this, the best path today is to keep moment a dependency of awesome-datepicker, even if conceptually it is a peer dependency. If users "become interested" in a specific version of a transitive dependency for their own app, they can use the resolutions feature of yarn to guarantee a single version ends up in their bundle, as described above.

What does this have to do with Mirage?

So – why did I write all this up on Mirage's repo?

Because this issue has come up many times in our little project's history, both for Pretender.js and Faker.js.

Faker has been something of an odd ball out, since it's not really a dependency of Mirage at all, rather a convenience that was packaged alongside it. That made sense several years ago, before Ember Auto Import made it easy for Ember apps to import packages directly from npm. Today, it doesn't, so we'll be dropping Faker from being bundled with Mirage soon (with instructions on how to easily install it).

Pretender, however, falls into the same category that moment did in Sally's story above – namely, it is an implementation detail for Mirage users, until it's not.

When a user of Mirage needs a feature in a newer version of Pretender than what Mirage currently depends on, they get stuck. Historically this has meant opening an issue requesting a version bump for Pretender. With yarn resolutions, they can at least get unstuck on their own terms, again assuming that Mirage works with the newer version of Pretender that they care about.

Maintainers can also help the ecosystem run into fewer issues like this by keeping their dependencies up-to-date as often as possible, with the help of tools like Dependabot, which automatically open PRs for outdated dependencies. Once a project updates its versions, if it can still test against older versions (and specify its support for these using the || or syntax in their package.json files), it will lessen the chance that other developers find themselves with duplicate dependencies and unexpected behavior.

Additional resources

There's way more to this topic - we've only scratched the surface.

Here's some more resources from the last few weeks if you want to dive deeper:

• Lenient libraries, strict applications – The EmberMap Podcast, Ep. #51
• Ember Auto Import Issue #184
• Ember CLI Dependency Checker Issue #96

And please chime in if I've missed anything!