Investigate swapping Underscore w/ Lo-Dash or upgrading Underscore

[jdalton]

_86 Upvotes_ It's been mentioned a few times, so I figured I'd track progress/questions with an issue.

_85 Upvotes_

[felixrabe]

Any updates on this?

[richieb]

Any update. I need some of LoDash's methods, including _.cloneDeep()

[glasser]

In Meteor, you probably want EJSON.clone.

Lodash continues to be something I/we are curious about but never seems to
rise to the priority level of, eg, making the Mongo connector scale better.
Maybe next bug week.

[richieb]

Thanks very much, David. I wasn't aware of EJSON.clone.

[jdalton]

Thanks for the interest in lodash. Let me know if you all revisit the issue in the future.

[richieb]

Hi, JDalton,

I am confident the Meteor Devs (also known as MDG, for Meteor Developer Group) will no doubt add LoDash to Meteor. But they started with Underscore as the safe bet at the time, I suppose. On top of that, I think they like to do their due diligence to make sure any package they add to the core Meteor platform is sufficiently tested and reliably integrated.

Therefore, I think they really just need a little push and the the time to include LoDash, but they wouldn't act unless there is enough interest. And based on the activity on this Github issue, there is not enough interest to rise to "sufficient."

LoDash is amazing and incredibly useful, btw. I have been using it with Backbone and Ember. And now that I am using Meteor, I expect to continue to use it. Thanks for your work on LoDash.

[glasser]

Lodash looks great. But just like every time we upgrade a Node minor
version, the day I swap in lodash will be a day where I do a full automated
and manual QA of Meteor. Which is something we're trying to make more and
more automated, but we're not there yet.

Minor underscore version bumps have caused regressions for Meteor. Swapping
out the entire implementation probably will too. (Usually it's a a matter
of a place where Meteor needs an improvement because it was using
Underscore wrong.)

[jdalton]

Minor underscore version bumps have caused regressions for Meteor.

Yap, that's because Underscore, and Backbone as well, don't follow semver.
Lodash follows it though, which is why we've pushed passed v2.0 ;)
We'll keep at it on our end. Ping me for questions/issues 📧

[dalgard]

+1

[reezer]

+1

Doing some benchmarks and profiling most time in our application is spent on underscore (and minimongo). So I am sure performance of many applications could be improved by switching. See the lodash website for in-depth benchmarks.

[sbking]

+1

[ghost]

👍

[yasinuslu]

+1

[dohomi]

+1

[mxab]

👍

[frozeman]

+1

[Lobosque]

+1

On Sun, Mar 16, 2014 at 5:15 PM, Fabian Vogelsteller <
notifications@github.com> wrote:

+1

Reply to this email directly or view it on GitHubhttps://github.com//issues/1009#issuecomment-37768810
.

[Lauricio]

+1

[ghost]

+1

[MauroJr]

+1

[aknuds1]

+1

I find lodash a great replacement for underscore, and use it whenever I can. I'm having problems fitting it into Meteor though, due to that integrating underscore already :(

[bestwestern]

+1

[ByScripts]

Voting for this too. Lodash is better and more reliable.

[darkadept]

I want Lodash too please!

[fungilation]

+1

[timbrandin]

+1

[ghost]

By the way. You can use Lodash instead of Underscore for your own code without problems. Just remove the dependency on standard-app-package and depend on the listed packages instead (select the release version you use). Make sure to remove underscore from your package.js. Then you can use the Lodash atmosphere package with mrt add lodash (or create your own).

I hope that helps some people, because I myself realised it quite late that there is no namespace conflict with _ because of the way that Meteor bundles the packages.

[glasser]

Despite the huge number of people saying vaguely that lodash is "faster" than Underscore over the years on this thread and pointing to various benchmarks, there haven't been too many concrete points about what that means.

Here's something I learned today on an old Meteor project that uses Underscore instead of Lodash: Lodash provides linear-time algorithms for large arrays passed to functions like _.difference, _.uniq, and _.union, where Underscore only provides quadratic algorithms for these functions. Using _.union in this particular project has apparently been leading directly to outages.

Had I understood that back when I was personally one of the active maintainers of the Meteor framework, I think that would have been much more motivating than the hundreds of low-information "+1", "it's better", and "it's faster" comments here. Unfortunately, I don't personally have the time to work on this change now, but I suppose add me to the chorus of upvotes at last :)

[mitar]

Maybe the easiest thing would be to swap underscore with lodash in one release candidate, allow developers to run their projects with it, and report if anything breaks.

[jdnichollsc]

My solution using lodash with Meteor ES2015:

meteor npm i lodash --save

and import lodash /imports/startup/client/index.js

import lodash from 'lodash';
window._ = lodash;

Regards, Nicholls

[vlasky]

@jdnichollsc we are talking about using lodash in the Meteor core packages including those on the server side.

[awatson1978]

I once tried the underscore to lodash refactor, just to get a sense of the scope of the problem. It's a gnarly refactor. The API for underscore is large, and used extensively throughout the Meteor codebase. As I recall, there were thousands of broken reference the first pass through; which were actually fairly straight forward to clean up; but once the references were cleaned up, there were hundreds of broken calls; related to dozens of mangled files. It was layers and layers of refactoring.

The best way to do the refactor, in my opinion, is to divide it into parts, and be slow and consistent. @jdnichollsc is probably right. The best bet is to bring both lodash and underscore into the codebase, and rewrite one package at a time. Separating the packages out into independent NPM packages is probably a good stepping-stone. If a package can be installed independently via NPM, it's probably decoupled enough that it's underscore references can be cleanly replaced with lodash.

[stubailo]

I wonder if TypeScript or Flow could be helpful in using a computer to identify places where the API is different.

[awatson1978]

API differences are documented in the Migrating page.

Underscore _.compose is Lodash _.flowRight  
Underscore _.contains is Lodash _.includes  
Underscore _.each doesn’t allow exiting by returning false  
Underscore _.findWhere is Lodash _.find  
Underscore _.flatten is deep by default while Lodash is shallow  
Underscore _.indexBy is Lodash _.keyBy  
Underscore _.invoke is Lodash _.invokeMap  
Underscore _.mapObject is Lodash _.mapValues  
Underscore _.pairs is Lodash _.toPairs  
Underscore _.pluck is Lodash _.map  
Underscore _.uniq by an iteratee is Lodash _.uniqBy  
Underscore _.where is Lodash _.filter  
Underscore _.isFinite doesn’t align with Number.isFinite  (e.g. _.isFinite('1') returns true in Underscore but false in Lodash)  
Underscore _.matches shorthand doesn’t support deep comparisons (e.g. _.filter(objects, { 'a': { 'b': 'c' } }))  
Underscore ≥ 1.7 & Lodash _.template syntax is _.template(string, option)(data)
Lodash _.memoize caches are Map like objects  
Lodash supports implicit chaining, lazy chaining, & shortcut fusion  
Lodash split its overloaded _.head, _.last, _.rest, & _.initial out into _.take, _.takeRight, _.drop, & _.dropRight  (i.e. _.head(array, 2) in Underscore is _.take(array, 2) in Lodash)

As I recall, this was the second or third step in the process; and where I hit a wall (in part, because I didn't have the above documentation).

The first step was to replace the Underscore references with Lodash. That breaks the builder, because it can't load up all the files. Once all the references are updated, then Meteor will start going through the build pipeline. Unfortunately, the build pipeline involves things like _.contains calls and _.each calls which exit by returning false. So then it starts complaining about the API not working. When you fix some of the API calls, it will get through one of the steps in the pipeline, but then complains about subsequent steps.

I'm handwaving a bit, since it's been at least a year since I tried that refactor, and I don't recall all the details. But it's a metric ton of errors.

[stubailo]

I wonder what the impact on user apps would be of a change like this. I suppose as long as we provide a shim to use underscore for parts of the app during a transition period it could be fine, but if it's this hard for the Meteor code base it could be even harder for apps.

[mitar]

BTW, this is the most upvoted issue for Meteor.

[abernix]

Just as a reminder: You can and have been able to use lodash in your own projects for a very long time now (since Meteor 1.3) and if you'd like to use lodash today, do not let anything stop you! While underscore is used extensively by Meteor internals, you can still meteor npm install --save lodash and import lodash from 'lodash' in any project and enjoy its _.glory (not a Lodash method, surprisingly!) I realize that's not the point of this issue but it's worth clearing up (again) the confusion which has and does exist!

I'm here to instill hope that this has not been forgotten and is in the works! Getting a bit more detailed: For those interested in how Underscore weighs in within the Meteor code, I'll point you to the Underscore tally counts I generated in researching this project a couple weeks ago:

• Totals method counts (inclusive of below)
  • Method counts in core packages
  • Method counts in meteor-tool

As has been stated throughout this issue, especially by those who have made the attempt to undertake this project (very much appreciated, to those who have tried!), there are a number of obstacles: very old underscore, Meteor-modified, different names, incompatible API, etc. Each underscore method has considerations which must be made and the maze of "change log" entries between "Meteor-mod-Underscore 1.5.2" and lodash@4.17.x is extensive. A word of caution to anyone just switching a dependency like this and seeing what happens: prepare yourself for a lot of bizarre edge cases. There is plenty of documentation out there which advises against exactly this.

While one option for this was to openly and willingly accept PRs for this transition and given the vast scope of Underscore in Meteor and the large potential for manual error in making the (sometimes complicated) changes, or even reviewing the changes, I'm currently pressing forward with an automated process to avoid both the time-sink the high margin-of-error.

I am (currently) in the process of writing AST transformations which will programmatically (using recast and jscodeshift, amongst other tools) go through and remove Meteor's dependency on underscore and replace it with lodash or native ECMAScript methods, on a case by case basis. (It's worth pointing out that @wmertens made a wise suggestion about using AST transformations earlier in this thread).

I'm aware of existing codemods which attempt to make this transformation automatically, but in trying those, I've certainly ran into many bumps, some of which are unique to the Meteor core (though rest assure, I'm taking solace and inspiration from their existence). We also have considerations of making sure we play nicely with those who want to keep using underscore and ensuring efficient bundling (a current Roadmap goal).

I'm hope that some of the tooling I'm building will be helpful to others too (even if this migration might be less common now than in the past) and I hope to share more info soon!

[pozylon]

In my opinion, while swapping underscore with lodash is a waste of time, swapping underscore with native functions is a plus.

https://github.com/you-dont-need/You-Dont-Need-Lodash-Underscore

In propably less than 2 years somebody will open an issue with the title: "Investigate swapping lodash with xyz"?

[jdalton]

@pozylon You might check out this post. Lodash is a collection of ~300 modules, considerably more than the handful of ES5/6/7 methods, and offers features beyond those of their built-in counterparts. Lodash works great combined with ES6 and there's even a lodash-es package. You might also dig that Lodash v5 is being written in ES6/7 too.

Unlike Underscore, which has been dormant for ~2yrs, Lodash has evolved with the language and ecosystem. This is also why Lodash continues to gain users with over 1,000 new package dependents each month for the last 2 years, more than 30,000 packages directly depending on it, and more than 110,000 packages (30% of npm) impacted by it in some way.

[sebakerckhof]

@abernix How will it be packaged? As a meteor package, an npm dependency in the package.js or just as peer dependency so users can use their own version (within limits)?

It'd be interesting to have separate counts for files that are server-only and files that are included in the client bundle. I think one of the major concerns is bundle size. Although this could be alleviated by only requiring the necessary lodash modules, at least until there is support for tree-shaking.

[pozylon]

@jdalton I truly appreciate what you have done for the whole js community and I use lodash myself.

Still I think getting rid of dependencies is preferrable over replacing dependencies if +/- same amount of time has to be invested. Lodash is more popular than ever and exxtremely well maintained but underscore too had some fans before the 2y rapunzel sleep...

[jdalton]

@pozylon

Still I think getting rid of dependencies is preferrable over replacing dependencies if +/- same amount of time has to be invested.

Lodash is modular, so folks can use as little or as much as they'd like. It makes sense to use something that's battle tested rather than reinventing the wheel of fundamentals.

Lodash is more popular than ever and exxtremely well maintained but underscore too had some fans before the 2y rapunzel sleep...

Judging by how long this issue has existed maybe Lodash's future is the least of the worries here. I understand your concern, but I think it's a bit premature/unwarranted (has the slight ring of NIH).

Update:
The work @abernix has put in is a pretty rad solution.

[mitar]

@abernix, are you thinking about something like what @pozylon posted above? So swapping underscore with ES6, and then having Babel translate that back to backwards compatible things if needed?

[abernix]

@sebakerckhof That's a great question! I've gone through a couple iterations/preliminary attempts at figuring the best solution for this. While a peer-NPM dependency implementation might be one way to avoid duplication and bundle bloat, the current implementation of Meteor's NPM peer dependency system wouldn't trickle up as far as the Meteor server bundle itself – a place where a lot of current _ references live and a reason why underscore is still included in Meteor itself right now.

Currently, I'm creating a lodash Atmosphere package, however the usage recommendation will be different for lodash than it was for underscore.

The current underscore loads the entire Underscore library into your application, regardless of which actual methods you use. The lodash package will only bundle and load what you use, will a little help from the developer.

Internally, the lodash package will contain files for each Lodash method, provided in one of two automated ways (at lodash publish-time): either by using lodash-cli to generate modules directly, or possibly just by offering providing a stub which re-exports lodash/moduleName from the NPM itself via Npm.depends – I haven't decided yet. Either way, it will not explicitly add them to the package using api.addFiles (which would eagerly load them).

The responsibility of the developers of Meteor apps (and packages) will then be to import or require the lodash package and specific method (or core) in a similar fashion as they do with other Atmosphere package exports by using the meteor/ prefix. They can either use core (for the entire core build) or ideally, Lodash's cherry-picked methods:

import methodName from 'meteor/lodash/methodName';

or

import _ from 'meteor/lodash/core';

In a similar way as to how underscore is used now, this will allow a single lodash library to exist in the bundle and be used by the Meteor core packages, developer packages and developer apps but allow the import scanner to only bundle lodash methods which are actually used by your app. This means that if you don't use _.deburr (maybe you should, it's cool!) then Meteor won't bundle deburr.

Also, since Meteor 1.5 will support dynamic imports, you could benefit further in some cases if you loaded more beefy lodash methods like debounce or template using dynamic imports:

import("meteor/lodash/debounce").then(debounce => {
  // Some debounce-y fun, if that's even possible.
  debounce(fun, 2000);
});

In a future reality, where Meteor moves more toward full NPM integration, a peer NPM dependency setup will be the natural course for this (as with the rest of Meteor packages). This might seem like a counter-measure to that right now, but I can assure you that this transition does not need to be any more complicated than it is already (nor wait and take any longer). Additionally, in the future, developers will only have to remove the meteor/ prefix to move to a true NPM peer dependency setup, but keeping in mind any incompatible lodash version changes, of course.

@pozylon Somewhere out there, some day, maybe a version of ECMAScript exists that might maybe natively provide all of the features that lodash offers, but I can assure you that it's not any time soon. Removing lodash entirely would be detrimental to the Meteor framework and cause us to have to duplicate a substantial amount of functionality that it provides. I do think that embracing newer ECMAScript standards is a direction that we will proceed in, but Lodash does offer benefits, like chaining which still come in very handy. That said, in cases where it makes sense I am selectively replacing things with native functions. I'll point out that the list of "You Don't Need Lodash Underscore" methods is quite short compared to the list of Lodash functions.

[sebakerckhof]

Yep, seems like a clean way forward.

[jdalton]

@abernix

Internally, the lodash package will contain files for each Lodash method, provided in one of two automated ways (at lodash publish-time): either by using lodash-cli to generate modules directly, or possibly just by offering providing a stub which re-exports lodash/moduleName

You'll have more tree-shaking and optimization opportunities going thelodash/moduleName route than lodash-cli. The cli will not make the jump to v5. It's being retired in favor of modern bundlers.

[abernix]

@jdalton

I did move in the direction of using lodash/moduleName stubs rather than generating actual lodash source modules with lodash-cli however I'm currently generating them by utilizing the includes export from lodash-cli's listings.js as a "dictionary" of lodash methods. Since this will be deprecated in the v5-future, do you have a best suggestion to use as a reference for auto-building our lodash module stubs? Looks like things are a bit under-construction at the moment, so happy to revisit that when it's finalized if you don't have an answer yet!

We won't need this stubbed-pattern long-term, but I'd like to easily generate new versions of the Meteor lodash package (ideally with fp modules as well!) until we can make that full switch.

[jdalton]

@abernix Using the mapping of lodash-cli is fine for v4. If the transition takes a while and v5 is out, there will be other options like using a babel-plugin. You can base it off of babel-plugin-lodash.

[hwillson]

To help provide a more clear separation between feature requests and bugs, and to help clean up the feature request backlog, Meteor feature requests are now being managed under the https://github.com/meteor/meteor-feature-requests repository.

Migrated to meteor/meteor-feature-requests#48.

[abernix]

Just as an update on the status of this (important) task before the updates move over to the feature request repository...

I did complete the work to replace underscore with lodash in Meteor. The only hesitation to applying that officially was that it actually increased the resulting client-side bundle size by slightly more than we would have liked. This was particularly true since the realization was had during the 1.5-push which was directly focused on decreasing client bundle sizes.

I'm aware of the reasons and I will develop a plan soon to resolve that and get this released. This will be even easier with the bundle-visualizer in Meteor 1.5.