upgrading from 4.0.2 to 4.0.7

[akoskm]

Upgrading from 4.0.2 to 4.0.7 introduces several devDependencies that weren't required in 4.0.2:

 "@babel/plugin-proposal-class-properties": "7.5.0",
 "@babel/plugin-syntax-dynamic-import": "7.2.0",
 "@babel/plugin-transform-runtime": "7.5.0",
 "babel-loader": "8.0.6",
 "babel-plugin-dynamic-import-node": "2.3.0",
 "file-loader": "4.0.0",
 "webpack-cli": "3.3.6",
 "webpack-dev-server": "3.7.2"

I followed https://github.com/rails/webpacker#upgrading and https://github.com/rails/webpacker/blob/master/CHANGELOG.md to upgrade. I also copied the contents of https://github.com/rails/webpacker/blob/master/lib/install/config/babel.config.js over my local version of the same file.

I'm just curious if I'm doing something wrong that makes these new dependencies appear or this is ok.

Some of these dependencies like @babel/plugin-transform-runtime, @babel/plugin-proposal-class-properties were used in babel.config.js at 4.0.2 as well but webpacker worked just fine without include these packages in package.json.

[akoskm]

Any idea why these new dependencies got added between these minor versions?

[jakeNiemiec]

Only webpack-dev-server should be in your devDeps. Can you directly link which document said to put (eg) file-loader into devDeps?

Can you additionally paste your entire package.json?

[akoskm]

Hi @jakeNiemiec, thanks for responding on this! After following the instructions here https://github.com/rails/webpacker#upgrading file-loader appeared with the other new devDependencies I listed. I didn't add anything to package.json manually, this is the diff after the update in package.json:

[jakeNiemiec]

The problem is that your devDeps are invalid to begin with. Can you paste your entire package.json?

[jakeNiemiec]

If you don't want to make it public, you can tag me in an unlisted https://gist.github.com and I can comment there.

[akoskm]

@jakeNiemiec I think I just did that. Mentioned you in the comment section of a secret gist.

[jakeNiemiec]

I'm still not sure why all those new dependencies appeared.

This is because your 4.0.2 package.json was invalid. See: https://github.com/rails/webpacker/blob/master/docs/yarn.md#add-an-npm-module-to-devdependencies

Did moving them to "dependencies" work?

[akoskm]

Did moving them to "dependencies" work?
There might be a misunderstanding. My application works after upgrading to webpacker 4.0.7. here's gem info from the project's folder:

$ gem info webpacker

*** LOCAL GEMS ***

webpacker (4.0.7, 4.0.2)
    Authors: David Heinemeier Hansson, Gaurav Tiwari
    Homepage: https://github.com/rails/webpacker
    License: MIT
    Installed at (4.0.7): /Users/akoskm/.rbenv/versions/2.6.1/lib/ruby/gems/2.6.0
                 (4.0.2): /Users/akoskm/.rbenv/versions/2.6.1/lib/ruby/gems/2.6.0

    Use webpack to manage app-like JavaScript modules in Rails

It's just the new packages that I don't know why are added.
Looking at yarn.lock some of these packages, like file-loader, "@babel/plugin-proposal-class-properties": "7.5.0", "@babel/plugin-syntax-dynamic-import": "7.2.0", "@babel/plugin-transform-runtime": "7.5.0", appear to be a dependency of @rails/webpacker@4.0.7.

[akoskm]

I'll start this upgrade process again but first I'll move the packages that should be in devDependencies there. Then proceed with these instructions: https://github.com/rails/webpacker#upgrading.

[akoskm]

@jakeNiemiec I just run the commands for upgrade and packages.json ended up with a substantially smaller diff than earlier, I don't know what caused this behavior.

I used the following to upgrade to 4.0.7:

bundle update webpacker
rails webpacker:binstubs
yarn upgrade @rails/webpacker@4.0.7
yarn upgrade webpack-dev-server --latest

after that my packages.json is:

$ git diff package.json
diff --git a/package.json b/package.json
index e8cd552d..9f364621 100644
--- a/package.json
+++ b/package.json
@@ -13,7 +13,7 @@
     "@fortawesome/free-regular-svg-icons": "5.9.0",
     "@fortawesome/free-solid-svg-icons": "5.9.0",
     "@fortawesome/react-fontawesome": "0.1.4",
-    "@rails/webpacker": "4.0.2",
+    "@rails/webpacker": "4.0.7",
     "actioncable": "5.2.3",
     "babel-plugin-transform-react-remove-prop-types": "0.4.24",
     "bootstrap": "4.3.1",
@@ -62,7 +62,7 @@
     "sinon": "7.3.2",
     "sinon-chai": "3.3.0",
     "style-loader": "0.23.1",
-    "webpack-dev-server": "3.3.1"
+    "webpack-dev-server": "3.8.0"
   },
   "babel": {
     "presets": [

We can close this issue because I can't reproduce the behavior I initially reported.

[akoskm]

@jakeNiemiec Okay, I think I realized what was happening back in July when I first tried to do this.
So the upgrade commands produced this small diff in package.json back then as well.
I started running into problems when i tried to start the dev server:

$ ./bin/webpack-dev-server
The CLI moved into a separate package: webpack-cli
Please install 'webpack-cli' in addition to webpack itself to use the CLI
-> When using npm: npm i -D webpack-cli
-> When using yarn: yarn add -D webpack-cli
internal/modules/cjs/loader.js:638
    throw err;
    ^

Error: Cannot find module 'webpack-cli/bin/config-yargs'

so I installed webpack-cli (altough with the -D options, I didn't about this https://github.com/rails/webpacker/blob/master/docs/yarn.md#add-an-npm-module-to-devdependencies). Now I installed it without -D (I'm always using -E though).

After the dev server started up I was getting errors such:

ERROR in Entry module not found: Error: Can't resolve 'babel-loader' in '/Users/akoskm/Projects/example/webapp'

after installing babel-loader as well and giving a shot to ./bin/webpack-dev-server again I got:

ERROR in ./app/javascript/packs/components/sidebar/index.jsx
Module build failed (from ./node_modules/babel-loader/lib/index.js):
Error: Cannot find module '@babel/plugin-syntax-dynamic-import'
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15)
    at Function.Module._load (internal/modules/cjs/loader.js:562:25)
    at Module.require (internal/modules/cjs/loader.js:692:17)
    at require (internal/modules/cjs/helpers.js:25:18)
    at module.exports (/Users/akoskm/Projects/example/webapp/babel.config.js:47:7)

and I could go on but eventually when I fix all these errors I'll end up with the extra dependencies that I listed in my initial message.

Which returns me to my original questions, why my project works with an older webpacker version (4.0.2) with less dependencies and why it doesn't when I upgrade to a newer webpacker?

[jakeNiemiec]

Do you have a babel.config.js OR a .babelrc file in your project root?

[jakeNiemiec]

Please make the appropriate changes listed in the changelog: https://github.com/rails/webpacker/blob/master/CHANGELOG.md#407---2019-06-03

and

[akoskm]

Done.

@babel/polyfill isn't used in my project so no changes there.

Regarding the fixes for 4.0.7 I did the following:

• copy/pasted the contents of the default babel.config.js from 4.0.7 tag
• returned @babel/preset-react and babel-plugin-transform-react-remove-prop-types plugins that are added when you turn on react support.

When starting the dev server I'm receiving the same errors:

$ ./bin/webpack-dev-server
The CLI moved into a separate package: webpack-cli
Please install 'webpack-cli' in addition to webpack itself to use the CLI
-> When using npm: npm i -D webpack-cli
-> When using yarn: yarn add -D webpack-cli
internal/modules/cjs/loader.js:638
    throw err;
    ^

Error: Cannot find module 'webpack-cli/bin/config-yargs'

EDIT
and everything works the same after: if I add webpack-cli then it complains about babel-loader, then @babel/plugin-syntax-dynamic-import.

[jakeNiemiec]

How are you adding webpack-cli? Just yarn add webpack-cli?

[akoskm]

Yes, except I add an exact version: yarn add -E webpack-cli .

[jakeNiemiec]

Try this:

• delete your yarn.lock file.
• delete node_modules
• yarn install

I see evidence that this may fix things for you. -E might have played a role in this, I would not use it in addition to your lock file.

[akoskm]

Unfortunately this makes no difference. After doing those steps running ./bin/webpack-dev-server still needs webpack-cli. After adding webpack-cli (without the -E) I need babel-loader, etc.

[akoskm]

Inspired by your comment that might the -E that this problem I updated every dependency in my package.json to ^version. Removed yarn.lock & node_modules and run yarn install.

yarn asked me to specify which version of babel-loader (8.0.6) and webpack-cli (3.3.8) I want to install.

I picked the latest stable for both. Running ./bin/webpack-dev-server doesn't require me to install anything and my app runs fine.

Thank you for your suggestions!

[seanders]

I just wanted to follow up on this thread. My application is upgrading from webpacker 4.0.0 to 4.0.7. The only consequential change that I can identify with respect to babel-loader is this:
https://github.com/rails/webpacker/compare/v4.0.0..v4.0.7#diff-b9cfc7f2cdf78a7f4b91a753d10865a2R24

Prior to this upgrade, this is where our babel-loader was location:
node_modules/babel-loader

After the upgrade to 4.0.7, I noticed that the location of the babel-loader seemed to move to this location:
node_modules/@rails/webpacker/node_modules/babel-loader

This change in location required us to explicitly add babel-loader as a dependency in our package.json (in addition to file-loader).

I'm guessing this change might have something to do with yarn's resolution algorithm.

Any thoughts on what explicitly might cause this change?

[seanders]

Seems like this is what I'm referring to: #1537

[jakeNiemiec]

Any thoughts on what explicitly might cause this change?

Please make note of the breaking changes.

I noticed that the location of the babel-loader seemed to move to this location:

It should be accessible from both of those locations. This #2191 (comment) may help.

[seanders]

I think that removing of the lock file and re-installing all the dependencies, incidentally moved the babel-loader package back to node_modules/babel-loader, which solved @akoskm 's issue.

According to this yarn contributor, babel-loader getting installed into the root node_modules(by virtue of being a dependency of @rails/webpacker) is a side-effect of yarn trying to minimize the size of the node_modules directory:

Relying on this is entirely undefined behavior, do not do it. Always list your dependencies explicitely. Note that this isn't unique to Yarn, the same is true for npm. And pnpm will break your application entirely.

See yarnpkg/yarn#5749 (comment)

If I'm understanding this correctly, the optimal fix is to explicitly list babel-loader as a direct dependency of any project using @rails/webpacker.

[jakeNiemiec]

@seanders You are entirely correct in your reasoning. I don't think that webpacker should be implicit about dependencies. Here is the list of those:

webpacker/package.json

Lines 14 to 53 in d393cc4

	"dependencies": {
	"@babel/core": "^7.4.5",
	"@babel/plugin-proposal-class-properties": "^7.4.4",
	"@babel/plugin-proposal-object-rest-spread": "^7.4.4",
	"@babel/plugin-syntax-dynamic-import": "^7.2.0",
	"@babel/plugin-transform-destructuring": "^7.4.4",
	"@babel/plugin-transform-regenerator": "^7.4.5",
	"@babel/plugin-transform-runtime": "^7.4.4",
	"@babel/preset-env": "^7.4.5",
	"@babel/runtime": "^7.4.5",
	"babel-loader": "^8.0.6",
	"babel-plugin-dynamic-import-node": "^2.2.0",
	"babel-plugin-macros": "^2.5.0",
	"case-sensitive-paths-webpack-plugin": "^2.2.0",
	"compression-webpack-plugin": "^2.0.0",
	"core-js": "^3.1.3",
	"css-loader": "^2.1.1",
	"file-loader": "^3.0.1",
	"flatted": "^2.0.0",
	"glob": "^7.1.4",
	"js-yaml": "^3.13.1",
	"mini-css-extract-plugin": "^0.7.0",
	"node-sass": "^4.12.0",
	"optimize-css-assets-webpack-plugin": "^5.0.1",
	"path-complete-extname": "^1.0.0",
	"pnp-webpack-plugin": "^1.4.3",
	"postcss-flexbugs-fixes": "^4.1.0",
	"postcss-import": "^12.0.1",
	"postcss-loader": "^3.0.0",
	"postcss-preset-env": "^6.6.0",
	"postcss-safe-parser": "^4.0.1",
	"regenerator-runtime": "^0.13.2",
	"sass-loader": "^7.1.0",
	"style-loader": "^0.23.1",
	"terser-webpack-plugin": "^1.3.0",
	"webpack": "^4.32.2",
	"webpack-assets-manifest": "^3.1.1",
	"webpack-cli": "^3.3.2",
	"webpack-sources": "^1.3.0"
	},

@swrobel Is correct in that issue. The only problem is that I personally come from the webpack/node.js side of things, I am simply not aware of a good way to fix this on the rails side of things. Do we just add all those dependencies onto run "yarn add @rails/webpacker@next"? Should we create a rake task that updates your package.json?

[seanders]

Sounds like a rake take might be the right approach. Not sure of a better way to keep those dependencies ('@rails/webpacker', 'babel-loader', 'file-loader') in lockstep.

[jrochkind]

Sounds like a rake take might be the right approach. Not sure of a better way to keep those dependencies ('@rails/webpacker', 'babel-loader', 'file-loader') in lockstep.

Hmm. What about:

1. Mark them as "peer dependencies" in webpacker itself.
2. Have the existing webpacker:install rake task include them to the host app's package.json. (This rake task already exists, but doesn't add these).

I am not sure a rake task is needed to update them in an existing package.json. If you update your webpacker to something that needs different versions than the originally-generated ones in your local package.json, the peer dependencies in the (new version of) webpacker should give you a very comprehensible error message telling you what the issue is, leading you to go fix your local package.json.

Apologies if I'm missing the point or talking about something irrelevant; I come from ruby side, and am trying to figure out what the heck is going on.

[elliotcm]

If it helps anyone at all, this is the package.json I ended up with after resolving all of these issues, albeit with some extra React stuff in there:

{
  "name": "<app name>",
  "private": true,
  "dependencies": {
    "@babel/core": "^7.0.0-0",
    "@babel/plugin-proposal-class-properties": "^7.7.4",
    "@babel/plugin-syntax-dynamic-import": "^7.7.4",
    "@babel/plugin-transform-destructuring": "^7.7.4",
    "@babel/plugin-transform-runtime": "^7.7.6",
    "@babel/preset-env": "^7.7.7",
    "@babel/preset-react": "^7.7.4",
    "@rails/webpacker": "^4.2.0",
    "babel-loader": "^8.0.6",
    "prop-types": "^15.6.0",
    "react-dom": "^16.12.0",
    "react": "^16.12.0",
    "yarn": "^1.21.0"
  },
  "devDependencies": {
    "webpack": "^4.0.0",
    "webpack-cli": "^3.3.10",
    "webpack-dev-server": "^3.9.0"
  }
}

[akoskm]

@jakeNiemiec this seems to be a recurring problem when trying to upgrade to the latest version. Yesterday I wanted to update form 4.0.7 to 4.2.2, following the instructions here https://github.com/rails/webpacker#upgrading.

I experienced the same issue as I did during my previous upgrade. Ruby was complaining about babel-loader not being present, then about "@babel/plugin-proposal-class-properties", and basically all the package in babel.config.js. In the end, my package.json would probably look like the one @elliotcm posted.

This is not the way to go, because when you make a fresh install of webpacker you don't need any of those packages, right?

So I figured that removing yarn.lock and generating a new one resulted in a substantially different yarn.lock, compared to what I ended up with after following the upgrade instructions.

With the fresh yarn.lock I wasn't required to add all those modules to package.json that ruby was complaining about earlier.

I think you should update the upgrade instructions and mention that as last step we should remove yarn.lock and let yarn install generate a new one.

[jrochkind]

Removing yarn.lock might work as a workaround, but it's a problem if that's recommended/required, it should not be a required step to upgrade webpacker.

Compare to removing your Gemfile.lock. You are removing all record of what version of dependencies in your tree were working, and rebuilding them all fresh. Comparable to doing a bundle update. While sometimes it makes sense to do this, imagine if you were forced to do a complete bundle update every time you wanted to update, oh, sprockets, or bundler, or even rails. It would lead to much increased work and frustration.

If it's necessary for you to remove yarn.lock, that's a problem to be fixed, not something to be documented/recommended. There may also be another solution. But yarn/npm management confuses me; it's definitely not as smooth as bundler.

[akoskm]

Removing yarn.lock might work as a workaround, but it's a problem if that's recommended/required, it should not be a required step to upgrade webpacker.

@jrochkind you're right, this sounds more like a workaround than something that should be done as a step of the upgrade process.

Compare to removing your Gemfile.lock

I don't work with ruby at all so can't relate to this. Removing the yarn.lock and doing yarn install was trivial and it fixed all my problems.

If it's necessary for you to remove yarn.lock, that's a problem to be fixed, not something to be documented/recommended.

I agree. After running into this problem the second time, I'm pretty sure that this is one possible solution to these problems.

[jakeNiemiec]

This is not the way to go, because when you make a fresh install of webpacker you don't need any of those packages, right?

@akoskm You should have those in your package.json. As I said, Webpacker is relying on undocumented behavior (having access to dependencies of dependencies). More discussion on this over here: #2371 (comment)

The best solution is to just add those dependencies yourself with yarn add myPackage. This will save you from future breakage from the yarn/npm side as well.

[akoskm]

@jakeNiemiec sorry, I missed that comment and didn't realize that this is and undocumented behavior.

I work with other projects (babel/yarn/webpack, no webpacker) and you have to list "@babel/plugin-proposal-class-properties" in your dev dependencies, if you want to require it in babel.config.js. The way how dependencies are handled here is different from other JS projects, so thanks for clarifying this.

[jakeNiemiec]

@akoskm I'll start this upgrade process again but first I'll move the packages that should be in devDependencies there.

...you have to list "@babel/plugin-proposal-class-properties" in your dev dependencies, if you want to require it in babel.config.js. The way how dependencies are handled here is different from other JS projects...

This is a common misconception, even in other JS projects. Babel needs to be in your dependencies since a production environment needs it to compile your packs. From the docs:

webpacker/docs/yarn.md

Lines 21 to 23 in ed6c121

	Be careful not to add any build or app related JS modules in this fashion. Adding JS modules to `devDependencies` [will block them from being installed in **any** production environment](https://yarnpkg.com/lang/en/docs/cli/install/#toc-yarn-install-production-true-false).
	Docs from JS modules may instruct you to use `--dev` or `devDependencies`, but this is generally under the assumption that you are using a `node.js` workflow.

Related issues #1880 (comment)
Longer discussion on this topic: #1212 (comment)

[akoskm]

This is a common misconception, even in other JS projects.

Sorry for the confusion, I was comparing webpacker to JS projects.

Babel needs to be in your dependencies since a production environment needs it to compile your packs.

Yup, I understand this. With webpacker, babel must be listed under dependencies, in other projects I list babel under devDependencies.