Webpacker alternatives - which path should we go to?

[andreslucena]

As its README says, Webpacker is dead:

Webpacker has been retired 🌅

Webpacker has served the Rails community for over five years as a bridge to compiled and bundled JavaScript. This bridge is no longer needed for most people in most situations following the release of Rails 7. We now have three great default answers to JavaScript in 2021+, and thus we will no longer be evolving Webpacker in an official Rails capacity.

For applications currently using Webpacker, the first recommendation is to switch to jsbundling-rails with Webpack (or another bundler). You can follow the switching guide, if you choose this option.

Secondly, you may want to try making the jump all the way to import maps. That's the default setup for new Rails 7 applications, but depending on your JavaScript use, it may be a substantial jump.

Finally, you can continue to use Webpacker as-is. We will continue to address security issues on the Ruby side of the gem according to the normal maintenance schedule of Rails. But we will not be updating the gem to include newer versions of the JavaScript libraries. This pertains to the v5 edition of this gem that was included by default with previous versions of Rails.

The development of v6 will not result in an official gem released by the Rails team nor see any future support. But Justin Gordon is continuing that line of development – including a focus on hot-module reloading features etc – under a new gem called Shakapacker that is based on the unreleased v6 work from this repository.

Thank you to everyone who has contributed to Webpacker over the last five-plus years!

Please refer to the 5-x-stable branch for 5.x documentation.

As it's a gem that we're using, and as a matter of fact we're in a release candidate version, we should make a decision about what's the best course of action, with its pros and cons.

I'll open a new thread inside this discussion to talk pros and cons about every one of these options before we make a decision.

[andreslucena]

1. jsbundling-rails with webpack or other bundler like esbuild or rollup

[ferblape]

To me the key is understand between jsbundling and importmaps which fits better with the nature of Rails engines and the extensions of Decidim.

Rails engines were abandoned by webpacker, the support was not very good (not to the level you would expect from a framework like Rails). I suspect that propshaft + jsbundling-rails has improved that support but I haven't played too much with it yet.

[alecslupu]

@ferblape , Importmaps-rails seems to have more and better support for engines ... From their documentation :

https://github.com/rails/importmap-rails#sweeping-the-cache-in-development-and-test
https://github.com/rails/importmap-rails#composing-import-maps

I have build this:

module MyEngine
  class Engine < ::Rails::Engine
    # ...
    initializer "my-engine.importmap", before: "importmap" do |app|
      app.config.importmap.paths << Engine.root.join("config/importmap.rb")
      app.config.importmap.cache_sweepers << Engine.root.join("app/assets/javascripts")
    end
  end
end

[ahukkanen]

I've looked into this gem as well as cssbundling-rails and basically they both do two things:

1. Adds the selected NPM package to the package.json (e.g. esbuild for JS and e.g. sass for CSS)
2. Adds the build script to the package.json which is run through node

Neither of these support providing additional lookup/import paths, so we would need to customize both of these.

I'm all for using e.g. esbuild to build the JS and e.g. the sass (or whatever) to build CSS but I don't believe these gems provide much added benefit to the build pipeline.

Otherwise I like this approach (using NPM packages and wrapping our own build scripts) but these gems do not seem to provide much that we need in Decidim. Particularly I like the fact that the JS and CSS building are separated to their own contexts / build scripts

At the end we would need to customize both build processes to add the Decidim lookup/import paths for the tools and that would basically mean creating our own build script. Which would make both of these gems unnecessary (as we already have similar functionality in Decidim today).

The benefit these would provide is that the implementor could choose the build tool they like the most but that benefit does not exist when the build scripts have to be customized for each tool separately.

[andreslucena]

2. importmap-rails

[andreslucena]

We have a starting point by @alecslupu in #8745

[ferblape]

I've found out that importmap is not supported by some "basic" browsers such as FF or Safari, nor most of the mobile browsers: https://caniuse.com/?search=importmap

Should this fact discard the option of importmaps-rails in the short term?

[ahukkanen]

@ferblape There's the shim/polyfill (es-module-shims) that makes them work in Firefox and Safari too.

But as I've said in other discussions, I don't support jumping into the "bleeding edge" bandwagon as the early adapters. Generally it just causes problems over problems (and wasted effort/time to constantly fix those issues) until the technologies are well established and supported everywhere.

I feel that for the time being, we should stick with well established conventions that have worked for the past 10 years or so (i.e. serving the single bundles). The browser implementations can still change and I feel using polyfills is like going back to the IE polyfill/Modernizr era.

Let's just focus on building Decidim, not fixing the tools all the time?

[andreslucena]

3. Webpacker as is implemented now

[andreslucena]

👎🏽 we're at a RC version (6.0.0-rc.5) and this will not be maintained anymore

[alecslupu]

I am against this, as it creates a lot of pain already ( while was maintained )

[microstudi]

Still suffering it with modules. Webpacak go away!

[ferblape]

Agree, it's being a bit painful with decidim extensions, and the v6 RC won't progress anymore

[justin808]

This is just not appropriate given the handoff to Shakapacker by DHH.

[andreslucena]

4. Shakapacker

[justin808]

@alecslupu and @microstudi,

So far, I sense the frustration was with the old version, which had many aspects that I didn't like. Or maybe there was some misconfiguration. The default "compile" mode to ensure assets were built was crazy slow. Most developers figured out that running webpack in watch mode or running the webpack-dev-server was much better. I can easily see how some developers missed this.

In terms of the ease with the old sprockets, having all globals is a nightmare for bigger apps.

At the core of webpacker is a configuration YAML file used by both the webpack compilation and the Rails view helpers to allow static assets to be built with webpack, which is a far better-supported system than sprockets.

If you're serious about modern frontends, I don't see how you can omit utilizing the JavaScript ecosystem.

Anyway, it seems that some think it's better to add a lot of code to your project rather than leveraging the open-source ecosystem.

Previously, with webpacker under the Rails core, hundreds of open PRs and issues were not getting addressed.

Now, we're moving as fast as the community contributions come in!

So I think you should give the NEW shakapacker a chance. It's not the old webpacker that you hated!

[ahukkanen]

The main issue we've had with Decidim since the Webpacker migration has been CSS compilation performance. It's not really an issue with Webpacker or Shakapacker (with their own problems, even with version 6 which we are already using), but an issue with the underlying Webpack itself.

Here's few test scripts I put together to help measure the difference between the CSS compilation speed:
https://gist.github.com/ahukkanen/33e5858ae56d216f6ec3986bdec72711

No Webpacker or Shakapacker there. Just to compare Webpack and without Webpack.

To use these:

1. Generate the Decidim development_app as per instructions
2. Go into the development_app folder and put both of these gist files to the bin folder of the app
3. Chmod those files to 755 to make them executable
4. Run both files in the console and see the execution time printed out into the console

Note that you need to run these in the Decidim development_app for the requirements to be met. Also note that they are not feature complete implementations, they just serve to prove a point.

My short testing shows that with the current amount of CSS, the compilation speed is about 5-6x faster when compiling directly with Dart Sass compared to passing it through Webpack. It can be also a misconfiguration with Webpack (maybe Webpack is using the non-compiled version of the sass library?) or maybe Webpack is doing a lot more than we really need in Decidim. I'd rather have fast compilation times than all those features Webpack provides.

It's a real pain developing front-end for Decidim when the CSS compilation takes over 20s+ in general, every time you make a small change in the CSS. Dart Sass alone compiles it in 3-4s which is still much more than the JS compilation speed but it's bearable. This could be probably further improved by applying all the current recommendations to the SCSS files, such as using @use instead of @import but that's a bigger change.

I feel that even if we could improve the Webpack performance, we wouldn't really need it for anything. My gut feeling is that most of that overhead is coming from Webpack itself and if not, feel free to prove me wrong.

In my opinion, we could live with all the workarounds we needed to add to Webpacker (or Shakapacker, doesn't really matter in this context, we probably still need those), and we ourselves haven't even missed the foo.js.erb or foo.scss.erb files that @microstudi mentioned above (they were hack'ish solutions to start with). Those other issues could probably be solved with Webpack mentioned by @microstudi but is it worth the effort to spend time configuring Webpack (which I also feel is a pain) or just provide the build tools that we really need in Decidim? We can find other ways around those issues.

At our end the compilation speed has been the real issue why I also think Webpacker has got bit of a bad rep around here. Even when it's not really Webpacker's fault, it's its underlying engine - Webpack.

Based on this comparison alone, I vote against both Webpacker and Shakapacker. It's not supposed to be any kind of attack against these great libraries, as much as it may seem like in a faceless internet discussion. It's just the amount of problems they bring into the Decidim ecosystem without adding much benefit.

Getting rid of Webpacker or Shakapacker does not mean we would need to get rid of the NPM ecosystem. The NPM ecosystem is still available one way or another with all the alternatives proposed here, even with Sprockets.

I still haven't experimented with cssbundling-rails but I am pretty sure it would provide compilation speeds close to running Dart Sass manually with Node. Or if not, we can build our own build pipeline, it's not that difficult these days as shown above.

I feel most of the hate presented here is directed towards Webpack (like JS failing to build due to configuration issues). Not Webpacker or Shakapacker.

[justin808]

@ahukkanen are you 💯 sure that you used the latest versions of webpack, Node, and other libraries. If you had older libraries or a misconfigured setup (like the old webpacker "compile" backstop), then surely you'll have many performance issues.

What doesn't make sense for this project is the need to reinvent the wheel of webpacker just to use "jsbundling-rails" which is pretty darn minimal. Just check out the commits and source. I contributed to the comparison https://github.com/rails/jsbundling-rails/blob/main/docs/comparison_with_webpacker.md

Given how much code was added to this project to build on top of jsubundling-rails, that decision really puzzles me. We have a much better community now that the rails core folks are gone from webpacker! So we'd rather work with you all to solve the common problems for everybody.

[ahukkanen]

@justin808 No, Decidim is not using the latest version of Webpack but it doesn't seem to improve the compilation speed with the provided examples. You can try it out following the above instructions and running npm un @decidim/webpacker && npm i webpack. The CSS compilation speed does not improve with that.

I don't feel going away from Webpack would be re-inveting the wheel. We really don't need most features in Webpack. Only thing we need is to transpile JS and CSS. Whether in one bundle or as separate files, undecided at this point. For the time being my personal opinion would lean towards the bundled files but there may be valid reasons to decide otherwise. It's still up for discussion (see the discussion below).

[ahukkanen]

I experimented a bit with sass-embedded and it cuts the CSS compilation time into about half from the experiment above when not using Webpack. So the whole Decidim SCSS compiles in less than 2 second when using sass-embedded (it's a drop-in replacement for sass, so you can just switch it in the example test-sass script posted above).

Yet again, if we wanted to use that with Webpack, it's not possible right now:
webpack-contrib/sass-loader#774
webpack-contrib/sass-loader#1031

I also checked and confirmed that under the hood, Webpack's sass-loader was already using the same implementation, so the comparison above is valid:
https://github.com/webpack-contrib/sass-loader/blob/bbb3cf27b940a8c6d82f0795e1977da89dd942cc/src/utils.js#L13

[andreslucena]

5. Sprockets

[arnaudlevy]

I loved it when everything just worked, with Sprockets. Since the javascript mess started in Rails (webpacker, importmap and so on), it stopped working simply and we lose so much time discussing and debugging stupid things.
Nostalgia is not an argument, but maybe we should stick to something really simple?

[alecslupu]

Based on the investigation done for implementing #8745, I would conclude the best approach for the task at hand would be

• Sprockets-Rails as asset server
• Import Maps for JS handling (I am still trying to address several downsides regarding page performance in testing, but is great for external libraries management)
• Sassc-rails for CSS handling ( I have not yet looked into it, but seems more faster that old sass)

Imho, the importmaps makes more sense when discussing SPA, but we can benefit from it as it would cache individual js files and the user impact after a deployment would be minimal ...

[takahashim]

How about dartsass-rails instead of sassc-rails?
https://github.com/rails/dartsass-rails

sassc-rails uses libsass, but libsass is deprecated.
https://sass-lang.com/blog/libsass-is-deprecated

[ahukkanen]

The downside with development/testing server performance may take some time to get fixed as Puma does not seem to support HTTP/2 yet:
puma/puma#2697

On production machines it probably won't be a problem as you can offload the HTTP/2 requests to nginx (or similar) and let the proxy server or load balancer handle the static asset serving.

I also read from some of the related puma discussions that not all load balancers support HTTP/2 yet but after I did a quick search about that myself, it seems that at least the major players, such as Amazon and Microsoft clouds, support HTTP/2 on their load balancer services. Same applies for CloudFlare. And also the most common open source load balancers, such as haproxy, seem to support HTTP/2. So I'm not sure if this claim is true or not.

That said, I am still a bit concerned about forcing HTTP/2 everywhere as it may cause compatibility issues here and there at least until the support is more widespread, both in the server and client side.

[alecslupu]

Hello!
We are not forcing the HTTP2 everywhere. As far i can tell, our current set-up would be HTTP1 compatible, but we are prepping the platform to be HTTP2 compatible.
I know for a fact that AWS is supporting HTTP2, as we have it enabled.

Even though puma does not support HTTP2 for development mode, it is still way faster webpacker precompile, which in Dockerized environment, when changing branches a lot, may cause issues like webpacker does not observe that assets needs to be recompiled, spending hours on figuring out what is wrong with the app. Other situation, you have the assets precompiled locally, and you don't see a bug happening only on prod.

We already have a docker compose file that we can modify to add nginx with HTTP2 protocol.

[ahukkanen]

Yep, as discussed before HTTP/1 can still serve as a fallback on default setups (with a performance hit), but I mean overall everywhere. If we use importmaps + HTTP/2 preference configuration, we also assume that all infrastructure out there is HTTP/2 compatible. I don't speak about Decidim itself but all the infrastructure built around it in different organizations.

Again, just pointing these things out. For most default configurations, it will be fine.

[ahukkanen]

As shakapacker was selected for the time being to move development forward without causing too much friction for different stakeholders, I think it would be a good time to write down a list of requirements what we have for the pipeline and what we have learned over these several migrations, in order to make the future migration easier:

• Fast SCSS compilation time (Dart Sass), at least as long as we are using SCSS (might change over time due to the switch to Tailwind)
• Ability to override any JS and CSS file shipped with the Decidim core gems from external modules
• Ability to control the priority order of the include paths from external modules, e.g. if decidim-y wants to override a JS/CSS file from decidim-x, it should be able to set its asset include priority higher
• Ability to inject configurations and styles within the existing core SCSS files from external modules (might change over time due to the switch to Tailwind but even with plan CSS, at least the ability to override any CSS variables from other modules)
• Ability to write the Tailwind CSS configuration file dynamically every time the assets compilation is started, in order to include all Decidim modules in its content configuration array (referring to the files where Tailwind should be looking for the Tailwind utilities)
• Easy developer workflow for the development app, preferably completely transparent for developers (good old Sprockets)
• Easy developer workflow for the test app, i.e. not having to run any extra commands to start the tests
• Easy developer workflow to ship their assets with their external modules (i.e. not having to release both, an NPM package + Ruby gem to ship the module)

Feel free to add to this list any requirements. I tried to keep the technology preferences (e.g. HTTP version, importmaps, etc.) out of this list, just to keep it focused on the features that developers need. There are some other discussions above about those technical preferences that need to be taken into account but those are up for future discussion.

[ahukkanen]

As we are experiencing very bad performance with the CSS building process especially with v0.28, I have been testing the build process using ESBuild instead of Webpack (with different configurations) and I wanted to share some conclusions and finding from this effort:

• The initial build time with ESBuild takes about 25% of the time consumed by Webpack (on my computer down to 5s from 20s with Webpack)
• There is only a small difference in the CSS rebuild times between ESBuild and Webpack (in favor of ESBuild, on my computer about 0.5s or 10-15% of the rebuild time).
  • This indicates that the initialization process of Webpack is unnecessarily heavy. The problem is not necessarily with Webpack itself but maybe how it is configured?
• ESBuild supports all the features we currently use from Webpack but it definitely requires some custom configuration for Decidim (which I have implemented).
• Tailwind (aka Tailbloat) is the major reason for the slowness of the CSS build but even without Tailbloat, Webpack startup time is slow (significantly faster than with Tailbloat but still slow).
• By commenting out the Tailbloat plugin from the PostCSS configuration, the build time is about 30% of the time compared to having that plugin (on my computer initial build time is down to 1.5s using ESBuild, i.e. abour 30% of the time with Tailbloat).
  • Same effect both with ESBuild and Webpack.
• Tailwind's @apply rules are extremely slow
  • Removing all the @apply rules prior to running the PostCSS can provide about 35% faster build time compared to having them.
  • There are some issues and discussions reported about the @apply rules being slow (see e.g. Slow compile when using @apply tailwindlabs/tailwindcss#443)
  • I have tried to implement some of the potential fixes mentioned in those issues, but they have zero to little impact on the compile time.
  • Having these rules not only causes technical burden but they also cause mental burden for newcomers. Why are we trying to abstract an already abstract concept, i.e. CSS, with less readable hyped gadget?
• Having SCSS vs no SCSS, there is very little difference because the used sass-embedded package is already optimized and fast. Doesn't matter if plugged into ESBuild or Webpack, the effect is about the same.

If you are interested in testing it yourself, you can find the results of the investigation from the following resources.

Running custom ESBuild asset packer with almost no changes to the current SCSS files:

• Using this fork: https://github.com/mainio/decidim/tree/feature/esbuild-assets-performance
• Create the development app
• Run bundle exec decidimpack within the development app

Running custom ESbuild asset packer without SCSS compiler in the equation (PostCSS only, requires changes to the SCSS files):

• Using this fork: https://github.com/mainio/decidim/tree/feature/postcss-only
• Create the development app
• Install this module within the development app: https://github.com/mainio/decidim-module-assetspacker
• Run bundle exec decidimpack within the development app

[alecslupu]

@ahukkanen we already migrated to esbuild back in #12238 , ofc any improvements, recommendations are welcomed :)

[ahukkanen]

@alecslupu No, the core still uses Webpack to build the assets, not ESBuild. It is possible to drop Webpack out of the equation (as shown above with the remarks).

Note that these experiments have been made against the current develop branch of Decidim, i.e. 0.29.0.dev.

[andreslucena]

Sounds really well @ahukkanen, I'll check it out these days when I can allocate some time.