[build] importing from hashed chunks makes caching terribly ineffective

[jacekkarczmarczyk]

Update

I've published a plugin that solves the issue for me: https://github.com/jacekkarczmarczyk/importmap-plugin
See #6773 (comment) for example usage in the issue reproduction repository

Describe the bug

Built files import other files which names contain content hash. So if the chunk A changes its contents then the output file changes its hash (A.123.js becomes A.234.js). So if there's other file that imports from A chunk then it also changes its contents and hash because import {...} from 'A.123.js' becomes import {...} from 'A.234.js'.

Imagine now that I'm defining an env variable with build time. Main chunk imports this file to show the build time in App.vue. However main chunk exports the following vue related function:

function lc (e, t, r, n, i, a, o, s) {
  var f = typeof e == 'function' ? e.options : e;
  t && (f.render = t, f.staticRenderFns = r, f._compiled = !0), n && (f.functional = !0), a && (f._scopeId = 'data-v-' + a);
  var u;
  if (o ? (u = function (d) {
    d = d || this.$vnode && this.$vnode.ssrContext || this.parent && this.parent.$vnode && this.parent.$vnode.ssrContext, !d && typeof __VUE_SSR_CONTEXT__ != 'undefined' && (d = __VUE_SSR_CONTEXT__), i && i.call(this, d), d && d._registeredComponents && d._registeredComponents.add(o);
...

that is being imported by all component chunks (note that this is just example, depending on the project there might be other user defined functions that are exported from main chunk, also in my actual project I've extracted vendors to separate chunks but still vue related functions were exported from main chunk).

So now when I build again env chunk will change its contents and hash/name, therefore main chunk will change it's contents and hash/name, therefor ALL other chunks will change their names. That makes caching very innefective. Also worth to mention that that technique worked fine in vue-cli - when build time changed only single small env.hash.js chunk was changed, all others remained unchanged (EDIT: there's also relatively small runtime chunk that also was changed)

Reproduction

https://github.com/jacekkarczmarczyk/vite-chunks-very-very-bad

Run yarn && yarn build && git add . && yarn build && git status

It behaves the same when displaying build time is moved to an async Home.vue route component, in this case hash change of main chunk is definitely unjustified

System Info

System:
    OS: Windows 10 10.0.19043
    CPU: (12) x64 Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz
    Memory: 3.34 GB / 15.87 GB
  Binaries:
    Node: 16.11.0 - C:\Program Files\nodejs\node.EXE
    Yarn: 1.22.10 - ~\AppData\Roaming\npm\yarn.CMD
    npm: 8.0.0 - C:\Program Files\nodejs\npm.CMD
  npmPackages:
    @vitejs/plugin-legacy: ^1.6.4 => 1.6.4
    vite: ^2.7.2 => 2.7.13

Used Package Manager

yarn

Logs

No response

Validations

• Follow our Code of Conduct
• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• Make sure this is a Vite issue and not a framework-specific issue. For example, if it's a Vue SFC related bug, it should likely be reported to https://github.com/vuejs/core instead.
• Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
• The provided reproduction is a minimal reproducible example of the bug.

Additional notes

I believe that this more like rollup issue (or whatever is used for generating chunks) but if it's possible to fix it by using some rollup's settings then vite should use these settings by default. And if it's not possible to fix it by using different settings then I think that chosing rollup was not the best choice. This page https://bundlers.tooling.report/hashing/js-import-cascade/
and this https://bundlers.tooling.report/hashing/avoid-cascade/ however suggest that this should not be an issue for rollup

[jacekkarczmarczyk]

Additional example 1 (with vue 3)

I've created a new branch in my reproduction so that maybe the problem is clearer:

https://github.com/jacekkarczmarczyk/vite-chunks-very-very-bad/tree/vue3

Run yarn && yarn build && git add ., then change <div>Home</div> to <div>Home2</div> in Home.vue and run yarn build && git status. View1 etc chunks should not be changed but that's what is happening:

Additional example 2 (without vue)

https://github.com/jacekkarczmarczyk/vite-chunks-very-very-bad/tree/no-vue

Update body of log() function in util.ts and rebuild. Only util chunk should be changed, but all 4 (index, foo, bar, util) are

Webpack example

https://github.com/jacekkarczmarczyk/vite-chunks-very-very-bad/tree/webpack-async

Change the text in util.js and run yarn build - only util and runtime chunks are changed, foo, bar and main remain the same

[anatolsommer]

I had a very similar problem and came up with this hacky solution:

import { defineConfig } from 'vite'
import { createHash } from 'crypto'
import vue from '@vitejs/plugin-vue'

export default defineConfig({
  plugins: [vue()],
  build: {
    rollupOptions: {
      output: {
        entryFileNames: 'assets/[name].js',
        chunkFileNames: chunkInfo => {
          if (chunkInfo.isDynamicEntry) {
            const hash = createHash('md5')
              .update(Object.values(chunkInfo.modules).map(m => m.code).join())
              .digest('hex')
              .substr(0, 6)
            return 'assets/[name].' + hash + '.js'
          } else {
            return 'assets/[name].[hash].js'
          }
        }
      }
    }
  }
})

When adding Cache-Control: must-revalidate, max-age=0 for /assets/index.js (my entry file) and Cache-Control: public, max-age=31557600 to all other assets it seems to work fine but I had no chance to really test it yet and I'm quite new to Vite/Rollup and ESM in general so this might be a terrible idea for some reason I don't know...

Every kind of feedback is very appreciated! :)

[redox]

@jacekkarczmarczyk I have very much the same problem due to some (react) lazy component loading. Did you manage to work-around the issue?

[jacekkarczmarczyk]

Yes, by using Vue Cli

[1391020381]

now ,How to configure Vite long-term cache?

[jfparadis-appomni]

There is an additional problem with circular dependencies causing all components to import the main index because Vite ejects a helper function there (a virtual module).

Here's how I prevent that helper function (and Vite polyfills) from being added to main index chunk (and keep it lean):

import { defineConfig } from 'vite'
import vue from '@vitejs/plugin-vue'

// https://vitejs.dev/config/
export default defineConfig({
  build: {
    target: 'ES2020',
    rollupOptions: {
      output: {
        // Workaround: Vite is bundling its plugins to the main index chunk, 
        // causing circular dependencies and cascading hash changes.
        manualChunks(id) {
          if (id.startsWith('vite/') || id.startsWith('\0vite/')) {
            // Put the Vite modules and virtual modules (beginning with \0) into a vite chunk.
            return 'vite';
          }
        },
      },
    },
  },
  plugins: [vue()]
});

[jacekkarczmarczyk]

@patak-dev sadly, #9870 didn't change much

Update: no improvement with vite 4.0.0-alpha.1 and rollup 3.3.x

Results with Vite 4.0.0-alpha.0

Example 1 (vue 2)

Steps: run yarn && yarn build && git add . && yarn build && git status

Vite 2.9 (main branch)

Vite 4.0 (main-v4 branch)

Example 2 (Vue 3)

Steps: run yarn && yarn build && git add ., then change <div>Home</div> to <div>Home2</div> in Home.vue and run yarn build && git status

Vite 2.9 (vue3 branch)

Vite 4.0 (vue3-vite4 branch)

Example 3 (no Vue)

Steps: run yarn && yarn build && git add ., then update body of log() function in util.ts and run yarn build && git status

Vite 2.9 (no-vue branch)

Vite 4.0 (no-vue-vite4 branch)

[jacekkarczmarczyk]

I don't think not much can be done here on Vite side (except using another bundler which I don't believe is going to happen soon), I've added the issue to rollup repo

[jacekkarczmarczyk]

For anyone interested - here's a proof of concept based on @lukastaegert's comment in rollup repo:

https://github.com/jacekkarczmarczyk/importmap-plugin (UPDATE: now supports SystemJS)

Feel free to steal the code and create a proper vite/rollup plugin. Any comments welcome

Plugin applied to the original reproduction (https://github.com/jacekkarczmarczyk/vite-chunks-very-very-bad/tree/main-stable-hash-plugin - yarn && yarn build && git add . && yarn build && git status) - working as expected (only modified chunk changed the hash):

[waynebrantley]

vite is fast and great, but this one feature really causes a lot of issues. For example on every update of our application - the dynamic loads all give errors because the bundle they need (even if it did not change) has a hash change - so the file does not exist.

@bluwy - is there plan to fix this caching issue - unsure if the suggestion above is the right fix. Just trying to get some eyes on this to maybe be fixed in 4.1?

[bluwy]

There really isn't a way to simply fix this as explained in #6773 (comment). The way bundlers work without a runtime proxy module loader means it would always have this hash cascading issue. Unless Vite implements it by default, which could be a possibility, but your best bet for now is to use the plugin: https://github.com/jacekkarczmarczyk/importmap-plugin.

[lukastaegert]

the dynamic loads all give errors because the bundle they need (even if it did not change) has a hash change - so the file does not exist.

This sounds like you need to fix your deployment strategy. My recommendation is to not delete the files from your previous deployment but just put the changed files next to it, that fixes this issue easily. Otherwise, there is no safe way to do a deployment without compromising long-running sessions. Another approach is not to use hashes but separate folders per deployment.

[waynebrantley]

@lukastaegert We do an automated deployment (OctopusDeploy) that contains the package of the site (client and server). This creates a brand new deployment, folder, etc with the files from the deployment (which would not contain old files by definition). This has the positive thing of being a brand new deployment with all new configs and such - but has the negative of old files not being there. With create-react-app (what we used before), this type of error occurred very infrequently (like once per month type of thing) instead of every deployment.

I agree we would compromise long-running sessions when the code they were needing changed. However, we bundle large dependencies separately - so rarely do the actually change - but with vite - appears they always change.

@bluwy thank you for the comment and consideration. I was just 'waiting for official fix', but will try above. Perhaps consider adding this side-effect to the docs in the meantime? maybe https://vitejs.dev/guide/features.html#async-chunk-loading-optimization and https://vitejs.dev/guide/build.html#chunking-strategy letting people that ViteJs will generate a new hash each time for dynamically loaded modules even if they did not change (resulting in a new filename each time) - eliminating the browser cache feature of generating 'stable chunks'.

Thanks again for a great product - our build times have never been faster!

[jacekkarczmarczyk]

Unless Vite implements it by default, which could be a possibility

@bluwy can I ask how high this possibility is? Is it something closer to "yeah, maybe, after we solve 1000 of other more important issues" or to "that's indeed a big problem we want to solve asap" (regardless of whether it's easy or hard to implement it).

@waynebrantley while my plugin seems to solve the problem in simple cases and I do use it in production with smaller projects, it wasn't tested (and I'm pretty sure it wouldn't work) in more complicated setups (like PWA or anything that relies on chunk names), so I'd prefer to see it as a built-in solution instead of 3rd party plugin.

Not sure if it needs some changes in rollup to make the change in Vite easier though, there doesn't seem to be much discussion on it in rollup repo (some of other discussions are linked in this issue), which surprises me a lot tbh as I don't believe that most of the projects are "release and forget" or that devs don't case about caching...

[bluwy]

@bluwy can I ask how high this possibility is? Is it something closer to "yeah, maybe, after we solve 1000 of other more important issues" or to "that's indeed a big problem we want to solve asap" (regardless of whether it's easy or hard to implement it).

It's more of "if someone's interested and creates a PR, we can discuss and review it" kinda situation. Since Vite's development comprises of community contributions, and sometimes sponsored work from metaframeworks, we don't have a list of things we have to do. For me, I usually implement features or bug fixes if I find them interesting.

[JaweenDeng]

@patak-dev sadly, #9870 didn't change much

Update: no improvement with vite 4.0.0-alpha.1 and rollup 3.3.x

Results with Vite 4.0.0-alpha.0

Example 1 (vue 2)

Steps: run yarn && yarn build && git add . && yarn build && git status

Vite 2.9 (main branch)

Vite 4.0 (main-v4 branch)

Example 2 (Vue 3)

Steps: run yarn && yarn build && git add ., then change <div>Home</div> to <div>Home2</div> in Home.vue and run yarn build && git status

Vite 2.9 (vue3 branch)

Vite 4.0 (vue3-vite4 branch)

Example 3 (no Vue)

Steps: run yarn && yarn build && git add ., then update body of log() function in util.ts and run yarn build && git status

Vite 2.9 (no-vue branch)

Vite 4.0 (no-vue-vite4 branch)

I meet same question and update vite@4.0.4,but not work,How did you solve it？

[yeion7]

I like the approach that parcel uses to resolve this issue

https://parceljs.org/features/production/#cascading-invalidation

[lukastaegert]

This is not straightforward as it requires you to have an intermediate loading handler that maps imports to their hashed counterparts when something is imported. That being said, this is possible by using import maps. While those are not yet widely adopted, this makes them work for at least 95% of browsers: https://github.com/guybedford/es-module-shims

Then you can basically completely remove hashes from file names and instead e.g. put them into import maps like this:

<script type="importmap">
{
  "imports": {
    "./my-chunk.js": "./my-chunk.js?ab34fa7"
  }
}
</script>

Here, ab34fa7 would be the content hash of ./my-chunk.js. This makes sure that whenever the content hash is changed, this specific chunk will be loaded again. Of course, your server needs to ignore the query parameter, but this is usually the case out of the box.

It should be easy to put this into a plugin, but I would not be surprised if someone has not done that already. It will completely avoid any hash cascades.

Note that it will not completely fix the original issue because long-running sessions through a re-deploy will still run into chunks that e.g. may be missing imports, or chunks that do not exist at all. I.e. chunk foo.js may have an import a before deployment, but an import b after deployment. If a long-running sessions through a re-deployment now triggers a dynamic import of foo.js expecting it to have an import a, it will likely crash with a runtime error. This would be avoided if you used content hashes on the actual files and deployed the previous deployment with the current deployment.

To fix that while preventing the hashing cascade, one could spin it differently:

All files have content hashes, but internally, they reference other chunks without hashes. To do that, one would need to create a bundle without hashes and in a post-processing step, calculate the content hashes of each chunk and change the file names and generate the import map.

<script type="importmap">
{
  "imports": {
    "./my-chunk.js": "./my-chunk-ab34fa7.js"
  }
}
</script>

[jacekkarczmarczyk]

It should be easy to put this into a plugin, but I would not be surprised if someone has not done that already

@lukastaegert Someone did, and it's already mentioned few times in this issue

[ijandc]

I had a very similar problem and came up with this hacky solution:

import { defineConfig } from 'vite'
import { createHash } from 'crypto'
import vue from '@vitejs/plugin-vue'

export default defineConfig({
  plugins: [vue()],
  build: {
    rollupOptions: {
      output: {
        entryFileNames: 'assets/[name].js',
        chunkFileNames: chunkInfo => {
          if (chunkInfo.isDynamicEntry) {
            const hash = createHash('md5')
              .update(Object.values(chunkInfo.modules).map(m => m.code).join())
              .digest('hex')
              .substr(0, 6)
            return 'assets/[name].' + hash + '.js'
          } else {
            return 'assets/[name].[hash].js'
          }
        }
      }
    }
  }
})

When adding Cache-Control: must-revalidate, max-age=0 for /assets/index.js (my entry file) and Cache-Control: public, max-age=31557600 to all other assets it seems to work fine but I had no chance to really test it yet and I'm quite new to Vite/Rollup and ESM in general so this might be a terrible idea for some reason I don't know...

Every kind of feedback is very appreciated! :)

this does not work with vite 4.x, the chunkInfo has no member named modules

[volkandkaya]

I had a very similar problem and came up with this hacky solution:

import { defineConfig } from 'vite'
import { createHash } from 'crypto'
import vue from '@vitejs/plugin-vue'

export default defineConfig({
  plugins: [vue()],
  build: {
    rollupOptions: {
      output: {
        entryFileNames: 'assets/[name].js',
        chunkFileNames: chunkInfo => {
          if (chunkInfo.isDynamicEntry) {
            const hash = createHash('md5')
              .update(Object.values(chunkInfo.modules).map(m => m.code).join())
              .digest('hex')
              .substr(0, 6)
            return 'assets/[name].' + hash + '.js'
          } else {
            return 'assets/[name].[hash].js'
          }
        }
      }
    }
  }
})

When adding Cache-Control: must-revalidate, max-age=0 for /assets/index.js (my entry file) and Cache-Control: public, max-age=31557600 to all other assets it seems to work fine but I had no chance to really test it yet and I'm quite new to Vite/Rollup and ESM in general so this might be a terrible idea for some reason I don't know...
Every kind of feedback is very appreciated! :)

this does not work with vite 4.x, the chunkInfo has no member named modules

Did you find a solution for v4?

[anatolsommer]

@ijandc + @volkandkaya
As it seems the new Rollup version that Vite 4 uses, solved the problem and now the entryFileNames: 'assets/[name].js' part alone (without chunkFileNames and the extra hashing) works quite fine.

[dukexie]

This problem also affects us, webpack uses contenthash, but it seems vite has no corresponding solution?

[volkandkaya]

Pretty sure this would cut my build time down by half due to Monaco Editor being included everytime.

[anatolsommer]

Since I got 8 downvotes: Seems like I don't get the problem (which is weird since people seemed to like my solution for Vite 2), it works perfectly fine for me. Multiple projects, many builds. As long as this is still about client side caching - no idea what that would have to do with build time.

(One component changed, every chunk except the one containing that code and the entry file stays unchanged.)

[dukexie]

@anatolsommer Yes, your screenshot above is awesome. I also very much hope that Vite 4 can also achieve this effect.
I think the above 8 downvotes are for entryFileNames: 'assets/[name].js' this one. Because there is still a need for [hash]. Otherwise there will be no cached value.

[anatolsommer]

@dukexie This is Vite 4 (it says vite v4.2.1 building for production) in the screenshot. Simply achieved by entryFileNames: 'assets/[name].js' (only entry files not chunks!) in the config since Vite 4 already uses hashes in filenames based on the content. Then I add Cache-Control: must-revalidate, max-age=0 header for entry files (/assets/index.js) and Cache-Control: public, max-age=31557600 to all other assets and caching works great.
(I also don't empty the output dir but run a job to delete files from older builds so already loaded pages don't get any 404 errors for assets from the previous build but that's a different story and not active in this screenshot.)

[dukexie]

@anatolsommer Oh I see, thank you.
But this is only a temporary method. After all, entry files max cache would be better.
I think this is a bug in vite build. Expect a fix.

[anatolsommer]

@dukexie I completely agree! Just as long as I can keep the entry file small and with ETags/304 I can live quite well with this workaround until there is an actual solution and tbh I think most people can. :)

[bhbs]

Implementing a solution for the Cascading Cache Invalidation problem using importmap as a feature of Vite might not be as complicated as it seems. As a PoC, I implemented the functionality to rewrite source code and generate importmap, and confirmed that it could pass the existing test cases.

• feat: 🧪 chunkMap #15373

Therefore, I have opened a Discussion proposing the implementation of a feature in Vite to solve the Cascading Cache Invalidation problem using importmap.

• RFC: Solve cascading cache invalidation problem using importmap #15372

[zhuchenglong]

@jacekkarczmarczyk I solved the hash change problem with importmap-plugin, but the ImportmapPlugin base is set to the same as' base 'option in Vite config and I get an Error: 'Error: Missing import map entry for entry file: /app/index.js', how to solve this?
output: { format: 'system', // 'system' or 'es' entryFileNames: 'app/index.js', // DO NOT INCLUDE HASH HERE chunkFileNames: 'chunks/[name].js', // DO NOT INCLUDE HASH HERE plugins: [ ImportmapPlugin({ base: '/admin/', // same asbase option in Vite config external: true, // external import maps work only for SystemJS indexHtml: 'index.html', // entry html file name hashLength: 8, }), ], },

[jacekkarczmarczyk]

@zhuchenglong Not sure why it's not working, but the plugin was just a proof of concept, and given that afair it's not compatible with Vite 4/5 and that there's a PR in Vite that seems to resolve the issue I'm not going to investigate.

[zhuchenglong]

@zhuchenglong Not sure why it's not working, but the plugin was just a proof of concept, and given that afair it's not compatible with Vite 4/5 and that there's a PR in Vite that seems to resolve the issue I'm not going to investigate.

@jacekkarczmarczyk Which version of vite are you using? I'm using vite4, but I haven't solved the hash change issue yet. Do you know which PR in Vite resolve the issue?

[jacekkarczmarczyk]

I'm not using Vite at the moment, PR is here #15373

[zhuchenglong]

@jacekkarczmarczyk Thank you for your answer. I have read #15373 , but I did not understand how to use chunkMap in the vite4/5 project. Can you give me a demo?