How to use next to CompressionPlugin

[emilsedgh]

Hi there.

According to the documentation Sentry doesn't support GZip Artifacts.

Now, if you are using Webpack, there's a very good chance that you have CompressionPlugin enabled like I do (who doesn't compress their assets?)

Is there any way to run this plugin next to CompressionPlugin?

Loading this plugin first doesn't seem to be effective.

[sibiraj-s]

Try this plugin. It can help. copy-asset-in-memory-webpack-plugin. We do this internally, just published it as a plugin.

{
  plugins: [
    new CopyAssetInMemoryPlugin({
      test: /\.js(.map)?$/,
      to: (fileName) => {
        return fileName.replace("js/", "sentry/"); // <-- writing all files into sentry folder from js folder
      },
    }),
    new CompressionPlugin({
      filename: "[file]",
      exclude: /sentry/, // <-- exclude sentry
      minRatio: 10,
      deleteOriginalAssets: true,
    }),
    new SentryCliPlugin({
      release: APP_VERSION,
      include: PATHS.outDir + "/sentry/", // <-- uploading only the uncompressed files
      urlPrefix: "~/js", // <-- correcting url prefix to match the original assets
    }),
  ];
}

Note: The plugin only supports webpack 5

[kamilogorek]

Closing the issue, as it seems like the original issue has a working solution and there doesn't seem to be more we can easily do here. If it's still an issue, please do not hesitate to ping me if it is still relevant, and I will happily reopen it.
Cheers!

[grug]

Is there any chance this can be reopened? Not everyone is able to serve their application behind a CDN or server that can do the compression for us and not being able to use the compression plugin without the in-memory-copy solution in this thread seems a bit disappointing for a library that gets 4,000,000 downloads a month.

[lforst]

Hi @grug, not gonna promise anything on this because the team's priorities are somewhere else right now, but if we were to tackle this, how would it be solved best in your opinion?

My solution brainstorm rn:

• Uncompress files before uploading to Sentry?
• Accept gzipped files on Sentry backend

If you wanna take a stab at this yourself - PRs are always welcome!

[lforst]

Ok, after some internal discussions, the Sentry backend should be able to handle compressed files by now. Our approach will most likely involve adding a flag/option to Sentry CLI and adjusting the Webpack plugin to use that flag.

I'm gonna reopen this and put it in our backlog

[kamilogorek]

@lforst we already have the necessary functionality in the CLI, you can pass custom headers to all requests, so it should only require changes to the plugin itself. However, this one should be only applied to upload requests, not release creation/finalization.
If we decide to update a plugin to v2 (cc @lobsterkatie), we should also improve cli's JS-api, so it allow for multiple headers (which is possible in the binary itself).

[grug]

Thanks for getting back to me so quickly on this everyone. This is amazing.

@kamilogorek when you say the functionality is already in the CLI, does that mean there's a way today to add Accept-Encoding: gzip to file uploads to Sentry via the CLI? I'd be happy to switch to that until something was ready in the sentry-webpack-plugin

[kamilogorek]

@kamilogorek when you say the functionality is already in the CLI, does that mean there's a way today to add Accept-Encoding: gzip to file uploads to Sentry via the CLI?

Yup.

sentry-cli sourcemaps upload ./dist --release=foo --header="Content-Encoding: gzip"

Note that its probably Content-Encoding, not Accept-Encoding, as you are the sender. Also make sure that if you use it, you also specify deflate.

[grug]

Ah yep, I meant Content-Encoding: gzip - my bad 😂

This is great - I really appreciate such a quick turnaround on this. Is there anywhere I can follow for progress on the webpack plugin, or will this thread be fine?

Regarding specifying deflate, where am I specifying that? I wouldn't have imagined I'd need to specify it in the Content-Encoding header when sending up the files?

[kamilogorek]

Is there anywhere I can follow for progress on the webpack plugin, or will this thread be fine?

Yup, should be fine with the reopened status.

Regarding specifying deflate, where am I specifying that?

You need to list them in the same header, and in order the compression was applied, eg. Content-Encoding: deflate, gzip

[grug]

You need to list them in the same header, and in order the compression was applied

I'm fairly sure that CompressionWebpackPlugin applies only gzip by default and can only apply one compression algorithm. Is it fine in that case to only specify Content-Encoding: gzip?

[kamilogorek]

Totally. Deflate is just another compression algorithm, so if it's not used, just skip it.

[grug]

Am I correct in assuming that I can roll this all up into the following command: sentry-cli releases files $SENTRY_RELEASE upload-sourcemaps dist --url-prefix '~/foo/bar' --header="Content-Encoding: gzip" - I assume so but just wanted to check since the example given was using the sentry-cli sourcemaps upload command :)

[kamilogorek]

You can as long as you are using sentry-cli@v2. v1 doesn't have global headers.

If you are using the old major version, then it'll be:

CUSTOM_HEADER="Content-Encoding: gzip" sentry-cli releases files $SENTRY_RELEASE upload-sourcemaps dist --url-prefix '~/foo/bar'

Webpack plugin has this old behavior too, so you can try it out (not sure if it will work, as it will apply that header to all sentry requests, buuut maybe it will; don't have enough spare time to validate atm).

new SentryCliPlugin({
  // other options
  customHeader: 'Content-Encoding: gzip'
})

[grug]

We're fetching the CLI in our pipelines via the recommended way in the docs. Looking at the script at https://sentry.io/get-cli/ it looks fairly unlikely we're going to get an old major release, I believe!

[kamilogorek]

Yup, get-cli will get you v2 by default.

[kamilogorek]

sentry-cli releases files $SENTRY_RELEASE upload-sourcemaps dist --url-prefix '~/foo/bar' --header="Content-Encoding: gzip"

and

sentry-cli sourcemaps upload dist --release=$SENTRY_RELEASE --url-prefix '~/foo/bar' --header="Content-Encoding: gzip"

are equivalent in v2.

[grug]

You are a legend. Thank you so much for giving so much of your time. I hope other people stumbling across this thread find it helpful, too.

One other thing I can think of is that the docs on troubleshooting sourcemaps reflect the previous state of our conversation where artefacts couldn't be compressed. I assume this will be on the todo list to reflect the current state but just wanted to point it out 😄

[grug]

Is there any way of verifying that this all works end to end other than waiting for an error to occur in production?

[kamilogorek]

I take my words back. I found some time, and unfortunately cannot make it work. It appears that the header itself is not picked up. I'll schedule some time to debug this and will get back to you soon.

[grug]

@kamilogorek that's interesting. I'm hoping that it's a simple one to track down and debug as it'd be great for the support for compressed artefacts 😄

[kamilogorek]

@grug you should be able to update cli to 2.4.0, pass --decompress flag and give it a go now :)

[grug]

Hey @kamilogorek - very cool. I've just had a look at the release notes and was just wondering if I still need to add --header="Content-Encoding: gzip" to my CLI command or if replacing that with --decompress is enough?

[kamilogorek]

No, you can skip all headers work entirely now, its being decompressed prior to sending.

[grug]

Excellent. Out of curiosity - will the webpack plugin also eventually support this?

[kamilogorek]

For sure, but only once they upgrade the cli to v2. Cannot tell the exact date, as updating it means dropping support for Node <12, but we're already discussing the move here.

[grug]

That's cool - I appreciate the info 😄

[lforst]

The newest version of the plugin uses CLI v2 so this should be resolved. Feel free to reopen if you encounter any issues!