[feat] Adds support for large assets (> 100 MB)

[sebiniemann]

This PR contains:

• bugfix
• feature
• refactor
• documentation
• other

Are tests included?

• yes (bugfixes and features will not be merged without tests)
• no

I didn't add any new tests, but expect the existing ones to be sufficient.

Breaking Changes?

• yes (breaking changes will not be merged unless absolutely necessary)
• no

List any relevant issue numbers:

• Extreme memory-intensive SHA256 calculation #3186

Description

Replaces hash.js with crypto (for NodeJS builds) and asmcrypto.js for browser builds to support assets with >100 MB.

After some benchmarks (see linked issue), crypto was chosen for NodeJS builds due to its (currently) superior performance. asmcrypto.js was then chosen instead of the Web Crypto API, as the digest function inside SubtleCrypto is async, while most of rollup is synchronous. Rewriting rollup for this seemed to be an overkill.

[sebiniemann]

@lukastaegert I had to fix an additional problem(?) after running all tests.

In line

rollup/src/Chunk.ts

Line 867 in d9c0a66

hash.update(options.format);

the format is directly added to the hash. Typescript however assumes that it can still be undefined at this point, which collides with update(data: string | Buffer).

Seeing many other lines which simply cast it to string, for example:

rollup/src/Chunk.ts

Line 1154 in d9c0a66

options.format as string,

I assume the the format being undefined is not actually possible at this point, given the current call path? Currently, I also added as string to this line, instead of surrounding it with an if clause.

[codecov]

Codecov Report

Merging #3194 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #3194      +/-   ##
==========================================
+ Coverage   90.19%   90.19%   +<.01%     
==========================================
  Files         165      167       +2     
  Lines        5905     5907       +2     
  Branches     1797     1797              
==========================================
+ Hits         5326     5328       +2     
  Misses        350      350              
  Partials      229      229

Impacted Files	Coverage Δ
src/utils/crypto.ts	100% <100%> (ø)
src/utils/FileEmitter.ts	100% <100%> (ø)	⬆️
browser/crypto.ts	100% <100%> (ø)
src/Chunk.ts	92.89% <100%> (ø)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b123169...0ea074c. Read the comment docs.

[sebiniemann]

There seems to be a problem with asmcrypto.js and modern V8 versions (see first circleci tests, NodeJS 12 builds).

After fixing the issue with Uncaught TypeError: Cannot read property '_stderr' of undefined to retrieve the actual error missing, it didn't really get much more useful:

(node:10968) V8: C:\Users\Sebi\Desktop\rollup\dist\shared\index.js:65 Invalid asm.js: Unexpected token

I am not sure what changed in V8 or asm.js that asmcrypto.js is now incompatible.

For now, NodeJS builds are stil capable to support large asset files by using the crypto library. For browser builds, I reverted it back to hash.js, cause it is synchronous in contrast to the Web Crypto API and also works under newer V8 versions, which may/are also used in browser.

[lukastaegert]

Very nice, thanks a lot! Only one minor thing I would change.

[sebiniemann]

Changed it to a paramter-less function :)

[lukastaegert]

Nice, that was quick! Thanks a lot!