Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: increase the maximum possible file size that jest-haste-map can handle #13094

Merged
merged 2 commits into from Aug 5, 2022

Conversation

lanpai
Copy link
Contributor

@lanpai lanpai commented Aug 5, 2022

Summary

Currently, the read file is decoded using UTF8 regardless of whether the worker needs to or not. This bottlenecks the maximum file size that the worker can hash to ~536MB (maximum length of a string) which causes a hard crash if exceeded. For files that don't need to be decoded and are only hashed, this change allows the worker to hash the file without decoding it, leaving it as a buffer. Since buffers have a maximum size of ~2GB on 64-bit systems, this increases the maximum possible file size significantly.

For files that were already successfully decoded in previous steps (eg. JSON files), the worker will continue to hash the already decoded content string instead of rereading the file.

Test plan

Tests for the hashing are already handled in jest-haste-map/src/__tests__/worker.test.js

@lanpai lanpai changed the title Only use content if already initialized fix: increase the maximum possible file size that jest-haste-map can handle Aug 5, 2022
Copy link
Member

@SimenB SimenB left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cool, works for me!

@SimenB SimenB merged commit 59e51df into jestjs:main Aug 5, 2022
@SimenB
Copy link
Member

SimenB commented Aug 7, 2022

@SimenB
Copy link
Member

SimenB commented Aug 25, 2022

@github-actions
Copy link

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 25, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants