FATAL ERROR out of memory after using 2.8GB RAM (convertPathData, defaultoption) #954
Comments
JoKalliauer
changed the title
|
You can try replacing |
|
@hansottowirtz But sadly i am using Windows 10. cmd.exe which means (English) cygwin: cygwin: MINGW64: (bash: node: commando not found) |
|
For Windows you have to set an environment variable like this: So this could become: set NODE_OPTIONS=--max_old_space_size=8192 && node C:/Users/jkalliau/AppData/Roaming/npm/svgo -i Input2.svg -o output2.svg |
|
@hansottowirtz in cmd.exe: I installed svgo from Cygwin, maybe I have to deinstall it and install it from cmd.exe |
|
Even
Did you got this working ? , i have installed from node command prompt but it still does not work and throws error your mentioned above |
It still does not work on Windows, with the error-messages above I tried on Ubuntu: #!/bin/bash
wget https://upload.wikimedia.org/wikipedia/commons/archive/2/2e/20180507223736%21Germany_%28%2Bdistricts_%2Bmunicipalities%29_location_map_2013.svg -O input.svg
svgo -i input.svg -o output.svgand I get: #
# Fatal error in , line 0
# API fatal error handler returned after process out of memory
#And the Comment from @hansottowirtz leads to wget https://upload.wikimedia.org/wikipedia/commons/archive/2/2e/20180507223736%21Germany_%28%2Bdistricts_%2Bmunicipalities%29_location_map_2013.svg -O input.svg
NODE_OPTIONS=--max_old_space_size=8192
/usr/bin/svgo -i input.svg -o output2.svgand results on Ubuntu in <--- Last few GCs --->
[10799:0x2650db0] 580183 ms: Mark-sweep 1380.5 (1459.1) -> 1380.5 (1460.1) MB, 1004.1 / 0.0 ms allocation failure GC in old space requested
[10799:0x2650db0] 581383 ms: Mark-sweep 1380.5 (1460.1) -> 1380.5 (1429.1) MB, 1199.7 / 0.0 ms last resort GC in old space requested
[10799:0x2650db0] 582150 ms: Mark-sweep 1380.5 (1429.1) -> 1380.5 (1429.1) MB, 766.9 / 0.0 ms last resort GC in old space requested
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x2aa043257c1 <JSObject>
1: /* anonymous */(aka /* anonymous */) [/usr/lib/node_modules/svgo/plugins/convertPathData.js:~652] [pc=0x2dd33817bb47](this=0x1881e95822d1 <undefined>,item=0x249aafffbc71 <Object map = 0x2925d02feb11>,index=54)
2: arguments adaptor frame: 3->2
3: filter(this=0x3f02cb2de551 <JSArray[146]>)
4: fn [/usr/lib/node_modules/svgo/plugins/convertPathData.js:~59] [pc=0x2dd33815a1f8](this=...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node::Abort() [node]
2: 0x11e7fec [node]
3: v8::Utils::ReportOOMFailure(char const*, bool) [node]
4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]
5: v8::internal::Factory::NewCode(v8::internal::CodeDesc const&, unsigned int, v8::internal::Handle<v8::internal::Object>, bool, int) [node]
6: v8::internal::CodeGenerator::MakeCodeEpilogue(v8::internal::TurboAssembler*, v8::internal::EhFrameWriter*, v8::internal::CompilationInfo*, v8::internal::Handle<v8::internal::Object>) [node]
7: v8::internal::compiler::CodeGenerator::FinalizeCode() [node]
8: v8::internal::compiler::PipelineImpl::FinalizeCode() [node]
9: v8::internal::compiler::PipelineCompilationJob::FinalizeJobImpl() [node]
10: v8::internal::Compiler::FinalizeCompilationJob(v8::internal::CompilationJob*) [node]
11: v8::internal::OptimizingCompileDispatcher::InstallOptimizedFunctions() [node]
12: v8::internal::StackGuard::HandleInterrupts() [node]
13: v8::internal::Runtime_StackGuard(int, v8::internal::Object**, v8::internal::Isolate*) [node]
14: 0x2dd337f842fd
Aborted (core dumped)
|
|
I believe you need to set the environment variable for NODE_OPTIONS. On Windows via:
On Linux via:
Alternatively, you can try svgcleaner. I found this reduces size more and is extremely fast, as it is written in Rust. |
|
Is there here any potential for for DoS attack? |
|
I managed to make this compress on Windows with Node.js 14.15.5 and That being said, maybe there are some improvements that can be made to reduce memory allocations? |
|
@TrySound might be worth documenting the workaround for all OS'es in README for such special cases. The above will work on Windows. I'm pretty sure we can improve the situation a bit later, but I'm not sure how much. |
|
BTW @TrySound 2.1.0 vs 2.2.2 has a huge speed penalty, at least on my Windows VM: We really need a benchmark script. BTW the speed issue is noticeable with smaller files too, but it's not so huge. https://github.com/svg/svgo/compare/v2.1.0...v2.2.2?w=1 Now, I'm not sure which commit is to blame yet. I could try to pinpoint it in the next days unless someone else beats me to it. I suggest that you finish with the prettier stuff so that it doesn't pollute the git history anymore; it should have been made in one commit. And also add npm scripts to check and format. |
|
Probably computing new styles. It does not cache anything yet. |
|
I'll open a new issue for the performance regression. |
|
I'll close this issue as it's quite open-ended, and we're actively trying to improve the performance of SVGO as we go. I haven't personally been able to reproduce it under the same conditions, but appreciate that given a large enough SVG or low enough available memory this will still occur. My hope is that the situation here will improve over the next few releases. Here are some stats showing how SVGO is performing with the
Overall, time has significant improved, memory usage has marginally improved, and compression has negligibly improved. The rationale for the time and memory improvements are mostly for #1918, and once merged will be used as the benchmark to ensure we understand the performance implications of new contributions. Meanwhile, we may explore solutions like chunking large SVG files or clearing caches when we're using too much memory, etc. That would undermine speed ofc, but may improve stability if users are having OOM issues. Rather than take action, since it's unclear when we've done enough, we'll just ensure all future work keeps memory/performance in mind, and that we profile SVGO and upstream dependencies to see where we can improve. |
|
I retried it on another computer, and I can't reproduce the bug either. Edit: Others can still reproduce the bug. |
|
I can certainly reproduce the issue: |
|
If you increase the old space size to 4000, it works fine. I'm not sure if it's worth the time to try to decrease the amount of RAM used on giant files. |
|
Yeah, I'm aware I just to make it clear that the issue does exist and it's not solved. |
Thanks for sharing, I figured on some environments it'd still be possible.
In my opinion, it is worth the time. It's not top priority, but making SVGO work with large files and be better equip to operate on smaller environments like phones or CI is favorable. For example, I wouldn't want OhMySVG crashing on my phone (PinePhone/Linux) because SVGO is using too much memory. If we can find better ways to store and process data that reduces our memory footprint, it's just better for everyone. Allocating memory takes time, which we're trying to get down, and reserving resources that other processes or the system memory cache could be using instead isn't considerate of our users or the other software installed on their machine. TL;DR: Large files not being our primary use case, or the assumptions that clients have the resources to spare, isn't a good reason not to try to use resources more efficiently. Edit: Sorry I'm less active right not btw, I'm sick. 😷 |
Prozessing the olderst version of File:Germany (+districts +municipalities) location map.svg (~73MB) fails afer needing more than 2,8GB RAM (I have 16GB RAM total, and during processing always more than 8GB free)
The text was updated successfully, but these errors were encountered: