Hexo eats all the memory when generating

[steveww]

I am building image gallery site. It will have many large images, at the moment with just a small percentage of the images loaded Hexo uses all computer memory when generating the site.

Machine is Linux quad core with 8GB of memory. Hexo eats it all and brings the machine to a standstill due to excessive swapping.

There are 5.7GB of image files. About 1k images.

Looks like Hexo loads everything into memory to generate the site. This is insane!

[yoshinorin]

Please try concurrency option.

https://hexo.io/docs/commands#generate

[SukkaW]

Hexo doesn't know what files inside source folder should be used during generation, thus Hexo loads everything under source folder.

Try to configure skip_render in your _config.yml to exclude your images to see if it helps.

[steveww]

None of the suggestions made any difference. About 1.5GB of memory is used for generate which is about the size of the directory including all the images. I have my suspicions about the filter https://github.com/hexojs/hexo-filter-responsive-images I think it may be the culprit. I think it may be loading every image into memory to resize them.

[curbengh]

https://github.com/hexojs/hexo-filter-responsive-images I think it may be the culprit. I think it may be loading every image into memory to resize them.

Does disabling the plugin helped?

[steveww]

I disabled the plugin but it make very little difference.

[curbengh]

I would like to reproduce this. May I know your folder structure, specifically where you put the images (source/, themes/foo/source/, etc)?
How do you organize the posts? Do you enable post_asset_folder?

[steveww]

I did not use post_asset_folder. Each post represents a different gallery, which is dynamically built using my script. This just lists the target folder and creates a justified-gallery DIV. I've attached a recursive list of the folder structure.

ls.txt.gz

Here is my script.
tag-imgdir.js.gz

The more images I add the more memory hexo takes when generating.

[stale]

This issue has been automatically marked as stale because lack of recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stevenjoezhang]

I found that the external_link filter is consuming a huge amount of memory. You can try to disable it in _config.yml

external_link:
  enable: false

[stevenjoezhang]

I'm closing this issue due to inactivity. You can join the discussion here: #4922