perf(processor/post): improve processing speed when `config.post_asset_folder` is enabled #5473

yoshinorin · 2024-04-18T17:12:55Z

refs:
- [Question] Use Hexo for generate more than 10000 posts #2579 (comment)
- Deployment is taking too much time #5456

I think other issues also related this PR

When slow?

config.post_asset_folder is enabled.

Why?

When config.post_asset_folder is enabled, Hexo calls the Post.toArray() function for each file. The time complexity of Post.toArray() is O(n). Additionally, Hexo uses the Post.find method to search for the post, and the average time complexity of find is also O(n).

So, if there are 100 articles and 100 files, the time complexity would be O(100 * 100 * 100) = O(n^3).

How to solve?

Avoid calling Post.toArray(). Instead, we will search based on whether the directory where the files are stored (assetDir) matches. Also, if the file found with PostAsset, use Post.findById. The time complexity of Post.findById is O(1).

Benchmark Environment

Below is bench env.

Machine

# OS
Microsoft Windows [Version 10.0.22631.3447]

# Cpu
AMD Ryzen 7 PRO 4750G with Radeon Graphics

# Memory
Capacity     Name             Tag
17179869184  Physical Memory  Physical Memory 1
17179869184  Physical Memory  Physical Memory 3

Hexo and Node.js

$ hexo -v
hexo: 7.2.0
hexo-cli: 4.3.1
os: win32 10.0.22631
node: 20.11.1
...
v8: 11.3.244.8-node.17

`package.json`

"dependencies": {
  "0x": "^5.7.0",
  "hexo": "file://C:\\Users\\<userName>\\development\\hexo\\hexo",
  "hexo-filter-nofollow": "2.0.2",
  "hexo-generator-archive": "git+https://github.com/hexojs/hexo-generator-archive.git#master",
  "hexo-generator-category": "git+https://github.com/hexojs/hexo-generator-category.git#master",
  "hexo-generator-index": "git+https://github.com/hexojs/hexo-generator-index.git#master",
  "hexo-generator-sitemap": "git+https://github.com/yoshinorin/_hexo-generator-sitemap.git#master",
  "hexo-generator-tag": "git+https://github.com/hexojs/hexo-generator-tag.git#master",
  "hexo-pagination": "git+https://github.com/yoshinorin/hexo-pagination.git#my-site",
  "hexo-renderer-ejs": "git+https://github.com/hexojs/hexo-renderer-ejs.git#master",
  "hexo-renderer-markdown-it": "git+https://github.com/hexojs/hexo-renderer-markdown-it#master",
  "hexo-server": "git+https://github.com/hexojs/hexo-server.git#master"
},

`_config.yml`

# Site
title: Dummy
subtitle:
description:
author: yoshinorin
language: ja_JP
timezone: Asia/Tokyo

meta_generator: false

# URL
url: https://example.net
root: /
permalink: :year/:month/:day/:title/
permalink_defaults:

pretty_urls:
  trailing_index: true  # https://example.com/foo
  trailing_html: false  # https://example.com/foo/index.html

# Directory
source_dir: source
public_dir: public
tag_dir: tags
archive_dir: archives
category_dir: categories
code_dir: downloads/code
i18n_dir: :lang
exclude:
  - '**/node_modules/**'
  - '**/.git/**'
skip_render:
  - '**/node_modules/**'
  - '**/.git/**'
ignore:
  - '**/node_modules/**'
  - '**/.git/**'

# Writing
new_post_name: :year/:month/:day/:title.md
default_layout: post
titlecase: false
filename_case: 0
external_link:
  enable: true
  field: site
  exclude:
render_drafts: false
post_asset_folder: true
relative_link: false
future: true
highlight:
  enable: true
  line_number: true
  auto_detect: false
  tab_replace:

# Category & Tag
default_category: uncategorized
category_map:
tag_map:

# Date / Time format
date_format: YYYY-MM-DD
time_format: HH:mm:ss
updated_option: mtime

# Pagination
## Set per_page to 0 to disable pagination
per_page: 10
pagination_dir: page

# Theme
theme: tranquilpeak # I'm using a theme that I've delete many features from https://github.com/LouisBarranqueiro/hexo-theme-tranquilpeak.

sitemap:
  path: "sitemap.xml"
  tags: false
  categories: false

markdown:
  render:
    html: true
    xhtmlOut: false
    breaks: true
    linkify: true
    typographer: true
    quotes: '“”‘’'
  plugins:
    - markdown-it-footnote
  anchors:
    level: 1
    collisionSuffix: 'v'
    permalink: true
    permalinkClass: header-anchor
    permalinkSymbol: ' '

nofollow:
  enable: true
  field: site

Theme

I'm using a theme that I've delete many features from https://github.com/LouisBarranqueiro/hexo-theme-tranquilpeak.

Benchmark Time (Before, After)

Processed data

Number of posts: 1773
Number of post assets: 1784
Avg of post content length: 3645

Number of pages: 23
Number of page assets: 81
Avg of page content length: 4217

Number of tags: 246
Number of categories: 170
Number of routes: 5335

Time

	Before	After
Cold Processing	33 sec	32 sec
Hog Processing	222 sec	23 sec

Command & Details

Cold Processing (run hexo clean before generate)

// run hexo clean before generate
$ hexo clean
$ Measure-Command { hexo g | Out-Default }

INFO  Files loaded in 19 s
...
INFO  5335 files generated in 13 s

TotalSeconds      : 33.5191346

Hot Processing (run hexo clean before generate)

// without hexo clean
$ Measure-Command { hexo g | Out-Default }

INFO  Files loaded in 3.52 min
...
INFO  1705 files generated in 8.64 s

TotalSeconds      : 222.3847807

After

Cold Processing (run hexo clean before generate)

// run hexo clean before generate
$ hexo clean
$ Measure-Command { hexo g | Out-Default }

INFO  Files loaded in 18 s
...
INFO  5335 files generated in 12 s

TotalSeconds      : 32.9146049

Hot Processing (run hexo clean before generate)

// without hexo clean
$ Measure-Command { hexo g | Out-Default }

INFO  Files loaded in 12 s
...
INFO  1705 files generated in 8.84 s

TotalSeconds      : 23.8749185

Gramegraph (Hot Processing only)

Before (Hot Processing)

After (Hot Processing)

github-actions · 2024-04-18T17:14:37Z

Flamegraph

https://47729dfa674ff522f2aeac959329bb003694b6b0-14-hexo.surge.sh/flamegraph.html
https://47729dfa674ff522f2aeac959329bb003694b6b0-16-hexo.surge.sh/flamegraph.html
https://47729dfa674ff522f2aeac959329bb003694b6b0-18-hexo.surge.sh/flamegraph.html

coveralls · 2024-04-18T17:15:52Z

Pull Request Test Coverage Report for Build 8753005942

Details

21 of 24 (87.5%) changed or added relevant lines in 1 file are covered.
2 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-0.05%) to 98.978%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
lib/plugins/processor/post.ts	21	24	87.5%

Files with Coverage Reduction	New Missed Lines	%
lib/plugins/processor/post.ts	2	97.73%

Totals
Change from base Build 8720332933:	-0.05%
Covered Lines:	9293
Relevant Lines:	9389

💛 - Coveralls

…t_folder` is enabled

yoshinorin · 2024-04-20T16:21:23Z

lib/plugins/processor/post.ts

@@ -268,29 +269,41 @@ function processAsset(ctx: Hexo, file: _File) {
  const PostAsset = ctx.model('PostAsset');
  const Post = ctx.model('Post');
  const id = file.source.substring(ctx.base_dir.length);
-  const doc = PostAsset.findById(id);
+  const postAsset = PostAsset.findById(id);


I renamed unclear naming.

yoshinorin · 2024-04-20T16:24:23Z

lib/plugins/processor/post.ts

  }

-  if (doc) {
-    return doc.remove();
+  // NOTE: Probably, unreachable.


Probably, will not be reached here unless the user intentionally changes the values in db.json. But, I'm not sure about this, so I haven't removed it.

yoshinorin · 2024-04-20T16:26:43Z

@hexojs/core
Dear team. This PR is ready for review.
Thanks :)

…t_folder` is enabled (hexojs#5473)

yoshinorin self-assigned this Apr 18, 2024

yoshinorin changed the title ~~WIP: perf(processor/post): improve processing speed if config.post_asset_folder is enabled~~ WIP: perf(processor/post): improve processing speed when config.post_asset_folder is enabled Apr 18, 2024

perf(processor/post): improve processing speed when `config.post_asse…

9385c81

…t_folder` is enabled

yoshinorin force-pushed the perf/improve-postprocesser branch from 131d295 to 9385c81 Compare April 18, 2024 17:29

refactor: use Post.findById if postAsset exists

44a14de

yoshinorin force-pushed the perf/improve-postprocesser branch from 2790445 to 44a14de Compare April 19, 2024 11:39

yoshinorin added 2 commits April 19, 2024 20:43

chore: add comment

f782736

chore: rename unclear variable name

03379e8

yoshinorin commented Apr 20, 2024

View reviewed changes

yoshinorin changed the title ~~WIP: perf(processor/post): improve processing speed when config.post_asset_folder is enabled~~ perf(processor/post): improve processing speed when config.post_asset_folder is enabled Apr 20, 2024

yoshinorin added the #perfmatters label Apr 20, 2024

yoshinorin marked this pull request as ready for review April 20, 2024 16:25

SukkaW approved these changes Apr 21, 2024

View reviewed changes

stevenjoezhang approved these changes Apr 23, 2024

View reviewed changes

SukkaW merged commit cc8c520 into master Apr 23, 2024
22 of 24 checks passed

SukkaW deleted the perf/improve-postprocesser branch April 23, 2024 12:03

yapinxxx pushed a commit to yapinxxx/hexo that referenced this pull request May 7, 2024

perf(processor/post): improve processing speed when `config.post_asse…

41818a5

…t_folder` is enabled (hexojs#5473)

yapinxxx pushed a commit to yapinxxx/hexo that referenced this pull request May 7, 2024

perf(processor/post): improve processing speed when `config.post_asse…

3be8421

…t_folder` is enabled (hexojs#5473)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

perf(processor/post): improve processing speed when `config.post_asset_folder` is enabled #5473

perf(processor/post): improve processing speed when `config.post_asset_folder` is enabled #5473

yoshinorin commented Apr 18, 2024 •

edited

github-actions bot commented Apr 18, 2024 •

edited

coveralls commented Apr 18, 2024 •

edited

yoshinorin Apr 20, 2024

yoshinorin Apr 20, 2024

yoshinorin commented Apr 20, 2024

perf(processor/post): improve processing speed when config.post_asset_folder is enabled #5473

perf(processor/post): improve processing speed when config.post_asset_folder is enabled #5473

Conversation

yoshinorin commented Apr 18, 2024 • edited

When slow?

Why?

How to solve?

Benchmark Environment

Machine

Hexo and Node.js

package.json

_config.yml

Theme

Benchmark Time (Before, After)

Processed data

Time

Command & Details

Cold Processing (run hexo clean before generate)

Hot Processing (run hexo clean before generate)

After

Cold Processing (run hexo clean before generate)

Hot Processing (run hexo clean before generate)

Gramegraph (Hot Processing only)

Before (Hot Processing)

After (Hot Processing)

github-actions bot commented Apr 18, 2024 • edited

Flamegraph

coveralls commented Apr 18, 2024 • edited

Pull Request Test Coverage Report for Build 8753005942

Details

💛 - Coveralls

yoshinorin Apr 20, 2024

Choose a reason for hiding this comment

yoshinorin Apr 20, 2024

Choose a reason for hiding this comment

yoshinorin commented Apr 20, 2024

perf(processor/post): improve processing speed when `config.post_asset_folder` is enabled #5473

perf(processor/post): improve processing speed when `config.post_asset_folder` is enabled #5473

yoshinorin commented Apr 18, 2024 •

edited

`package.json`

`_config.yml`

github-actions bot commented Apr 18, 2024 •

edited

coveralls commented Apr 18, 2024 •

edited