Build time exploding with many pages

[klieret]

My Environment

Software	Version(s)
Operating System	Ubuntu 19.10
jekyll	4.0.0
github-pages	n/a

---

Expected Behaviour

The build time with n identical pages should scale linearly (or in other words, the build time of a single page should stay constant).

Current Behavior

The build time per page (!) grows to more than a second per page (!):

Code Sample

Speed testes as follows:

jekyll new speed_test_jekyll  
cd speed_test_jekyll
mkdir pages
cd pages
# create 150 pages (simply copy the example post 150 times over)
for i in {1..150}; do /bin/cp ../_posts/*-welcome-to-jekyll.markdown $i.md; done
cd ..
bundle exec jekyll build

Further investigations

• Using build --verbose one can see that the time is spent per page, rather than e.g. at the beginning or end
• Removing inclusions from the templates doesn't change anything

[ashmaroli]

@klieret Thank you for opening an interesting ticket.
From the test steps, it looks like you're using the minima theme..? If so can you run the tests again and generate the graph by removing jekyll-seo-tag from the mix..?

Follow these steps to do so:

• Create a new directory at source labeled _includes.
• Copy _includes/head.html from the minima gem on your system to the source directory.
• Remove {% seo %} from <source_dir>/_includes/head.html

[klieret]

Thank you for your incredibly fast reply!

Stripping down head.html didn't change anything

I think the culprit is the navigation in header.html. Removing everything there makes everything run instantaneously again.

[ashmaroli]

Stripping down head.html didn't change anything

Thank you for testing.
This is interesting. Hope you cleared the .jekyll-cache directory between runs.

I think the culprit is the navigation in header.html.

Probably. There is a Liquid {% for ... %} loop and a couple of Liquid map filters piped to site.pages..
But I'm not able to wrap my head around how increasing the no.of documents inside _posts directory would affect site.pages...
Documents under _posts are inside site.posts while site.pages remain constant (in your test site) with just index.markdown, about.markdown and the css artefacts..

[ashmaroli]

Ah..! What your test script does is to copy the 2020-03-21-welcome-to-jekyll.markdown from _posts directory over to pages directory 150 times..
That explains everything..
To make matters worse, there's also a Liquid where filter piped to site.pages.....

If anything, this ticket can serve as a call for the need to optimize the include in the minima theme..
Thanks.

[klieret]

Ah, perhaps I should have stressed that: As in the code snippet above, I was considering a case with many pages (the markdown files are created in the pages directory), so indeed the number of links will increase (if not excluded)

[klieret]

Exactly!

[klieret]

I'll open a report in the minima theme then!

Thanks a lot for helping me figure this out!

[ashmaroli]

@klieret If you'd like to experiment, we've a pull request here on jekyll/jekyll that'll speed things up on your test-site out-of-the-box: #7992
Once you generate the test-site, edit your Gemfile to point to the pull-request branch:

gem 'jekyll', github: 'jekyll/jekyll', ref: 'refs/pull/7992/head'

Now running bundle exec jekyll build should be much faster. It is slated for release as part of Jekyll 4.1..
(You can use the master branch similarly to see the difference between v4.0 and the future.. 😉)

[klieret]

Indeed, wow! This makes a huge difference, reducing the time from 75s to just 6.5s (150 pages)! Thanks for pointing this out, can't wait till the 4.1 release :)

[ashmaroli]

@klieret 75s for 150 pages is a lot. Even on Windows, the build times were just 30s for 150 pages (with Jekyll 4.0 released gem)

I'm curious. Will you be able to plot a graph for the test-site builds with the above PR branch..? Thanks.

(You can easily avoid the disk-cache when using that branch by passing --disable-disk-cache to the build command.)

[klieret]

Hmm, perhaps my system was busy yesterday, I can't quite reproduce the high numbers.

But I just ran the normal v4.00 version and the #7992 branch a couple of times and get the following:

So the behaviour stays the same (and is probably due to the scaling of the where clause etc, I assume)

[ashmaroli]

What's 1.8*new supposed to be..?

[klieret]

Oh I just noticed that the curves are identical but for a factor of ~1.8, so 1.8*new is the new values multiplied by 1.8 (can be ignored)

[ashmaroli]

Ah! Lastly, may I know how you generate the graphs..?
Thanks for all the input either ways.

[klieret]

More boring than you'd think (unfortunately), I just jotted down the numbers

import matplotlib.pyplot as plt
import numpy as np

fig, ax = plt.subplots()
n_files = [10, 50, 100, 150, 200]
ax.plot(
    n_files,
    [0.323, 1.5, 4.416, 10.92, 19.184],
    color="black",
    label="v4.0.0"
)
ax.plot(
    n_files,
    [0.452, 0.704, 2.098, 5.442, 11.483],
    color="red",
    label="new"
)
ax.plot(
    n_files,
    1.8*np.array([0.452, 0.704, 2.098, 5.442, 11.483]),
    color="yellow",
    label="1.8*new"
)
ax.set_xlabel("Number of posts")
ax.set_ylabel("Seconds to build")
ax.legend()

But it would be probably easy to write a short shell script to automatize the number taking to some csv file and then load this I assume.

[ashmaroli]

😃 When it comes to crunching data, Python's got you covered.. hehe..
Once again, thank you for opening this ticket and sharing your insights.