Improve caching when creating multiple bundles

[JustinAzoff]

We have some automation that puts together multiple bundles with different sets of packages. Lately this has gotten to be quite slow. It appears that this is because git information isn't always cached.

I created a test manifest that fully pins each package to a specific commit, but there looks to be a missed opportunity to use cached package information:

here's how I am testing:

[bundle]
https://github.com/corelight/log-add-vlan-everywhere = 6222dfc6948cb567e6f54b8b0e2a1f83c3b9a052
https://github.com/corelight/SIGRed = b462f27066ea878c91fd604387f3d585e2fef325
https://github.com/corelight/callstranger-detector = 381851e9d100406b8a5aec07884bf55feb1017b4
https://github.com/0xxon/cve-2020-13777 = 454691a54356373d7af8324c008516f8f5725f27

then I run

zkg -vvv bundle --force ./bundles/test.bundle --manifest ./test-faster.manifest

Each run the output looks like

$time zkg -vvv bundle --force ./bundles/test.bundle --manifest ./test-faster.manifest 
2021-12-16 14:52:45 DEBUG    init Manager version 2.12.0
2021-12-16 14:52:45 DEBUG    found source clone of "zeek" at /Users/justin/.zkg/clones/source/zeek
2021-12-16 14:52:46 DEBUG    getting info on "https://github.com/corelight/log-add-vlan-everywhere"
2021-12-16 14:52:46 DEBUG    checked out "https://github.com/corelight/log-add-vlan-everywhere", branch/version "6222dfc6948cb567e6f54b8b0e2a1f83c3b9a052"
2021-12-16 14:52:46 DEBUG    getting info on "https://github.com/corelight/SIGRed"
2021-12-16 14:52:47 DEBUG    checked out "https://github.com/corelight/SIGRed", branch/version "b462f27066ea878c91fd604387f3d585e2fef325"
2021-12-16 14:52:47 DEBUG    getting info on "https://github.com/corelight/callstranger-detector"
2021-12-16 14:52:48 DEBUG    checked out "https://github.com/corelight/callstranger-detector", branch/version "381851e9d100406b8a5aec07884bf55feb1017b4"
2021-12-16 14:52:48 DEBUG    getting info on "https://github.com/0xxon/cve-2020-13777"
2021-12-16 14:52:49 DEBUG    checked out "https://github.com/0xxon/cve-2020-13777", branch/version "454691a54356373d7af8324c008516f8f5725f27"
2021-12-16 14:52:49 DEBUG    getting info on "https://github.com/corelight/log-add-vlan-everywhere"
2021-12-16 14:52:50 DEBUG    checked out "https://github.com/corelight/log-add-vlan-everywhere", branch/version "6222dfc6948cb567e6f54b8b0e2a1f83c3b9a052"
2021-12-16 14:52:50 DEBUG    getting info on "https://github.com/corelight/SIGRed"
2021-12-16 14:52:50 DEBUG    checked out "https://github.com/corelight/SIGRed", branch/version "b462f27066ea878c91fd604387f3d585e2fef325"
2021-12-16 14:52:50 DEBUG    getting info on "https://github.com/corelight/callstranger-detector"
2021-12-16 14:52:51 DEBUG    checked out "https://github.com/corelight/callstranger-detector", branch/version "381851e9d100406b8a5aec07884bf55feb1017b4"
2021-12-16 14:52:51 DEBUG    getting info on "https://github.com/0xxon/cve-2020-13777"
2021-12-16 14:52:52 DEBUG    checked out "https://github.com/0xxon/cve-2020-13777", branch/version "454691a54356373d7af8324c008516f8f5725f27"
Bundle successfully written: ./bundles/test.bundle

real	0m9.381s

I figure this can't work if you used = master as the version, but pinning a specific version should be able to be cached, right?

[JustinAzoff]

I have some commits here that help a bit: master...JustinAzoff:package-manager:faster-bundles

There were a few simple optimizations that could be made to speed up the git bits.

I also found that if you attempt to bundle a local git repo, it ends up being cloned twice, once just to get a list of the tags. Since it's already a local repo, that clone can be skipped.

[ckreibich]

Thanks Justin. I have a batch of zkg work coming up and hopefully can resolve this one then.

[JustinAzoff]

Cool! it's not a totally full fix, but changing the process of building bundles to cloning all repositories across the manifests, then rewriting the manifests to use the local clones results in being able to build all the bundles in 2 minutes vs 20.

Just the branch I have gets things down from 9s to 6s or so. The "clone first then bundle local repos" takes 2.75s.