Glob speed really slow

[mjmjelde]

Howdy,

I just started using nfpm and am trying to package some larger packages with it, so I might have hit an edge case with what I am trying to use nfpm on. The packages that I am trying to make are gigabytes in size, but also contain thousands to hundred of thousands of files in them. When I try to package these with nfpm, it never gets to the packaging stage. I debugged a little bit, and it get to the ExpandContentGlobs function in nfpm/files/files.go during the validate stage and never seems to finish that.

For example, this current package I am trying to make contains 159k files and is 4.9G in size. Using fpm on this takes roughly three minutes to package it up. However, using nfpm never finished (I killed the process after an hour and a half). Is there anyway to speed up the globbing function? I tried with both disable_globbing set to true and false but nothing changed.

[erikgeiser]

We probably shouldn't try to append uniquely but append first and unique later. That should reduce the complexity. I'll bench it and see if I can get some improvement.

[erikgeiser]

Yep, my benchmarks show there is room for improvement:

goos: darwin
goarch: amd64
pkg: github.com/goreleaser/nfpm/v2/files
cpu: Intel(R) Core(TM) i5-6267U CPU @ 2.90GHz
BenchmarkExpandContentGlobs-4       	       1	8943350742 ns/op	5948665080 B/op	 2963056 allocs/op
BenchmarkExpandContentGlobsFast-4   	       2	 690795220 ns/op	503257972 B/op	 1260368 allocs/op
PASS
ok  	github.com/goreleaser/nfpm/v2/files	11.305s

I'll open an issue and submit a PR if I find some time to clean up my proof-of-concept.

[mjmjelde]

Awesome, thanks for testing this out so fast!

[erikgeiser]

Checkout #388, I simulated your use case with 150k files and was able to reduce the time from 50 mins to 9 seconds.

[mjmjelde]

Just tested it out and it works great! A build that I was doing that took over 8 hours (and never finished might I add) finished in just over 7 minutes now. Thanks!

[erikgeiser]

Thanks for your feedback. I'm glad we caught that one. Next time I'll think twice about Big-O 😅