The latest version returns an error: too many open files

[screwyprof]

After updating to the latest version I get the following error:
open <file-name>: too many open files

I've got the following target in my Makefile:

LOCAL_PACKAGES="gitlab.com/cool-company/"
GO_FILES = $(shell find . -name "*.go" | grep -v vendor | uniq)

fmt: ## format code
	@gofumpt -l -w $(GO_FILES)
	@gci -w -l $(LOCAL_PACKAGES) $(GO_FILES)	

Interestingly enough gci doesn't have this problem

[mvdan]

What system are you on? If it is unix-like, can you show the output of ulimit -S -n and ulimit -H -n?

[screwyprof]

I'm on mac Big Sur. Don't know the previous limit as I temporary fixed it by setting the ulimit -n 10240.

I've tried to use find + xargs, but it doesn't help:

❯ find . -path ./vendor -prune -o -name \*.go -print0 | xargs -0 gofumpt -w -l

 ulimit -S -n
256

❯ ulimit -H -n
unlimited

[mvdan]

What if instead you use a directory, like gofumpt -l -w .? It could be that the problem is supplying all the files at once rather than letting gofumpt walk the directories to find Go files itself.

[mvdan]

Also, can you reproduce if you use gofmt from Go 1.18 instead of gofumpt?

[screwyprof]

gofmt doesn't have this issue. I'm using 'go 1.17'.

❯ gofumpt -l -w .
open handler/uuid.go: too many open files
open <file-name>: too many open files
...
fcntl <file-name>: too many open files

❯ sysctl -a | grep maxfiles
kern.maxfiles: 122880
kern.maxfilesperproc: 6144

[mvdan]

Please upgrade to Go 1.18 and compare gofumpt to gofmt from that version, as they should be opening the same amount of files with the same amount of parallelism. gofumpt is a modified version of gofmt from Go 1.18, and 1.18 is the version that made gofmt parallel. So if gofumpt is opening too many files, gofmt from 1.18 should be doing the same, and we should further investigate the issue with upstream.

[mvdan]

I should also note I tried to reproduce with ulimit -n 256; gofumpt **/*.go in $(go env GOROOT)/src and I couldn't get a failure.

[screwyprof]

I cannot simply update to go1.18 in my environment it maybe break some stuff related to deps management as they've deprecated some features. Not to mention that some colleagues of mine already complained about new version screwing up some old packages for some reason. Do you by any chance have a ready-to use Dockerfile?

I am not sure, but you may not have that much files in your root path which breaks it. But really it could be anything else. I've also noticed that macOS doesn't simply allow to set the ulimit to a higher value. The last comment on superuser.com suggest turning on a performance mode.

Update: mvdan.cc/gofumpt@v0.2.1 works great for me atm.

[screwyprof]

I've created a simple Dockerfile to test it with go1.18

❯ cat Dockerfile
FROM golang:1.18 AS build
RUN go install mvdan.cc/gofumpt@latest

FROM scratch
WORKDIR /root/
COPY --from=build /go/bin/gofumpt /gofumpt

# Set runner as a non existent user
USER 65534:65534
ENTRYPOINT [ "/gofumpt" ]
CMD ["--help"]

Then built the image:

docker build -t gofumpt .

Running it like in the example bellow works:

docker run -it --rm -v $(pwd):/root gofumpt -w -l .

Thanks for the trouble!

[mvdan]

I was finally able to reproduce by formatting 10k tiny Go files. The problem does not appear to be there with gofmt, so this is definitely something we're doing wrong.

Thanks for bearing with me, as this one was a bit tricky to repro.