strange latency peaks in fasthttp comparing with net/http

[maciejrozanski]

Hi,

I have a very simple production service which basically do HTTP redirects and serves ~1000 rps.
Currently deployed on GCP Cloud Run and usually working on 1 instance on 1 CPU.

On net/http it works very stable:
95th percentile always below 2ms

I've tried to move to fasthttp, there is much improvement in case of 50th percentile (from 1.5ms to 0.8 ms)

but also nondeterministically a 2-3 time a day for around 30 min strange spikes in latency happen

No warnings, no instance restart, no memory / cpu peaks, no memory leak or alloc problems. Just stops after around 30 min.
What can it be?

Newest fasthttp on 1.20.

[erikdubbelboer]

What does your fasthttp.Server struct look like? Have you tried running a profile during one of these spikes?

[maciejrozanski]

AD1. no customisation here, just:

	if err := fasthttp.ListenAndServe(":"+port, requestHandler); err != nil {
        }

AD2. Sadly profiling / debugging disabled on prod, but I can try to enable it for a short time. No repro on local env with wrk

[erikdubbelboer]

Are you able to share more code? Just based on the information I have now there is nothing I can do I'm afraid.

[maciejrozanski]

I have attached profiler to the app and results are strange. Those peaks look like GPC bug, but app in general is slow.
App is just getting GET request and calculating one HEADER in response, on localhost on single core runs with 40k req/sec. On the other hand, on GCP I need 20 instances to handle 10k req/sec. Is it anyhow normal? My business logic takes ~15% of CPU time.

[erikdubbelboer]

How are you deploying this on GCP?

[akuzia]

Experiencing same issues in Docker on premise on several machines.

Was able to reproduce localy with simple:

func main() {
	if err := fasthttp.ListenAndServe("0.0.0.0:8080", func(ctx *fasthttp.RequestCtx) {
		ctx.WriteString("ok")
	}); err != nil {
		log.Fatal(err)
	}
}

--- upd

Just to be sure benchmarked net/http, it produced similar results

--- upd 2

To rule out gc tested above fasthttp example with GOGC=off, no difference:

[erikdubbelboer]

@akuzia if you have it with fasthttp, net/http and with GC disabled then it's probably not cause by the Go process but something else in your Docker setup instead.

[akuzia]

@erikdubbelboer I've retested running compiled app natively with both vegeta and wrk, to rule out testing suite issues, results are pretty much the same. For a few requests latency spikes from ~200 microseconds to several milliseconds.

compiled on:
kernel: 6.5.11
go 1.21.4

I agree that it does not look like an issue with fasthttp, probably smth to do with system network/sockets configuration. I'd gladly accept any further leads to potential source of an issue.