About the problem that the performance of http3 is lower than that of http2.

[masx200]

My browser is Chrome102.
my os is windows 11.
my cpu is intel i5-6200u .

caddy version 2.5.1

https://github.com/librespeed/speedtest-go

speedtest-go version 1.1.5

When I used the caddy reverse proxy speedtest-go service, I found that the performance of using http3 was less than half that of http2.

https://www.rfc-editor.org/rfc/rfc9114.html

[masx200]

{
	debug
	local_certs
	auto_https disable_redirects
	email xxxxxxxxxxxxxxxxxxxxxx

	acme_dns xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

	servers {
		protocol {
			experimental_http3
		}
	}
}

https://xxxxxxxxxx:xxxx {
	tls {
		issuer acme
	}
	encode zstd gzip
	header {
		x-xss-protection 1;mode=block
		Content-Security-Policy upgrade-insecure-requests

		Strict-Transport-Security max-age=31536000

		X-Content-Type-Options nosniff

		X-Frame-Options DENY

		Referrer-Policy no-referrer-when-downgrade
	}
	reverse_proxy https://xxxxxxxx:xxxx {
		transport http {
			tls_insecure_skip_verify
			tls_server_name xxxxxxxxxxxxxxxxxxxx
		}

		trusted_proxies private_ranges
	}
}

[bt90]

Could you add more details? Is the backend running on the same host as caddy? From where did you execute the test? What kind of OS/hardware is involved?

[mholt]

This isn't particularly surprising. H3 is mostly only faster for high latency links. Otherwise it's more complex in general.

I think http3 performance issues should be reported upstream though, since we just use the library: https://github.com/lucas-clemente/quic-go

[masx200]

This may be related to the fact that http3 consumes more cpu resources.

[masx200]

I have both backend services caddy and speedtest-go running on the same computer. my os is windows 11.
my cpu is intel i5-6200u .

My client is running on a Snapdragon 865 Android 12 phone, tested in LAN, using Chrome 102.

The average speed was 510mbps when http2 was used and 230mbps when http3 was used.

[mholt]

If you want to collect a profile, that would be helpful. We can't do much by guessing what might be slow. It could be a protocol issue, or it could be some inefficiencies in the library we're using. But again, I don't think that's particularly surprising as HTTP/3 only barely became standardized recently, and has only been a draft protocol. Optimizations are probably a lower priority.

[otbutz]

@marten-seemann is this something ticket worthy for quic-go or are those results to be expected?

[marten-seemann]

I have both backend services caddy and speedtest-go running on the same computer.

This is exactly the same issue that has been opened 100s of times in quic-go. Localhost performance is not at all indicative of real-world performance. I'd close without further action needed.

[otbutz]

Localhost performance is not at all indicative of real-world performance.

i agree but OP stated that his client is not on the same host:

My client is running on a Snapdragon 865 Android 12 phone, tested in LAN, using Chrome 102.

So we end up with a very typical reverse proxy setup IMHO:

service <-- (HTTP/1.1 via localhost) --> caddy <-- (HTTP/3 via LAN) --> client

[masx200]

@otbutz

service <-- (HTTP/2 via localhost) --> caddy <-- (HTTP/3 via LAN) --> client

[mholt]

@otbutz Are you sure? @masx200 also says:

I have both backend services caddy and speedtest-go running on the same computer.

In any case, I do kind of agree with Marten here. HTTP/3 is known to use more CPU to reduce latency on the wire. LANs and localhost won't have latency, so yes it's more likely to bottleneck at CPU. I think this is just documenting a known fact of the protocol rather than an issue to be fixed.

However, you're welcome to collect an actual profile sometime and maybe contribute optimizations to the HTTP/3 lib?

(Closing, but feel free to continue discussions if needed.)

[otbutz]

I think Marten's objection was more directed against pure localhost tests, since HTTP/3 has a disadvantage here with a fixed MTU, as far as I'm not mistaken. But either way, it's an upstream issue if anything at all.

[zephyros-dev]

Just want to add that my previous speedtest with HTTP/2 almost saturate the bandwidth of Gigabit LAN cable network (~960mbps download/ 640mbps upload), but with the newer HTTP/3 version it falls down to about 200Mbps for download/upload, so much more than half. After some digging I found this inside the caddy logs:

{"level":"info","ts":1664899232.3432162,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details."}

I follow the recommendation to increase the Linux buffer size, and the results improved significantly (~840Mbps download/ 440Mbps upload). My stack is like this:

service <-- rootless podman network --> caddy <-- rootless podman port 80/443 binding --> (HTTP/3 via LAN) --> client

The speed is still only about 90% of what's before, but that's as far as I'm willing to investigate for now.
Update: In the above test, I turn on firefox console to check if HTTP/3 works correctly, so it might have impact the speed. With that turn off, HTTP/3 speedtest is as good as HTTP/2 now

[mholt]

Yeah, the buffer size warning is critical if you want the best performance.

Also note that a WIP CL to the Go project will offer even more improvements to performance to UDP generally: quic-go/quic-go#3563

[bt90]

The current master might also be a bit faster with quic-go v0.29.0: quic-go/quic-go#3467 (comment)

[bt90]

@masx200 @zephyros-dev caddy 2.6.2 has been released with quic-go 0.29.1.

[mholt]

quic-go 0.30 has some significant performance improvements, it looks like.

[bt90]

Also quic-go/quic-go#3524 which is scheduled for v0.31.0

[lxhao61]

{
	debug
	local_certs
	auto_https disable_redirects
	email xxxxxxxxxxxxxxxxxxxxxx

	acme_dns xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

	servers {
		protocol {
			experimental_http3
		}
	}
}

https://xxxxxxxxxx:xxxx {
	tls {
		issuer acme
	}
	encode zstd gzip
	header {
		x-xss-protection 1;mode=block
		Content-Security-Policy upgrade-insecure-requests

		Strict-Transport-Security max-age=31536000

		X-Content-Type-Options nosniff

		X-Frame-Options DENY

		Referrer-Policy no-referrer-when-downgrade
	}
	reverse_proxy https://xxxxxxxx:xxxx {
		transport http {
			tls_insecure_skip_verify
			tls_server_name xxxxxxxxxxxxxxxxxxxx
		}

		trusted_proxies private_ranges
	}
}

Configure the reverse_proxy module in this way to support HTTP/3? Or is it supported by version 2.5.1, which is not supported by the current version?

[francislavoie]

@lxhao61 I don't understand what you're trying to say. Please elaborate.

I'm not sure I understand why you're commenting here if you want to to discuss H3 transport support in reverse_proxy, that discussion should happen in the issue you linked, #5086

[mholt]

Going to turn off replies here, as I think the issue is well-addressed by now, and it seems to be getting off-topic. Further performance reports or questions should be directed upstream where the implementation is. Thanks!