why -h2 lower have qps ?

[abasitt]

I am running fortio deployment in kubernetes with service expose via NodePort. The fortio client is running on an external machine as docker container.

fortioclient(external docker container) -------> fortio-server inside kubernetes with 6 replicas and nodeport service

When I run below.
docker run fortio/fortio load -logger-force-color -c 52 -qps 0 -t 30s -a -r 0.0001 [fd74:ca9b:3a09:868c:10:9:65:2]:30155/echo

The results are.

06:34:19.013 r1 [INF] scli.go:124> Starting, command="Φορτίο", version="1.63.3 h1:t4FoQ70znmYEEST3eyMLDqGaoNJPUDZNvoPRRH0WsaQ= go1.21.7 amd64 linux", go-max-procs=40
Fortio 1.63.3 running at 0 queries per second, 40->40 procs, for 30s: [fd74:ca9b:3a09:868c:10:9:65:2]:30155/echo
06:34:19.014 r1 [INF] httprunner.go:121> Starting http test, run=0, url="[fd74:ca9b:3a09:868c:10:9:65:2]:30155/echo", threads=52, qps="-1.0", warmup="parallel", conn-reuse=""
06:34:19.014 r1 [WRN] http_client.go:170> Assuming http:// on missing scheme for '[fd74:ca9b:3a09:868c:10:9:65:2]:30155/echo'
Starting at max qps with 52 thread(s) [gomax 40] for 30s
------
Sockets used: 52 (for perfect keepalive, would be 52)
Uniform: false, Jitter: false, Catchup allowed: true
IP addresses distribution:
[fd74:ca9b:3a09:868c:10:9:65:2]:30155: 52
Code 200 : 2330302 (100.0 %)
Response Header Sizes : count 2330302 avg 75 +/- 0 min 75 max 75 sum 174772650
Response Body/Total Sizes : count 2330302 avg 75 +/- 0 min 75 max 75 sum 174772650
All done 2330302 calls (plus 52 warmup) 0.669 ms avg, **77670.1 qps**

When i run with -h2 flag, the qps drops significantly.

docker run fortio/fortio load -logger-force-color -c 52 -qps 0 -t 30s -a -r 0.0001 -h2 [fd74:ca9b:3a09:868c:10:9:65:2]:30155/echo
06:36:01.382 r1 [INF] scli.go:124> Starting, command="Φορτίο", version="1.63.3 h1:t4FoQ70znmYEEST3eyMLDqGaoNJPUDZNvoPRRH0WsaQ= go1.21.7 amd64 linux", go-max-procs=40
Fortio 1.63.3 running at 0 queries per second, 40->40 procs, for 30s: [fd74:ca9b:3a09:868c:10:9:65:2]:30155/echo
06:36:01.385 r1 [INF] httprunner.go:121> Starting http test, run=0, url="[fd74:ca9b:3a09:868c:10:9:65:2]:30155/echo", threads=52, qps="-1.0", warmup="parallel", conn-reuse=""
06:36:01.385 r1 [INF] http_client.go:157> H2 requested, switching to std client
06:36:01.385 r1 [WRN] http_client.go:170> Assuming http:// on missing scheme for '[fd74:ca9b:3a09:868c:10:9:65:2]:30155/echo'
Starting at max qps with 52 thread(s) [gomax 40] for 30s

---

Sockets used: 52 (for perfect keepalive, would be 52)
Uniform: false, Jitter: false, Catchup allowed: true
IP addresses distribution:
[fd74:ca9b:3a09:868c:10:9:65:2]:30155: 52
Code 200 : 769958 (100.0 %)
Response Header Sizes : count 769958 avg 0 +/- 0 min 0 max 0 sum 0
Response Body/Total Sizes : count 769958 avg 0 +/- 0 min 0 max 0 sum 0
All done 769958 calls (plus 52 warmup) 2.023 ms avg, **25661.4 qps**
Successfully wrote 10226 bytes of Json data to 2024-02-19-063601_fd74_ca9b_3a09_868c_10_9_65_2_30155_echo_a598ed5ec894.json

I thought h2 should have a better qps. Any reason this drop in qps ?
I am trying to achieve 52k rps with 52 connections. Is this even achievable ? Because I do see the increase in qps if i increase -c e.g. to 200 and more.

My endgoal is to test istio proxy with 52k qps http2 traffic but first i need to make sure I am able to generate the traffic correctly. Thank you.

[ldemailly]

When using h2 it forces to use the standard go http client instead of the custom "fast" client (what H2 requested, switching to std client in the output indicates), the fast client only implements http1.1, fast client is... faster (do confirm that what you see is client bottleneck (maxing out cpu on the client))

You might need to increase the number of connections. This being said the gap seems a bit high so maybe something else is going on (in theory h2 is more efficient than http1.1 - what happens if you use http1.1 with std client for comparison?)

[abasitt]

Thank you so much for the pointer about std-client and the client-side bottleneck.
using -std-client, the results are pretty same to h2 so this explains i guess.

docker run fortio/fortio load -logger-force-color -c 52 -qps 0 -t 30s -a -r 0.0001 -stdclient [fd74:ca9b:3a09:868c:10:9:65:2]:30155/echo
00:36:17.230 r1 [INF] scli.go:124> Starting, command="Φορτίο", version="1.63.3 h1:t4FoQ70znmYEEST3eyMLDqGaoNJPUDZNvoPRRH0WsaQ= go1.21.7 amd64 linux", go-max-procs=40
Fortio 1.63.3 running at 0 queries per second, 40->40 procs, for 30s: [fd74:ca9b:3a09:868c:10:9:65:2]:30155/echo
00:36:17.231 r1 [INF] httprunner.go:121> Starting http test, run=0, url="[fd74:ca9b:3a09:868c:10:9:65:2]:30155/echo", threads=52, qps="-1.0", warmup="parallel", conn-reuse=""
00:36:17.231 r1 [WRN] http_client.go:170> Assuming http:// on missing scheme for '[fd74:ca9b:3a09:868c:10:9:65:2]:30155/echo'
Starting at max qps with 52 thread(s) [gomax 40] for 30s 

---

Sockets used: 52 (for perfect keepalive, would be 52)
Uniform: false, Jitter: false, Catchup allowed: true
IP addresses distribution:
[fd74:ca9b:3a09:868c:10:9:65:2]:30155: 52
Code 200 : 823914 (100.0 %)
Response Header Sizes : count 823914 avg 0 +/- 0 min 0 max 0 sum 0
Response Body/Total Sizes : count 823914 avg 0 +/- 0 min 0 max 0 sum 0
All done 823914 calls (plus 52 warmup) 1.890 ms avg, 27457.7 qps
Successfully wrote 10334 bytes of Json data to 2024-02-20-003617_fd74_ca9b_3a09_868c_10_9_65_2_30155_echo_6645c6ca30a2.json

increasing number of connections does help though but hard for me to replicate the actual app behavior. The CPU is fine, with above loads, am hitting 10% CPU on the node.

[ldemailly]

Glad to hear!

See also the faq, the throughput is limited always per connection by the latency (at minimum the network roundtrip if it was say 100ms you could max do 10qps per connection for instance)

[abasitt]

Thank you