Benchmark of tiny-http and may-minihttp

[kolbma]

I've compared tiny-http with may_minihttp, which is one of the fastest benchmark competitor TechEmpower Benchmark.
This is not doable with the tiny-http from official master branch. If you hit the thread/file limit of your system the Server would simply exit, because of the return of None by Server::incoming_requests().

I've used my advanced branch with some changes hopefully going in version 0.13: https://github.com/kolbma/tiny-http/tree/dev-20240117

For may is to tell, that the server and benchmark isn't using much std lib implementations.
My tiny checks also HTTP headers and body if the request makes sense and doesn't trust it. While may is the most basic, just the protocol works if the client is doing correct requests.
Next there is an optimized garbage collector, optimized collections, a probably faster_ json serializer_ than the famous serde, async and a lot more why it is there on the top of benchmarks.
My purpose wasn't to compete with it, but to compare the difference if you would use tiny the usual productive way.

The result in percent is based on the assumption that may would be 100%.

Starting with plaintext benchmark

may:

wrk -c 1024 -d 60s -t 20 http://127.0.0.1:8081/plaintext
Running 1m test @ http://127.0.0.1:8081/plaintext
  20 threads and 1024 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     2.56ms    3.29ms 147.45ms   85.09%
    Req/Sec    23.54k     3.88k   50.39k    62.85%
  28116447 requests in 1.00m, 3.30GB read
  Socket errors: connect 19, read 0, write 0, timeout 0
Requests/sec: 467851.36
Transfer/sec:     56.22MB

tiny:

wrk -c 1024 -d 60s -t 20 http://127.0.0.1:8082/plaintext
Running 1m test @ http://127.0.0.1:8082/plaintext
  20 threads and 1024 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   392.35us  319.98us  13.38ms   87.44%
    Req/Sec    27.86k    11.16k   54.56k    60.00%
  18316965 requests in 1.00m, 2.15GB read
  Socket errors: connect 19, read 0, write 0, timeout 0
Requests/sec: 304775.54
Transfer/sec:     36.62MB

1024 connections hit my local system limits. But nevertheless equal socket errors and tiny reaches 65%.

may:

wrk -c 512 -d 60s -t 20 http://127.0.0.1:8081/plaintext
Running 1m test @ http://127.0.0.1:8081/plaintext
  20 threads and 512 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.76ms    2.49ms  29.95ms   85.64%
    Req/Sec    23.66k     3.67k   47.99k    60.61%
  28259218 requests in 1.00m, 3.32GB read
Requests/sec: 470270.88
Transfer/sec:     56.51MB

tiny:

wrk -c 512 -d 60s -t 20 http://127.0.0.1:8082/plaintext
Running 1m test @ http://127.0.0.1:8082/plaintext
  20 threads and 512 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   408.08us  337.34us  12.30ms   88.22%
    Req/Sec    25.85k    13.34k   53.95k    56.79%
  18544694 requests in 1.00m, 2.18GB read
Requests/sec: 308567.70
Transfer/sec:     37.08MB

Result: 65.6%

may:

wrk -c 100 -d 60s -t 20 http://127.0.0.1:8081/plaintext
Running 1m test @ http://127.0.0.1:8081/plaintext
  20 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   245.67us  249.73us  11.78ms   87.50%
    Req/Sec    20.60k     2.18k   43.65k    66.23%
  24605451 requests in 1.00m, 2.89GB read
Requests/sec: 409402.22
Transfer/sec:     49.19MB

tiny:

wrk -c 100 -d 60s -t 20 http://127.0.0.1:8082/plaintext
Running 1m test @ http://127.0.0.1:8082/plaintext
  20 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   214.35us  105.97us   7.30ms   74.66%
    Req/Sec    16.33k     1.58k   24.32k    68.75%
  19530421 requests in 1.00m, 2.29GB read
Requests/sec: 324965.30
Transfer/sec:     39.05MB

Result: 79.4%

The 100 connections are the half of the connection limit of my tiny (default 200).
And you see an approach ;-)

Now the json benchmark

may:

wrk -c 1024 -d 60s -t 20 http://127.0.0.1:8081/json
Running 1m test @ http://127.0.0.1:8081/json
  20 threads and 1024 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     2.53ms    3.08ms  57.37ms   84.23%
    Req/Sec    23.65k     4.01k   44.72k    62.08%
  28245418 requests in 1.00m, 3.84GB read
  Socket errors: connect 19, read 0, write 0, timeout 0
Requests/sec: 469926.30
Transfer/sec:     65.43MB

tiny:

wrk -c 1024 -d 60s -t 20 http://127.0.0.1:8082/json
Running 1m test @ http://127.0.0.1:8082/json
  20 threads and 1024 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   397.46us  317.50us   9.34ms   87.44%
    Req/Sec    28.39k    13.54k   54.54k    52.74%
  18663299 requests in 1.00m, 2.54GB read
  Socket errors: connect 19, read 0, write 0, timeout 0
Requests/sec: 310537.54
Transfer/sec:     43.24MB

Result: 66%

may:

wrk -c 512 -d 60s -t 20 http://127.0.0.1:8081/json
Running 1m test @ http://127.0.0.1:8081/json
  20 threads and 512 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.76ms    2.50ms  33.72ms   85.67%
    Req/Sec    23.98k     3.70k   48.99k    61.39%
  28631117 requests in 1.00m, 3.89GB read
Requests/sec: 476398.07
Transfer/sec:     66.33MB

tiny:

wrk -c 512 -d 60s -t 20 http://127.0.0.1:8082/json
Running 1m test @ http://127.0.0.1:8082/json
  20 threads and 512 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   408.58us  341.11us  11.84ms   88.85%
    Req/Sec    25.77k    10.42k   54.45k    68.90%
  18485024 requests in 1.00m, 2.51GB read
Requests/sec: 307573.11
Transfer/sec:     42.83MB

Result: 64.6%

may:

wrk -c 100 -d 60s -t 20 http://127.0.0.1:8081/json
Running 1m test @ http://127.0.0.1:8081/json
  20 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   239.13us  230.59us   7.60ms   86.61%
    Req/Sec    20.90k     2.15k   26.49k    64.14%
  24994039 requests in 1.00m, 3.40GB read
Requests/sec: 415865.99
Transfer/sec:     57.90MB

tiny:

wrk -c 100 -d 60s -t 20 http://127.0.0.1:8082/json
Running 1m test @ http://127.0.0.1:8082/json
  20 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   234.28us  182.06us   7.47ms   92.47%
    Req/Sec    16.04k     1.57k   42.88k    70.99%
  19161659 requests in 1.00m, 2.61GB read
Requests/sec: 318827.51
Transfer/sec:     44.39MB

Result: 76.7%

As last run I've tried 6 connections with 6 threads, something my Laptop should drive parallel without many thread mixing...

may:

wrk -c 6 -d 60s -t 6 http://127.0.0.1:8081/json
Running 1m test @ http://127.0.0.1:8081/json
  6 threads and 6 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    31.65us   40.87us   5.77ms   99.94%
    Req/Sec    27.04k     1.01k   28.98k    69.19%
  9697662 requests in 1.00m, 1.32GB read
Requests/sec: 161359.98
Transfer/sec:     22.47MB

tiny:

wrk -c 6 -d 60s -t 6 http://127.0.0.1:8082/json
Running 1m test @ http://127.0.0.1:8082/json
  6 threads and 6 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    43.45us    3.81us 168.00us   76.50%
    Req/Sec    20.21k   579.70    21.25k    55.21%
  7248615 requests in 1.00m, 0.99GB read
Requests/sec: 120609.20
Transfer/sec:     16.79MB

Result: 74.7% (for plaintext 74%, too)

Conclusion

Before bench-marking I had no idea how tiny-http might compare to other widely use HTTP servers.
Now I know, it is ok.

Of course the results are not perfect, but with 60 seconds runs it should be ok to do the benchmark on a laptop with applications in the background.

If you have a look at the official results, you'll see, that with results between about 65% and nearly 80% it is in the area of hyper, ntex, actix, asp-core and in front of some very famous frameworks.
And remarkable without any special benchmark optimization.
In the json results the named competitors are a step further in front. Mostly they don't use serde I think.

For you guys interested in trying the benchmarks, may-minihttp has the examples/techempower_no_db.rs and I'll push the lastest version to my techempower-bench branch.