FAQ
How is Fortio different from wrk?
-
Fortio can run at fixed QPS, not just "max". The most meaningful latency data comparison is typically done at a set load, as opposed to "as fast as possible".
fortio -qps 1000will run at 1k requests per second for instance. -
Fortio can run for an exact number of calls (useful for functional metrics test), for a duration, or until interrupted.
-
Fortio is not only a load generation tool but also a server, that let's you test proxies, ingresses and other system while keeping the dependencies minimal and reliable
-
Fortio can generate grpc load in addition to optimized http1.1; and standard go tls and http2.
-
Fortio is also a go library, you can integrate it easily in your go code/tests, see this example. It doesn't have any dependencies beside go itself and grpc (for the optional grpc tests)
-
Fortio's includes powerful and battle tested stats and histogram code
-
Fortio's integrates a simple yet powerful web UI to trigger test and graph and compare results (uses Chart.js)
How is it different from httpbin?
-
Fortio server side is not as powerful and varied as httpbin in general as it is meant as a load and performance testing client and server, not a general http protocol server.
-
As mentioned in the previous question, Fortio contains a client generation and simple yet powerful web UI to trigger test and graph and compare results
-
Fortio can generate probabilistically programmed http return codes e.g "status=503:0.5,429:1.5" will return ~ 98% of normal 200 success and 0.5% of 503 and 1.5% of 429 simulated http error codes.
-
Fortio can delay responses to it's default echo handler to any go duration, and since 0.6.0, with percentage of distribution as well, e.g. "delay=250us" or "delay=1ms:10%,7ms:1%"
-
Those 2 features above can be combined in a single request.
-
Fortio has a handy text/human readable /debug endpoint echoing back all the request and more.
The default values for fortio as a load generator are set such as someone trying fortio load some.web.site doesn't actually generate overwhelming traffic: by default it only creates 8 connections and runs at 8 qps for 5 seconds, so it makes 1 request per second per connection, ie 5 requests on each connection for a total of 40.
Clearly that's not going to give you the best throughput or latency or meaningful histograms.
So for a production environment I recommend you use a lot more connections (and the higher your latency the higher the number of connections is needed to maximize throughput - remember that the maximum throughput you can get from 1 connection is 1/latency; ie if your typical latency is 50ms, you can only ever do 20qps (1/0.050) per connection). A good starting point would be 2-4x the number of cores. Or start with -c 32 at least.
To get an idea of what your system can do, try -qps 0 to test at max qps for a short period of time (-t 30s to run for 30s); then run at ~75% of that qps; e.g. if you get 10k from running at max, try -qps 7500 to get an idea of the "warm not but not crazy hot" latencies to expect for your system. Then run for a longer duration (at least 5 minutes: -t 5min). Be careful to check you don't get warnings about sleep falling behind (which means that fortio+your system can't sustain that qps divided equally across the requested number of connections).
If using the command line, don't forget to use -a to auto save the json results in the current data directory. And to use labels -labels "x y z" so the file name and the json include some description of your run. You can later run fortio report to browse graphically your results.
Please be responsible and not use Fortio across internet links you do not own or pay for. Fortio is mean for local/nearby (network wise) testing of your high performance services, not over slow internet or vpn or shared links that would get swamped when using high qps. And obviously do not "load test" services you do not own (you can use Fortio for very low qps e2e availability/reachability testing though, like -c 1 -qps 5 -t 0).
Newer features and flags are also recommended: -uniform to spread the requests in time across the connections and -nocatchup for long runs avoid ever exceeding the target -qps.
Note that if you have extremely low latency services being tested (lucky you!) then also change -r for the histogram resolution to match for instance if most of the data is around 1ms latency you'd use -r 0.0001 (1/10th ms instead of the default 1ms). Note though at these latencies, client environment, go code behavior etc will likely have an impact on any end to end latency you obtain, the default values come from a sweat spot of typical good but not exceptional service, eg with latencies in 10-100ms range.
This question is a subset of the above, you need to pass the histogram resolution that matches the range of response time your environment produces otherwise if you have a low latency it may all fall in the default 1ms bucket resolution. A rule of thumb would be to use 1/10th of you average latency, so for 1ms use -r 0.0001 (the resolution unit is in SI seconds)
sudo setcap 'cap_net_bind_service=+ep' `which fortio`
fortio server --http-port 80
Use -data-dir x where x is a writeable directory
Since 0.4.2 the go get version uses the current directory for storing/reading JSON results
Fortio is written in Go (plus some html5 and javascript for the optional UI) so you first need to Install go (golang 1.8 or later)
Fortio uses a short vanity package URLs (fortio.org/fortio) - so when you fork https://github.com/fortio/fortio (click the upper right 'Fork' button, and 'Star' it too while you are at it ;-) ) you need to place
your fork in ~/go/src/fortio.org/fortio (like it happens automatically when you do go get fortio.org/fortio)
From there you should be able to go install . and run fortio (you need to put ~/go/bin in your PATH).
Don't forget to test and follow the
other steps in the Readme Contributing
section
If you see something like
Error parsing reference: "fortio/fortio.build:v5 as build" is not a valid repository/tag
it means your docker version is too old to understand multistage builds. Please upgrade to docker CE 17.05 or newer.
Yes see github.com/fortio/fortio-demo-chart and github.com/fortio/demo-deployment that runs https://demo.fortio.org and https://debug.fortio.org/