Just a side note: I think this is both a great initiative 👍 , and at the same time a tricky task as Travis instances are known for rather wild fluctuations in throughput.

(See also travis-ci/travis-ci#352 -- not sure if the problem is still relevant to the same extent as of today as the thread is a bit old; the thread discussion has some interesting ideas btw)

That's true. We'd probably have to spin up our own boxes.

@njsmith has kindly pointed me to this article https://pythonspeed.com/articles/consistent-benchmarking-in-ci/ .
I did some cursory experiments, and (after getting past initial gotchas) it looked promising.

We could establish a handful of basic metrics with a relatively high precision, and add one or more CI gates to track them; using either sheer instruction counts as reported by valgrind, or adjusted scores as per https://github.com/pythonspeed/cachegrind-benchmarking .

We could probably start with CPython and CPython+Cython checks. PyPy JIT was not fluctuating that much as I feared either, but our microoptimizations are largely geared towards CPython anyway. PyPy checks could be added as a further improvement.

The sqlalchemy tests contains something along these lines, focused only on function call counts.

I can point to the relevant bits if it may be useful