Add pageserver component perf tests

[bojanserafimov]

Motivation:
We're currently only testing the pageserver as a part of a zenith cluster with 1 writer node and 0 reader nodes. But we want our work on the storage format to be able to support our future use cases without another major redesign, so we should test how it performs on workloads that will be required of it in the near future, like get_page_at_lsn for non-latest pages.

The scope of this issue is to create a minimal test suite for measuring get_page performance in isolation. That would help with perf work on the following issues:

• Improve heuristics for compaction #1357
• Take page images in pageserver #1142
• maybe Increase WAL redo speed #1339 (maybe not but it's a relevant issue)

I have some preliminary results, will share below.

[bojanserafimov]

Initial experiment: I wrote 10k updates to the same row that contains only an integer. Then I forced checkpoint and ran get_page for 0.1% of the relevant LSNs, where an update to that page was made. On the plot above, we have:
x: LSN
y: get_page runtime in microseconds

This test was ran on my laptop with release build, on top of the heikki-kvstore branch. To reproduce, here's my wip branch.

takeaways:

• We can easily get 140ms latency for a get_page request. Not good, but fixable
• Latency grows linearly with number of updates, and there's room for more updates before pageserver takes an image
• NOTE: This test doesn't stress disk lookup time, since all the relevant wal is in the latest layer. But the test suite is capable of testing this, and that's what I'm planning to try next.

non-takeaways:

• Average redo time was 1.4 microseconds, but I'm only measuring one kind of wal entry, so don't take this number seriously at all.

[bojanserafimov]

Not needed for 0.3 milestone. After some thinking I realized that it's not clear how probable it is that we'll ever have two compute nodes share a pageserver. A read node in a different region will require a local pageserver, since the safekeeper-pageserver connection is the only one that can afford latency. We might only need random LSN reads in a few niche cases:

1. Read only repllica for stricter isolation of readers for security. Useful for sharing data across organizations
2. Time travel queries
3. ???

[bojanserafimov]

Not needed for 0.3 milestone. After some thinking I realized that it's not clear how probable it is that we'll ever have two compute nodes share a pageserver. A read node in a different region will require a local pageserver, since the safekeeper-pageserver connection is the only one that can afford latency. We might only need random LSN reads in a few niche cases:

1. Read only repllica for stricter isolation of readers for security. Useful for sharing data across organizations

2. Time travel queries

3. ???

Resuming work on this issue. Random historical queries are maybe not important, so I will only focus on more immediately impactful tests:

1. get_page performance for latest pages
2. get_page performance from recent wal

[bojanserafimov]

I tested get_page performance for the latest page for each key. To initialize some pageserver state, I used three different workloads:

workload 1: pgbench small

pgbench -s5 -i
pgbench -c1 -t5000

results

Total pages: 9742
Fastest: 19.925µs
Median: 44.113µs
99th percentile: 169.875µs
Slowest: 6.017257ms

workload 2: pgbench big

pgbench -s100 -i
pgbench -c1 -t100000

results

Total pages: 193366
Fastest: 17.465µs
Median: 33.979µs
99th percentile: 125.403µs
Slowest: 19.226091ms

workload 3: pgbench big and long

pgbench -s100 -i
pgbench -c1 -t1000000

results

Total pages: 200609
Fastest: 20.031µs
Median: 45.195µs
99th percentile: 103.583µs
Slowest: 38.300479ms

workload 4: 100k updates to tiny table

Total pages: 27
Fastest: 29.431µs
Median: 45.56µs
99th percentile: 865.437539ms
Slowest: 865.437539ms

Not sure how the cache is configured, and how that might obscure results I'd get on a bigger database. Now working on getting this test merged into main so others can play with it

Clarification: I'm not measuring the get_page latency that the compute node experienced. I'm measuring the get_page latency of sending direct pagestream api requests to the pageserver after pgbench is done. I'm sending one request at a time.

[jcsp]

This is stale, the perf testing story has moved on.