pageserver: use adaptive concurrency in secondary layer downloads

[jcsp]

Problem

Secondary downloads are a low priority task, and intentionally do not try to max out download speeds. This is almost always fine when they are used through the life of a tenant shard as a continuous "trickle" of background downloads.

However, there are sometimes circumstances where we would like to populate a secondary location as fast as we can, within the constraint that we don't want to impact the activity of attached tenants:

• During node removal, where we will need to create replacements for secondary locations on the node being removed
• After a shard split, we need new secondary locations for the new shards to populate before the shards can be migrated to their final location.

Summary of changes

• Add an activity() function to the remote storage interface, enabling callers to query how busy the remote storage backend is
• In the secondary download code, use a very modest amount of concurrency, driven by the remote storage's state: we only use concurrency if the remote storage semaphore is 75% free, and scale the amount of concurrency used within that range.

This is not a super clever form of prioritization, but it should accomplish the key goals:

• Enable secondary downloads to happen faster when the system is idle
• Make secondary downloads a much lower priority than attached tenants when the remote storage is busy.

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

3060 tests run: 2927 passed, 0 failed, 133 skipped (full report)

---

Code coverage* (full report)

• functions: 31.4% (6334 of 20179 functions)
• lines: 47.3% (47882 of 101230 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
539dcbf at 2024-05-13T17:48:46.323Z :recycle:}