scrubber: add separate find/purge garbage commands

[jcsp]

Problem

The previous garbage cleanup functionality relied on doing a dry run, inspecting logs, and then doing a deletion. This isn't ideal, because what one actually deletes might not be the same as what one saw in the dry run. It's also risky UX to rely on presence/absence of one CLI flag to control deletion: ideally the deletion command should be totally separate from the one that scans the bucket.

Related: #5037

Summary of changes

This is a major re-work of the code, which results in a net decrease in line count of about 600. The old code for removing garbage was build around the idea of doing discovery and purging together: a "delete_batch_producer" sent batches into a deleter. The new code writes out both procedures separately, in functions that use the async streams introduced in #5176 to achieve fast concurrent access to S3 while retaining the readability of a single function.

• Add find-garbage, which writes out a JSON file of tenants/timelines to purge
• Add purge-garbage which consumes the garbage JSON file, applies some extra validations, and does deletions.
• The purge command will refuse to execute if the garbage file indicates that only garbage was found: this guards against classes of bugs where the scrubber might incorrectly deem everything garbage.
• The purge command defaults to only deleting tenants that were found in "deleted" state in the control plane. This guards against the risk that using the wrong console API endpoint could cause all tenants to appear to be missing.

Outstanding work for a future PR:

• Make whatever changes are needed to adapt to the Console/Control Plane separation.
• Make purge even safer by checking S3 Modified times for index_part.json files (not doing this here, because it will depend on the generation-aware changes for finding index_part.json files)

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

2340 tests run: 2225 passed, 0 failed, 115 skipped (full report)

---

Flaky tests (1)

Postgres 16

• test_timeline_deletion_with_files_stuck_in_upload_queue: debug

Code coverage (full report)

• functions: 53.3% (8606 of 16148 functions)
• lines: 81.5% (49950 of 61269 lines)

---

_{The comment gets automatically updated with the latest test results
8de421d at 2023-10-26T11:12:50.815Z :recycle:}