Pbench Server RELAY protocol and intake API

[dbutenhof]

This attempts to begin to capture and formalize discussions about the Pbench Relay protocol, as a sketch of how the new Pbench Server POST /relay/{uri} API might work internally. I've been starting to think about how to refactor the existing upload code in order to accommodate both the "push" and "pull" models. This is complicated by the divergent and simplistic nature of the PoC relay code. @webbnh and I have discussed the fairly simplistic idea of segregating each upload into two distinct files which the server can pull separately: this flow leverages that idea.

1. The client must present an authentication bearer token for a Pbench Server user.
2. The client must present a "base" relay URI which allows the Pbench Server to execute two GET operations. NOTE: the relay "base" URI is opaque to the Pbench Server; it need only be complete and unique.
   • GET <base>?metadata=true returns an application/json document defining "metadata" including an MD5 hash and a filename. (And possibly additional data such as Pbench Server metadata as a JSON object.)
   • GET <base>?metadata=false returns the application/octet-stream of the tarball.

For example, the relay metadata response might look like this:

{
  "md5": "bigfrikinhexstring",
  "name": "fio_config_date",
  "upload": "2023-05-01T23:29:59+00:00",
  "metadata": {
    "server.deletion": "2023-06-29",
    "global.project.name": "AAP"
  }
}

On activation, the Pbench Server relay API will

1. Validate that there's an authenticated user
2. Execute a GET on the base relay API with ?metadata=true. If this URI fails to respond, returns a non-success status code, or fails to yield a valid application/json payload (we may want to include a "version" and/or "type" key), the relay API will fail with a BAD_REQUEST error.
3. Confirm that a Dataset with the resource_id provided in the relay metadata does not already exist; return OK if it does.
4. Create a new Dataset with the resource_id and name from the relay metadata. NOTE: the metadata could include a timestamp for upload to the relay, which we could use as "uploaded" timestamp.
5. Create an UPLOAD operational status for the dataset as WORKING to track the progress.
6. Execute a GET on the base relay API with ?metadata=false. If this fails to respond, delete the Dataset and return a failure.
7. Read the entire response application/octet-stream payload into a temporary file, accumulating an MD5 hash and length
8. Validate the hash against the relay metadata MD5, and validate the received data length against the content-length header; if either fails to match, fail with BAD_REQUEST.
9. Publish the dataset to the cache manager.
10. Add metadata, including metadata.log (if any) and any potential metadata defined by the relay. (This might include an indication of the relay identity, or at least that the dataset came from a relay.)
11. Finalize the Dataset state, marking the UPLOAD operational status as DONE. (On failure, the Dataset is deleted.)

[webbnh]

That all sounds very good to me, however...I'd like to make a modest amendment to the proposed approach.

Rather than using a metadata query parameter, I'd like to propose that the upload URI for the dataset be included in the metadata.

Thus, the GET request to the Pbench Server would include the URI for the metadata file on the relay; the Pbench Server would issue a GET request to that URI which would generate a JSON response from the relay yielding the contents of the metadata file; upon receipt, the Pbench Server would issue a GET request for the dataset using the URI from the JSON response which would download the dataset from the relay.

This allows the relay to be totally unaware of the files that it is serving (although, I think the framework will peek at the file and guess what the Content-Type header should be), and the relay doesn't need to offer support for any query parameters relevant to the files. (I believe it would also potentially allow the use of an S3 service or similar in lieu of the relay.)

All of the "Pbenchisms" would be encapsulated (by the Agent) in the metadata file, and the relay wouldn't need to know anything about it. From the relay's perspective, there would be two files uploaded, and then those two files would be downloaded, and it wouldn't have to do or store anything out of band -- it just receives and sends individual files.

If we want to transfer something fancier than a single file (pair), we have two options. We can load up the GET request generated by the Agent for the Pbench Server with a list of targets (e.g., as one or more query parameters), or we can add multiple entries to the metadata file, each with its own dataset URI and attendant metadata (that is, in the metadata file we could have metadata which is global to the upload operation as well as metadata which is specific to each file). This also means that we aren't dependent on the having the relay offer a "here are the files that I've got" request, because the Pbench server will be pointed at one or more metadata files which contain the necessary information, rather than it needing to obtain it directly from the relay.

Just as an aside, your comment about GET <base> is very important: the URL used by the Agent to reach the relay may be very different from the URL to be used by the Pbench Server to reach the relay -- the URI portion should be the same, but the host, port, and/or protocol could be different...so, we will need to be prepared to assist the user appropriately in dealing with that.