Allow clients to select a "profile" via ACME API query parameters

[aarongable]

This bug is an umbrella/tracking bug, acting as a one-stop-shop to see progress on the multiple sub-tasks necessary to achieve this 2024 OKR.

The design we have landed on is

1. the CA (and the WFE?) will be configured with multiple profiles, each of which has a human-readable name (and maybe a guaranteed-unique ID).
2. the client will specify the profile they want as a query parameter on their directory URL -- this means profile selection only requires users to update their directory url, and doesn't require client software itself to be updated -- and the parameter will be reflected back in the directory object so it gets used in the new-order request as well.
3. the SA will store the requested profile on the Order object so it can be used at finalization time

Subtasks:

• CA: Support multiple profiles identified with unique IDs #6966
• The SA can store a requested profile alongside an order #7324
• The RA can plumb a requested new-order profile to the SA
• The RA can plumb a stored or requested finalize profile to the CA #7366
• WFE: Add support for certificate profiles #7332
• Publish website documentation concerning profiles, link in user facing errors #7392
• Public announcements and documentation regarding how a client can request a profile

[orangepizza]

I'm sure it'd currently in active discussing internally, but is there conclusion about how DB should save profiles?
it needs to not replaced to some other profile even if config doesn't have that profile, but not sure if it editing of a profile in boulder config file (needs to) / (must not) retroactively change old orders made with that profile name:
and does OCSP/CRL processor needs to able to recall how old profile no longer in config file was configured, or one can just decide by how long cert last?

[aarongable]

We haven't yet come to a conclusion on how best to store the profile in the database. The two key considerations are:

1. Inside the orders table vs in a new table (having everything in one place is handy, but adding a column to an already-large table is expensive and difficult)
2. Storing a profile name vs a content-addressed profile hash ID (the former means we can use the same profile identifier everywhere including in the WFE, the latter prevents accidental issues where a profile is changed while keeping the name the same)

The OCSP and CRL systems do not need to know anything about profiles, as far as I'm aware at this time.

[orangepizza]

Wonder if it can use https://mariadb.com/kb/en/instant-add-column-for-innodb/ , if mariadb is same version with what docker image grabs it should possible.
About hash/string, I kinda tilt with string, because updating profiles something done in past like allowing new feature/ slight change backdate, and invalidate old orders (becasue CA no longer knows it) because of it sounds weird.

[aarongable]

Wonder if it can use https://mariadb.com/kb/en/instant-add-column-for-innodb/ , if mariadb is same version with what docker image grabs it should possible.

We've discussed this option in the past, and instead opted for other solutions. It is an option again this time, but only one option.

About hash/string, I kinda tilt with string, because updating profiles something done in past like allowing new feature/ slight change backdate, and invalidate old orders (becasue CA no longer knows it) because of it sounds weird.

Yes, we've modified the profile in the past. We've also had an incident because of it. So immutable profiles is definitely something we're considering.

[orangepizza]

think profile inside order is too early to be immutable, because profile update will need to happen someday (like BR, new or old not allows old one as-is), it only needs to be immutable once it enters finalization and passed to CA. WFE or RA could have hold map of old to new profile hashs and update it at the start of finalization, but now that map needs to be set by human with raw hashes and looks more error prone

[aarongable]

it only needs to be immutable once it enters finalization and passed to CA.

True, good point.

WFE or RA could have hold map of old to new profile hashs and update it at the start of finalization, but now that map needs to be set by human with raw hashes and looks more error prone

Nah, the CA could just have two maps: human-readable profile name to profile, and content-addressed profile ID to profile. At precertificate issuance time, it would use the first map to look up the correct profile based on the profile name associated with the order. It would then bake the corresponding profile ID into the issuance token. Then at final certificate issuance time, it would use that unique ID to look up the exact same profile for the final certificate. The unique content-addressed hash IDs would be computed inside the CA when it loads its config.