Introduce RecordTransmitter

[sondreso]

Issue
Resolves #1334
Resolves #1447
Resolves #1328
Resolves #1502

Approach
Introduce RecordTransmitter concept, along with two concrete implementations (shared-disk and in-memory). This allows records to be used as inputs and outputs to steps in the workflow manager, while the actual transportation method is abstracted away. This allows the transmitters (references to the records) to be transported as lightweight objects over the network from the client to the compute side, while the actual data is only pulled from storage to the compute node.

Assumptions and limitations

• No RecordTransmitter that uses the ert3 storage and/or the new^2 ert-storage yet
• We assume the record transmitter can be cloudpickled
• Validation in ert3 that all commands are specified as transportable commands is removed, as you can now use arbitrary UNIX commands
• The Record as-is does not contain all information that you need to work with it. Specifically, the name and mime of the record is almost always bundled together with the record, since you need this information to interact with the record outside pure data access.
• Currently we do: ert3 config -> ee-config -> PrefectEnsemble. We should cut the middle layer, and use the ert3 config the construct the PrefectEnsemble directly using the builders.
• This PR makes it possible to get the result from the evaluation using the monitor. Currently, this will give you a transmitter for every output in the ensemble. This is accomplished by the ensemble sending all transmitters to the evaluator as part of the ensemble finished message, and then clients can connect to a separate endpoint to get the results after a STOPPED event is received.

[jondequinor]

Terminology:

• record key: e.g. coeffs. Represents all coeffs from a user's perspective.
• concrete record: e.g. coeffs_7788cf9c-7dec-11eb-9439-0242ac130002, is a sampled coeff e.g. {"a": 2.4, … } but with a globally unique name

I'm talking about ert3/ee interfacing here:

1. Parameters and other locally sampled records are at some point Record in ert3
2. User provided records (e.g. large files/blobs in azure/storage) are also Record in ert3.
3. All Record will get an accompanying RecordTransmitter and will be transmitted() in ert3.
4. After transmit(), already existing azure/storage things will remain in storage, but locally sampled records will be saved in storage.
5. A DAG is created from the record key IO mapping
6. An accompanying io map from record key -> iens -> (RecordTransmitter) is provided
7. A RecordTransmitter will contain configuration options (location, ???)
8. The Prefect Evaluator creates an ensemble from DAG and io mapping
9. In the ensemble, input/output information is indicating that it is file-based IO for UnixSteps, memory based for FunctionStep. Possibly Stream-based for some specially configured FunctionStep.

EE/Evaluation of steps:

1. A UnixStep, for all inputs, gets some information about the IO (e.g. FileIO) and a RecordTransmitter. Usually .dump(file_io.location)
2. ???
3. Profit

[markusdregi]

Sounds like a good plan! 👏 I did not understand this one:

A RecordTransmitter will contain configuration options (location, ???)

Do you mean that the RecordTransmitter will contain configuration options for where the record should be transmitted? It is probably the location that made me think this was possibly the disk location of data produced in the unix step? Because that knowledge I think should reside within the unix step directly :)

[jondequinor]

Do you mean that the RecordTransmitter will contain configuration options for where the record should be transmitted? It is probably the location that made me think this was possibly the disk location of data produced in the unix step? Because that knowledge I think should reside within the unix step directly :)

You're right, it will be in the ensemble and the step.

[markusdregi]

Thanks for the clarification 👍

[sondreso]

We should also fix: #1502

[markusdregi]

I've gone through most of this PR except the actual implementation in ert_shared. That part will have to wait until later today or tomorrow morning, but in the meantime feel free to address and resolve my comments 👍🏻

[sondreso]

https://github.com/equinor/komodo-releases/pull/1259 Needs to be merged.

[jondequinor]

@markusdregi addressed recent batch of comments. Thanks for reviewing!

[markusdregi]

Thanks for reviewing!

Happy to do so 👍🏻 I have no further comments... Good job all of you 🚀 I suggest you rebase, squash and do a last internal review among the authors and then merge it.

[jondequinor]

lgtm