KYVE Getter

[christopherbrumm]

This thread refers to the idea of having general data Getters for outsourced data retrievability described here.

The KYVE Core Team is currently working on an integration to validate and store the required data enabling support for a Getter using external data. Therefore, it's required to define which data needs to be included in the dataset in order to provide the planned functionalities.

Proposed solution:
The Getter interface implements three methods: GetShare, GetEDS and GetSharesByNamespace, that all must be filled with data by KYVE. To achieve this, we propose the following schema for the stored data:

{
  key: <height>,
  value: {
    shares: [
      {
         <row, col>: <response from share.GetShare(row, col)
      }
    ]
    eds: <response from share.GetEDS>,
    sharesByNamespace: [
      {
         <namespace>: <response from share.GetSharesByNamespace>
      }
    ]
  }
}

This would enable the Getter to execute all queries by using validated and archived data by KYVE.

Open questions:

1. Usually, queries need the header as input. But with the proposed schema, this won't work for the KYVE dataset since it only uses height as the unique identifier. This is why the KYVE Getter itself would need to use the height as input when querying certain shares. Do we need to also use the header as a unique identifier, or is height enough? If height isn't sufficient, we can simply use the header as a key in the integration.
2. For certain purposes some data consumers could want to query blobs without relying on any service that converts the available shares into the requested blob. Should this be included as well? Although it wouldn't be required for the tasks as Getter. The proposed dataset could be extended by simply adding the following to the value:

blobs: [
  {
     <namespace>: <response from blobs.GetAll>
   }
]

[christopherbrumm]

After the final commitment for a certain dataset schema, the validation and archiving will be started on the KYVE Devnet in order to start implementing the Getter itself.

[Wondertan]

This is a good starting point for the KYVE-getter implementation. By the end of this workstream, we need the KYVE getter and the infrastructure to integrate Getters generally(#3243). We can start with any, but solving general issues are more preferable at the start.

[Wondertan]

Answering questions:

This is why the KYVE Getter itself would need to use the height as input when querying certain shares. Do we need to also use the header as a unique identifier, or is height enough? If height isn't sufficient, we can simply use the header as a key in the integration.

The height is enough to identify. Only the actual implementation type KyveGetter struct will have the header as an input param in methods like GetShareByNamespace. The client implementation will then use the height from header and do necessary request to KYVE protocol pools by using the header only.

For certain purposes some data consumers could want to query blobs without relying on any service that converts the available shares into the requested blob. Should this be included as well? Although it wouldn't be required for the tasks as Getter. The proposed dataset could be extended by simply adding the following to the value:

All the data consumers must be light nodes. This is the goal and the only way we get trust-minimized access to data. The light node is exactly the converting service that converts namespace data into blobs. If everyone runs a light node, then everyone runs a converting service, so we don't need to support GetAll, but only GetSharesByNamespace.

[Wondertan]

This is why the blobs that has been validated through KYVE can be accessed in a trust-minimized way as well, which is why I proposed to store the raw blobs as well.

In this case, you propose to trust KYVE validators, but the trust minimized means no trust in validators. We need to validate things locally on your machine, which is why light nodes/clients exist in the first place.

[christopherbrumm]

Regarding the required dataset for the Getter: Beside the GetSharesByNamespace, is it required to archive the response from share.GetEDS and the shares mapped to row, col? Or can everything required be reconstructed by the GetSharesByNamespace response?

[Wondertan]

Eventually, we will need GetEDS to provide the full archival mode. For now, we can stick to a single GetSharesByNamespace and keep other methods stubbed.

Or can everything required be reconstructed by the GetSharesByNamespace response?

The GetSharesByNamespace only returns a portion of the eds related to a particular namespace, so no, you can't reconstruct everything. Everything can be constructed using GetEDS, though, as it contains the whole block/data square, but that's something for later when the full archival is needed.

[christopherbrumm]

As KYVE aims to provide full archiving with the least amount of data to store, wouldn't the more efficient approach be to use GetEDS?

We could implement both solutions, but I would be more in favor of creating the Getter using the archived GetEDS to provide all other queries. Or are there other reasons why you would start with GetSharesByNamespace?

[Wondertan]

We need to start from namespaces because its a prevailing usecase. Rollups want to archive and pay for their personal data, not the data of everyone.

Also, technically GetEds fetches the whole block, so by starting from it on the client side, we would force clients to download the whole block when they only need a little piece.