Epic: in-process off chain indexing

[tac0turtle]

Summary

Indexing data from a chain allows teams to build complex front ends that are not limited based on the nodes performance. We have seen data teams spend countless hours building complex systems allowing them to build front ends.

State streaming is a good step towards allowing teams to build off chain indexes. It has its limitations. State streaming is not a first class citizen forcing off chain actors to need to decode data. This leads to complex software being built.

lastly the state machine is creating countless more writes which are needed for querying. This increases the amount of io a state machine does. In order to reduce over head, create a more performant state machine it should only hold the state needed for going to the next block. Extra information for queries should be handled with a in process off chain indexer.

This epic proposes changes to the state machine and the creation of an in process off chain indexer allowing users to build more complex applications without being prohibited by maintaining complex pieces of software.

The feature should have a plugin based system allowing teams to extend the indexing functionality to create a richer schema than the default which will be offered by the cosmos sdk team.

There are a few things to be aware of. The state machine has a differentiation between deleted data and pruned data. Deleted data refers to the removal of data due to an action. Pruning of data within in the state machine refers to data that is not needed for the state machine to continue and is removed but it is useful for users to know this information later on.

Problem Definition

Indexing of state events and blocks is a complex process with countless steps needed in order to get enough information to build complex applications.

state streaming is not a first class citizen within the software forcing users to decode the data received.

the state machine is storing more data than it needs to due to queries. Reducing h to e amount of data the state machine stores allows the state machine to have less io there fore be more performant.

Work Breakdown

Phase 1:

• ADR
• User feedback
• POC

Phase 2:

• off chain plugin system to extend data being indexed.
• tbd.