You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently as much as possible, we depend on upstream system's properties to ensure our exactly once processing.
Take mysql / PG for instance, we need to depend on their WAL log.
If we provide our own WAL, we don't depend on external systems for the exactly once processing part. This reduces complexity and increases maintainability.
For DML, we can also provide the exactly once processing. Currently after running DML, it may not get committed immediately. If recovery happens, we lose the records.
The text was updated successfully, but these errors were encountered:
I just realized that log store seems not a good choice. Log store, as part of the Hummock, relies on the 1-time-per-second checkpoints to commit (i.e. make sure it's persisted and won't be lost in any way). While, in this case, we want to commit the DML changes as fast as possible, the 1-second commit latency sounds too long.
Then it comes back to the very early discussion - shall we set up a Kafka before RW to hold the DML requests? That is, when the frontend node accepts DML statements, it writes it into Kafka and return OK to users as long as the Kafka producer acknowledges.
Take mysql / PG for instance, we need to depend on their WAL log.
If we provide our own WAL, we don't depend on external systems for the exactly once processing part. This reduces complexity and increases maintainability.
Suppose we have our own WAL, but we are still the consumer of upstream WAL which means we still need to depend on the WAL in external system. For example, when a recovery occurs, we still need to reset the upstream offset and resume the consumption .
Currently as much as possible, we depend on upstream system's properties to ensure our
exactly once
processing.Take mysql / PG for instance, we need to depend on their WAL log.
If we provide our own WAL, we don't depend on external systems for the
exactly once
processing part. This reduces complexity and increases maintainability.For DML, we can also provide the
exactly once
processing. Currently after running DML, it may not get committed immediately. If recovery happens, we lose the records.The text was updated successfully, but these errors were encountered: