You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In the current Meltano implementation, we have an is_sorted property that supports resuming streams when unexpectedly interrupted. This works well when the replication_key is ascending, typically dates. However, there are use cases that deal with descending replication_keys, and the existing configuration does not accommodate these scenarios.
Proposal
Introduce support for descending replication_keys in Meltano. This would involve adding logic similar to the existing logic for upstream replication keys. To do this, I propose to add a new boolean field is_descending similar to is_sorted, which would allow the system to handle flows sorted in a descending fashion. (or maybe we can do something to make it work with is_sorted??)
Once this is implemented, the updated code could look something like this:
What would happen when more recent records are added to the source? It's unlike the more common and currently supported reverse case where older records should not appear in the source under normal circumstances.
Or did you have a specific source or use case in mind that doesn't have that risk?
Feature scope
Other
Description
Feature Scope
Core Functionality (Stream Resumption, Replication Keys)
Description
Problem
In the current Meltano implementation, we have an
is_sorted
property that supports resuming streams when unexpectedly interrupted. This works well when thereplication_key
is ascending, typically dates. However, there are use cases that deal with descendingreplication_keys
, and the existing configuration does not accommodate these scenarios.Proposal
Introduce support for descending
replication_keys
in Meltano. This would involve adding logic similar to the existing logic for upstream replication keys. To do this, I propose to add a new boolean fieldis_descending
similar tois_sorted
, which would allow the system to handle flows sorted in a descending fashion. (or maybe we can do something to make it work with is_sorted??)Once this is implemented, the updated code could look something like this:
Let me know what you think or how we can implement this idea, or if you are already working on something for this. ✌️
The text was updated successfully, but these errors were encountered: