Add Pause exporting endpoint in gateway

[deepthidevaki]

We must pause exporting while taking backup. See doc .

Currently pause exporting is exposed in broker . But we would like to expose this in the gateway so that there is a single point of entry for all cluster management operations. See RebalancingService as reference.

Goal:

• An endpoint to pause exporting on all partitions. User sends one request to this endpoint to pause exporting on all partitions. Gateway must distributed request to all brokers and coordinate them.
• If the operation succeeds, then it must be guaranteed that no records are exported until resume exporting is invoked. This must be guaranteed even after a restart or a change of leader.
• If the operation fails, then one or more partitions may not have paused the exporting. In this case, the user must retry the operation.
• If the operation fails, a few partitions may have paused exporting. To clean it up, users should call resume exporting explicitly.

Decide the request and response format before working on this issue.

[lenaschoenburg]

@deepthidevaki One concern I have with this is that there are edge cases where new brokers join the cluster or brokers lose data & restart. In those cases the pause might have succeeded but a new broker will not know about this and continue exporting. I'm wondering if we need a new way to persist the pause state that survives data loss.

[deepthidevaki]

Those are valid points. But let's keep it simple now. New brokers joining is currently not a supported feature. We could also treat data loss as a special case and ignore for it now. Users can always add additional steps to cancel backup process while any of the above cases occurs.

Also I'm hoping that we can get rid of the need to pause eventually and be able to take backups without pausing. So it may not be worth the effort to make it perfect. We could always revisit the approach if it is absolutely required.

[lenaschoenburg]

There are two cases that are problematic:

1. When the pause is issued, one or more brokers are unavailable. The gateway would need to recognize this and fail the pause. This is a bit unfortunate but doable, I think.
2. When the pause is issued, all brokers are available and actually pause exporting. The gateway considers the pause as successful. Shortly after, one broker dies, loses data and is brought up again. It'll start without the exporterPaused flag and might start exporting. This is unlikely because it'll probably not immediately become leader after a data loss but it can happen.

I currently see no way to prevent the second case.

[deepthidevaki]

Case 1 must be handled. We can decide what is the semantics of failure. The simplest would be to leave the cluster in a partial state - some brokers have paused the exporting, while others not and let users deal with it. If we want to make it "atomic", it would require more work or a different approach.

I would argue that we can ignore case 2 (data loss) now.

[lenaschoenburg]

I would argue that we can ignore case 2 (data loss) now.

I sort of disagree here. For the user (either human or some backup service) it's probably difficult to tell whether data loss has happened.

Although I suppose we could query the exported position after pause and before resuming. If the positions have changed, the backup can be considered invalid.

[deepthidevaki]

For the user (either human or some backup service) it's probably difficult to tell whether data loss has happened.

The data loss shouldn't be treated like other failures like node-restart or network partition. We can guarantee recovery only in some cases. For example:- any number of nodes can be restarted and the cluster will recover without user intervention. The worse thing that can happen is system is not available for a short time. But, we can guarantee the cluster can recover from data loss only if data loss happens in < quorum replicas. So far this must be ensure by the user and will be following a special process (eg;- as part of maintenance replacing hardware). In that sense the "user" must be aware of when a data loss happens. Even if this process is automated, it wouldn't be difficult to add necessary action to abort/retry ongoing backups.

[deepthidevaki]

That said, if you have some idea how we can handle data loss scenario, we could discuss it. But as I said, I'm not sure if it is worth the effort.

Also I'm hoping that we can get rid of the need to pause eventually and be able to take backups without pausing. So it may not be worth the effort to make it perfect. We could always revisit the approach if it is absolutely required.

[lenaschoenburg]

That said, if you have some idea how we can handle data loss scenario, we could discuss it.

First thing that came to mind was to move the pause flag to ExportersState.
But I understand your point that we don't consider data loss like any other failure and that it is a valid choice to leave it to the user to handle this.

[deepthidevaki]

First thing that came to mind was to move the pause flag to ExportersState.

This would be lost after the data loss as well, no?

[lenaschoenburg]

Yes, although eventually it'd become part of the snapshot & thus be available again after replication. But since it's not part of the log, it's not enough 😞. We had this idea floating around where exporters are also RecordProcessors, that'd make it easy to read/write pause and resume commands to the log stream, I suppose.

For now I'll follow your opinion and simply leave everything as is - data loss must be detected & handled by the user.

[lenaschoenburg]

I've found one more issue that doesn't involve data loss: Not all brokers are members for all partitions. After a restart, the set of partitions that a broker is a member of can change.

That means that we need to pause processing "globally", on a per broker basis and can't re-use the existing pause flag which can only be set for partitions that the broker is a member of.

[deepthidevaki]

I've found one more issue that doesn't involve data loss: Not all brokers are members for all partitions. After a restart, the set of partitions that a broker is a member of can change.

When would this happen? Isn't the configuration static?

[lenaschoenburg]

When would this happen? Isn't the configuration static?

Oh, you are right! I thought it might change dynamically but RoundRobinPartitionDistributor seems to get a static list of members and partitions so the distribution should always be the same. Same for FixedPartitionDistributor of course, the only other PartitionDistributor. My bad!

[lenaschoenburg]

With that sorted, I think I have a plan how to implement this:

• Gateway needs a way to send command to a (broker, partition) pair. So far it can only send to leaders of a partition or to arbitrary members of a partition, see BrokerClient.
• Gateway needs to know about all brokers and partitions, even unavailable ones. That information seems to be not yet available to the gateway, it usually uses the BrokerTopologyManager which only reports the current topology. Gateway knows about all brokers and partitions via BrokerTopologyManager.
• Brokers receives the AdminRequest and pauses exporting for that partition. If pausing fails, ErrorResponseWriter is used to respond with an error, otherwise an empty AdminResponse is sent.
• Gateway actuator
  • Add a new actuator at the gateway, let's say POST /exporting/pause.
  • Gateway sends a AdminRequest with a new type PAUSE_EXPORTING for each (broker, partition) pair.
  • If all requests finish successfully, gateway responds with a success status code, on failure or time out a failure status code is returned.

We could then extend the actuator to make retrying more efficient:

1. Respond with a list of (broker, partition) pairs where pausing has failed or timed out.
2. Support pausing for a given list of (broker, partition) pairs.

[deepthidevaki]

Sounds good 👍

3. Add a new actuator at the gateway, let's say POST /exporting/pause.

What would be in the response to the user?

[lenaschoenburg]

What would be in the response to the user?

I thought we could start with just success/failure via the HTTP status code, let's say 200 OK for success and 503 or 504 for failure. As an extension:

We could then extend the actuator to make retrying more efficient:

1. Respond with a list of (broker, partition) pairs where pausing has failed or timed out.
2. Support pausing for a given list of (broker, partition) pairs.

I haven't thought about a json schema for those yet.

[lenaschoenburg]

I'm shelving this until we triaged #9996, maybe writing the actual actuator would be easier that way 🙏