Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Consumers Pause #1218

Open
Jarema opened this issue Feb 26, 2024 · 0 comments · May be fixed by #1234
Open

Implement Consumers Pause #1218

Jarema opened this issue Feb 26, 2024 · 0 comments · May be fixed by #1234
Labels
accepted The defect or proposal as been accepted good first issue Good for newcomers help wanted

Comments

@Jarema
Copy link
Member

Jarema commented Feb 26, 2024

Proposed change

Implement the new API for pausing consumers.

The Server PR with related change has been merged here: nats-io/nats-server#5066
Schemas PR, which is probably the best source of information how to implement the feature:
https://github.com/nats-io/jsm.go/pull/522/files
ADR issue: nats-io/nats-architecture-and-design#266

Details

Request and response schemas:

// io.nats.jetstream.api.v1.consumer_pause_request
type JSApiConsumerPauseRequest struct {
	PauseUntil time.Time `json:"pause_until,omitempty"`
}

// io.nats.jetstream.api.v1.consumer_pause_response
type JSApiConsumerPauseResponse struct {
	JSApiResponse
	Paused         bool          `json:"paused"`
	PauseUntil     time.Time     `json:"pause_until"`
	PauseRemaining time.Duration `json:"pause_remaining,omitempty"`
}

Consumer info includes 2 new fields:

Paused bool json:"paused,omitempty"
PauseRemaining time.Duration json:"pause_remaining,omitempty"

The paused state and time time would need to be persisted to the raft layer such that server restarts would not unpause paused consumers. This is done using the consumer configuration that has a new value:

PauseUntil time.Time json:"pause_until,omitempty"

If in doubt, refer to the Server/Schema PR

Feature be available in the server version 2.11
It's implementation should be hidden behind feature server_2_11.
This feature should be disabled by default, and switch to enabled by default when 2.11 is released.

Hints

Probably the easiest way to implement it, is to look how calls to methods for creating / deleting streams and consumers are done.

Use case

It is difficult to schedule maintenance on central resources on a large distributed system where 100s or 1000s of clients are accessing data in a stream.

This change allows to pause a Consumer such that it appears healthy but just doesnt deliver any messages.

During the pause maintenance can happen and resources accessed by clients will not be under constant pressure, later the stream can be unpaused and work will continue.

This happen without impacting running clients - other than they would see pending messages in stream info but not get any deliveries.

Applies to push and pull consumers.

Contribution

No response

@Jarema Jarema added proposal Enhancement idea or proposal good first issue Good for newcomers help wanted accepted The defect or proposal as been accepted help-wanted and removed proposal Enhancement idea or proposal help wanted help-wanted labels Feb 26, 2024
@yordis yordis linked a pull request Mar 14, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
accepted The defect or proposal as been accepted good first issue Good for newcomers help wanted
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant