revoking / removing tasks from queue, while workers are offline

[urzbs]

Issue Description

Overview

I'm currently utilizing Celery within a distributed system to handle various tasks across remote workers. However, there are instances where these workers may be offline due to reasons such as cloud maintenance or other unforeseen circumstances. My primary system acts as a task producer, enqueuing tasks into different Celery queues.

Problem Statement

In certain scenarios, it becomes necessary to terminate or remove specific tasks from execution entirely. To achieve this, I'm employing the following method:

app.control.revoke(task_id, terminate=True)

This successfully terminates the process and marks the task state as "REVOKED".

Issue

The challenge arises when the worker associated with the task's queue is offline. In such cases, the broadcast signal sent by .revoke() is lost. Consequently, even if tasks have been revoked, upon restarting the workers, these tasks are still executed.

Attempted Solution

I've consulted the Celery documentation regarding persistent-revokes and followed the instructions outlined here: Celery Documentation - Persistent Revokes. However, the problem persists as the .state file cannot be created when there's no connection, such as when the worker is offline.

Request for Assistance

I'm seeking guidance on ensuring that revoked tasks are never executed, regardless of the worker's online/offline status. Specifically:

• Is there a method to purge a specific task from the Redis broker or mark it as "REVOKED" even when the worker is offline?
• Are there alternative strategies or configurations within Celery that can address this issue effectively?

Your insights and assistance in resolving this matter are greatly appreciated.

[alexpotv]

I'm also looking for a solution to this problem!

Right now, my best bet is to manually access the queue in Redis, and to delete the tasks from the queue. This is neither desirable nor pretty, so any alternative would be appreciated!