Dedupe strategy

[estahn]

I would like some sort of dedupe strategy, e.g.

• Submit Task 1
• Submit Task 2
• Submit Task 1

While Task 1 is in the pool no other tasks of the same kind should be added to the queue.

[alitto]

Hey @estahn, thanks for reaching out!
I understand that could be useful under some circumstances, however, the data structure pond uses to keep track of tasks currently in the pool (a channel) is not well suited for traversal, so it would require a significant refactoring. Also, tasks submitted to the pool do not provide an identifier or key that could be used to perform the deduping, so this would require adding extra methods to the pool I think.
How would the interface for submitting tasks with an idempotency key look like?

Another options is to implement this feature with a standalone function that wraps a task and guarantees it's only executed once, something like this (oversimplified):

func Dedupe(idempotencyKey string, task func()) func() {
    return func() {

        // If task is already running, do not execute
        if (isRunning(idempotencyKey)) {
            return
        }

        // Mark task as running
        markRunning(idempotencyKey)

        // Mark task as completed
        defer markCompleted(idempotencyKey)

        // Execute task
        task()
    }
}

Which could then be used as follows:

pool.Submit(Dedupe("Task 1", fun(){ ... }));
pool.Submit(Dedupe("Task 2", fun(){ ... }));
pool.Submit(Dedupe("Task 1", fun(){ ... }));

The benefit of this approach is it works with just about any worker pool implementation, so it's not tied to a particular concurrent execution pattern. In fact, it could very well be a library on its own 🙂

Please let me know your thoughts. Have a nice day!

[estahn]

Sounds good to me.