Can we leverage go's Context package to cancel jobs with ANTS

[joncodo]

Summary

The main reason to cancel jobs in the first place is so Users can start a task that can take a long time to complete and change their mind and cancel it. EE also needs this I think.

What we are doing now

Right now, we are cancelling jobs that are running ants like this:

(Using Nats)
docker-compose.yml

nats:
    image: 'nats'
    ports:
      - "4222:4222"
    hostname: nats-server

api/task/task.go

func CancelTask(ctx *gin.Context) {
	// pull the task name from the query params
	taskName := ctx.Query("taskName")

	nc, err := nats.Connect(nats.DefaultURL)
	if err != nil {
		logger.Error("err connecting to nats server", err)
	}

	// Simple Publisher
	**err2 := nc.Publish(taskName, []byte("Hello World"))**
	if err2 != nil {
		logger.Error("err2", err2)
	}
}

And then in each plugin, we do something like this:

_, errNc := nc.Subscribe("github", func(m *nats.Msg) {
  fmt.Printf("Received a message: %s\n", string(m.Data))
  scheduler.Release()
  progress <- 1
  logger.Info("You cancelled the collector with nats", false)
  close(progress)
})

This causes us to us run a NATS service at all time and also requires the cancel listener in each plugin.

What we could do in the future

We might be able to use the Context package to cancel tasks. The main benefits of this is that we will no longer need NATS in the docker-compose file and the plugins will not have to implement the cancel method.

Unknowns:

1. Can you cancel ants with context? I know you can cancel go routines with context but ants seems to not give that option.

Research:

1. Post on SO for how to cancel processes with Context: https://stackoverflow.com/questions/59607744/how-to-kill-running-goroutines-from-outside/59608070#59608070
2. ANTS Creator saying to use Context: Cancel all task when has any task return error panjf2000/ants#169
3. ANTS Creator talking about Context: how to pass context and other args in pool_func? panjf2000/ants#81

[joncodo]

For now, since I was unable to get Context working quickly, I will go with the original solution with NATS until we have time to revisit this or @hezyin sees it as a high priority.

Implementation in : https://github.com/merico-dev/lake/pull/478

[joncodo]

@klesh If this is possible, it would make our code much cleaner but I am not able to get it to cancel ANTS with the context Cancel method. Can you take about an hour to try this out and show me your findings?

[klesh]

closed by #492