Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

优先级队列支持 #59

Open
choleraehyq opened this issue Nov 18, 2019 · 12 comments
Open

优先级队列支持 #59

choleraehyq opened this issue Nov 18, 2019 · 12 comments
Assignees
Labels
proposal Proposal for this repo waiting for response waiting for the response from commenter
Milestone

Comments

@choleraehyq
Copy link
Contributor

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
优先级调度。比如除了 Submit 之外,提供 SubmitUrge 接口之类的,提供多优先级调度功能

Describe the solution you'd like
A clear and concise description of what you want to happen.
最简单的办法,可以做三个队列,最高的是 urgent,最低的是 delayed,中间的就是现在的队列。调度策略可以开成 interface 方便定制,简单的策略可以类似于内核 io 调度的 deadline 调度器。

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
只能做多个 pool,但是也不好实现多个 pool 之间的优先级调度。

Additional context
Add any other context or screenshots about the feature request here.
确实有高优先级调度的需求

@choleraehyq choleraehyq added the proposal Proposal for this repo label Nov 18, 2019
@choleraehyq
Copy link
Contributor Author

/cc @panjf2000

@panjf2000
Copy link
Owner

panjf2000 commented Nov 18, 2019

有点像 #55 #56 ,按你的设想,优先队列无非就是每次先尝试从高优先队列取 worker,没有的话是等待还是降级到低一级队列去取?按照 ants 现在的架构,引入优先队列没有意义,因为不管是从高优先队列还是普通队列里取出 worker,都是直接去提交运行了,并没有优先的概念。

还有,为什么你说的用多个 pool 满足不了你的需求?为什么一定要在一个 pool 里实现优先调度?

@panjf2000
Copy link
Owner

我猜你想说的是在 worker struct 里加优先的队列吧?worker 每次先尝试从优先 channel 里取任务执行?

@panjf2000 panjf2000 added the waiting for response waiting for the response from commenter label Nov 18, 2019
@choleraehyq
Copy link
Contributor Author

你说的这些都是实现问题了。从功能上来说,就是希望当池子被用满的时候,高优先级队列可以被确保插队来执行。现在我理解是不能的吧。现在如果池子已经满了,有一些请求阻塞在 Submit 这里,怎么保证高优任务优先执行呢

@choleraehyq
Copy link
Contributor Author

多个 pool 就有很多管理难度,我就需要监控多个 pool,参数里也要调整多个 pool 的大小,而且高优队列可能处于长期闲置状态,在平时浪费资源。

@panjf2000
Copy link
Owner

  1. 关于优先级任务的性质:我还是觉得这个和 是否可以添加SubmitQueue类似的接口 #55 非阻塞在PoolOverload时进入等待队列 #56 一样偏向于业务需求,本质上来讲不属于 goroutine pool 的职责范围;
  2. 关于优先级任务的实现:就算最后要做这个功能,我也不赞同通过引入多个队列来实现,这种做法太粗糙且会极大增加代码复杂度(多个队列多个锁),我目前想到的比较合理的方案是利用 Priority Queue 来实现,现在 ants 的 workers 已经抽象成接口了,可以通过实现这个接口来完成。

@choleraehyq
Copy link
Contributor Author

  1. 在外部实现,这很难。一个比较好的优先级调度功能,肯定是侵入里面的。试想一下,你想基于 go scheduler 实现一个自己的优先级调度功能,怎么实现?很难实现,只能侵入进去。
    至于说业务逻辑不业务逻辑,linux 调度器是有优先级调度的,erlang 调度器也是有优先级调度的,可能他们都是为了实现一些业务逻辑罢。
  2. 实现上可以再考虑,我说的多个队列这种只是借用了 deadline 调度算法的实现,实际上可以开一个接口出来,比如 linux io 调度器的接口是 https://elixir.bootlin.com/linux/latest/source/kernel/sched/sched.h#L1684,仅供参考。

@choleraehyq
Copy link
Contributor Author

priority queue 太简单了,很容易就饿死了。

@changkun
Copy link

changkun commented Dec 2, 2019

也许这个对你有帮助:https://pkg.go.dev/github.com/changkun/sched/simsched?tab=doc#Trigger

@panjf2000 panjf2000 added this to the v2.3 milestone Dec 21, 2019
@tmpbook
Copy link

tmpbook commented Jun 25, 2020

这么复杂的场景直接用消息队列不香吗?

@choleraehyq
Copy link
Contributor Author

@tmpbook 这跟消息队列没啥关系

@panjf2000 panjf2000 modified the milestones: v2.3, v3 May 23, 2021
@ItsFunny
Copy link

ItsFunny commented Jul 3, 2021

priority queue 实现是完全可以接受的,调度算法本质上都大同小异,自己做降级不就ok了

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
proposal Proposal for this repo waiting for response waiting for the response from commenter
Projects
None yet
Development

No branches or pull requests

5 participants