Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce CoroutineDispatcher.limitedParallelism #2918

Merged
merged 9 commits into from Oct 12, 2021

Conversation

qwwdfsad
Copy link
Member

@qwwdfsad qwwdfsad commented Sep 6, 2021

Fixes #2919
Fixes #2943

@qwwdfsad qwwdfsad requested review from dkhalanskyjb and removed request for dkhalanskyjb September 6, 2021 16:22
@qwwdfsad
Copy link
Member Author

qwwdfsad commented Sep 7, 2021

You may notice that we have a similar class LimitingDispatcher, the implementation core of Dispatchers.IO.

I will take care of it in a separate PR along with general cleanup of "public" classes such as ExperimentalCoroutineDispatcher and the related API

@qwwdfsad qwwdfsad marked this pull request as ready for review September 7, 2021 13:13
    * Support dispatchYield
    * Fix doc
    * Short-circuit limitedParallelism(x).limitedParallelism(y) for y >= x
    * Extract Ktor-obsolete API to a separate file for backwards compatibility
    * Make Dispatchers.IO being a slice of unlimited blocking scheduler
    * Make Dispatchers.IO.limitParallelism take slices from the same internal scheduler

Fixes #2943

class DefaultDispatchersTest : TestBase() {

private /*const*/ val EXPECTED_PARALLELISM = 64
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's wrong with const?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It cannot be declared in the class

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's private and so can't leave this file anyway.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, but it obfuscates IDE search a bit by introducing two classes with the same name for IDE.

That's our default convention in tests -- declare "effectively constant" in screaming snake case until the language is fixed

next = queue.poll() ?: return
dispatch(next, true)
}
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Historical note to save a future-someone some double-checking) This class is simply moved to the Deprecated file with no changes.

}
}

private fun tryAdd(block: Runnable): Boolean {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe rename to addAndTryDispatching, or just inline this? Adding the task to the queue always succeeds, so the name is somewhat misleading.

Copy link
Member Author

@qwwdfsad qwwdfsad Sep 24, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. I've extracted the method to reduce the bytecode size due to dispatchInternal being inlined

override fun dispatch(context: CoroutineContext, block: Runnable) {
dispatchInternal(block) {
if (dispatcher.isDispatchNeeded(EmptyCoroutineContext)) {
dispatcher.dispatch(EmptyCoroutineContext, this)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is it ok not to pass this as the coroutine context here?

Copy link
Member Author

@qwwdfsad qwwdfsad Sep 24, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean context argument or this as CoroutineDispatcher context element?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general, we do not have any reasonable contracts or use-cases to context parameter, it's mostly legacy from the early versions of coroutines.

I've changed it to this for at least some consistency, but in general, we do not ensure/check/have established/test/ contract of this parameter

@qwwdfsad
Copy link
Member Author

I've also tweaked the contract for direct dispatchers and prohibited limited parallelism for Dispatchers.Unconfined but left it as is for MainCoroutineDispatcher as it is always backed by a single thread

@qwwdfsad qwwdfsad merged commit 1df0be5 into develop Oct 12, 2021
@qwwdfsad qwwdfsad deleted the sliceable-dispatcher branch October 12, 2021 11:40
@qwwdfsad
Copy link
Member Author

Thanks to @1zaman and @denis-bezrukov for an extra review and valuable comments!

yorickhenning pushed a commit to yorickhenning/kotlinx.coroutines that referenced this pull request Oct 14, 2021
….IO unbounded for limited parallelism (Kotlin#2918)

* Introduce CoroutineDispatcher.limitedParallelism for granular concurrency control

* Elastic Dispatchers.IO:

    * Extract Ktor-obsolete API to a separate file for backwards compatibility
    * Make Dispatchers.IO being a slice of unlimited blocking scheduler
    * Make Dispatchers.IO.limitParallelism take slices from the same internal scheduler

Fixes Kotlin#2943
Fixes Kotlin#2919
pablobaxter pushed a commit to pablobaxter/kotlinx.coroutines that referenced this pull request Sep 14, 2022
….IO unbounded for limited parallelism (Kotlin#2918)

* Introduce CoroutineDispatcher.limitedParallelism for granular concurrency control

* Elastic Dispatchers.IO:

    * Extract Ktor-obsolete API to a separate file for backwards compatibility
    * Make Dispatchers.IO being a slice of unlimited blocking scheduler
    * Make Dispatchers.IO.limitParallelism take slices from the same internal scheduler

Fixes Kotlin#2943
Fixes Kotlin#2919
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants