Upgrade tokio to 1.22.0 and replace async-std with tokio #12646

dmitry-markin · 2022-11-08T10:56:51Z

Resolves #12499.

polkadot companion: paritytech/polkadot#6262

cumulus companion: paritytech/cumulus#1917

dmitry-markin · 2022-11-08T11:17:36Z

Doesn't work yet, because the tests rely on async-std being used. Converted into draft.

altonen

Overall LGTM but I have two questions:

is there a reason why async_std::spawn in tests is not converted to tokio::spawn?
the example code in rust-libp2p repository instructs to configure executor for the swarm in tokio chat example. Should we configure the default executor similarly in case params.executor is None in service.rs?

I'm paranoid about the second bullet because I burned two weeks debugging bizarre async execution issues after having configured libp2p incorrectly.

koute · 2022-11-10T04:09:49Z

Nice! 👍

We should completely nuke async-std so that it isn't used anywhere in our codebases. Doesn't really make sense to have two different executors, especially when tokio's solid.

…d-libp2p-primitives

bin/node/cli/tests/websocket_server.rs

dmitry-markin · 2022-11-18T11:17:20Z

the example code in rust-libp2p repository instructs to configure executor for the swarm in tokio chat example. Should we configure the default executor similarly in case params.executor is None in service.rs?

I'm paranoid about the second bullet because I burned two weeks debugging bizarre async execution issues after having configured libp2p incorrectly.

Interestingly, the PR libp2p/rust-libp2p#3097 has just landed in libp2p that makes the specification of executor explicit during the construction of Swarm: SwarmBuilder::new has been replaced with SwarmBuilder::with_executor + some alternative constructors with predifined executors for tokio, async-std, etc.

I'm going to change the API of NetworkWorker::new so it won't accept None executor anymore in this PR, and later when libp2p-0.50 is released we can move to the new SwarmBuilder::with_executor constructor.

CC @altonen

…d-libp2p-primitives

dmitry-markin · 2022-11-27T18:07:28Z

I suspect that I have converted some futures::executor::block_on into runtime.block_on where it wasn't strictly necessary. I also don't like that in some tests we create tokio runtime explicitly and split sync initialization and async execution, and in others we just run everything under #[tokio::test]. So, may be a follow-up PR/commits are needed to clean these things up @altonen @bkchr.

melekes · 2022-11-28T05:25:52Z

libp2p 0.50 was released last week. here is the PR that updates substrate: #12734

let me know if you want me to change executors to tokio.

melekes

👍

melekes · 2022-11-28T07:50:19Z

client/beefy/src/tests.rs

 	use sc_block_builder::BlockBuilderProvider;
 	use sc_client_api::BlockBackend;

 	sp_tracing::try_init_simple();

-	let mut net = BeefyTestNet::new(2);
+	let runtime = Runtime::new().unwrap();


dummy question: can't we just use #[tokio::test]?

We can. I think, historically some tests were synchronous and later converted to async ones with manual runtime instantiation, and I just repeated the mistake and did the same in other places. Probably this doesn't make sense, so I'm going to convert all these tests to #[tokio::test] in a follow-up PR.

client/network/src/config.rs

altonen · 2022-11-28T16:21:27Z

client/network/src/service/tests/service.rs

@@ -87,7 +93,7 @@ fn notifications_state_consistent() {
 		);
 	}

-	async_std::task::block_on(async move {
+	runtime.block_on(async move {


I hope we can get rid of all these block_on calls in the future in some follow-up PR

…d-libp2p-primitives

dmitry-markin · 2022-12-05T08:18:36Z

bot merge

…12646) * Replace deprecated libp2p feature specs with correct ones * Bump tokio to 1.21.2 * Replace async-std libp2p primitives with tokio ones * minor: rustfmt * Fix TestNet to run initialization in the tokio context * Convert telemetry test from async-std to tokio * Convert notifications tests from async-std to tokio * Convert chain sync tests from async-std to tokio * Ditch async-std completely * Make executor mandatory * Bump tokio to 1.22.0 * minor: rustfmt * Explicitly use tokio runtime in tests * Move more tests to explicit tokio runtime * Explicitly set multithreaded runtime in tokio test * minor: rustfmt * minor: fix comment * Replace async-std with tokio in MMR tests

dmitry-markin added 3 commits November 8, 2022 12:19

Replace deprecated libp2p feature specs with correct ones

7b8d8c5

Bump tokio to 1.21.2

ad2fd7e

Replace async-std libp2p primitives with tokio ones

9e7398c

dmitry-markin requested a review from altonen November 8, 2022 10:56

dmitry-markin requested review from acatangiu and andresilva as code owners November 8, 2022 10:56

minor: rustfmt

20eaa28

dmitry-markin marked this pull request as draft November 8, 2022 11:16

altonen reviewed Nov 8, 2022

View reviewed changes

Merge remote-tracking branch 'origin/master' into dm-upgrade-tokio-an…

6c2028f

…d-libp2p-primitives

dmitry-markin mentioned this pull request Nov 10, 2022

Upgrade tokio to 1.22.0 paritytech/polkadot#6262

Merged

dmitry-markin added A3-in_progress Pull request is in progress. No review needed at this stage. and removed A0-please_review Pull request needs code review. labels Nov 10, 2022

dmitry-markin added 4 commits November 15, 2022 13:42

Fix TestNet to run initialization in the tokio context

1a1539a

Convert telemetry test from async-std to tokio

3140fa8

Convert notifications tests from async-std to tokio

e73c6fe

Convert chain sync tests from async-std to tokio

064c7fa

bkchr reviewed Nov 15, 2022

View reviewed changes

bin/node/cli/tests/websocket_server.rs Outdated Show resolved Hide resolved

Ditch async-std completely

398d10c

dmitry-markin added 4 commits November 24, 2022 16:57

Merge remote-tracking branch 'origin/master' into dm-upgrade-tokio-an…

fe95866

…d-libp2p-primitives

Make executor mandatory

bee686d

Bump tokio to 1.22.0

449de4f

minor: rustfmt

0b2d272

dmitry-markin added 3 commits November 27, 2022 17:37

Explicitly set multithreaded runtime in tokio test

b6f4713

minor: rustfmt

a51414d

Merge remote-tracking branch 'origin/master' into dm-upgrade-tokio-an…

3f7e116

…d-libp2p-primitives

dmitry-markin mentioned this pull request Nov 27, 2022

Replace async-std with tokio in PVF subsystem paritytech/polkadot#6353

Closed

dmitry-markin changed the title ~~Upgrade tokio and replace async-std libp2p primitives with tokio ones~~ Upgrade tokio to 1.22.0 and replace async-std with tokio Nov 27, 2022

dmitry-markin mentioned this pull request Nov 27, 2022

Upgrade tokio to 1.22.0 and replace async-std with tokio paritytech/cumulus#1917

Merged

dmitry-markin marked this pull request as ready for review November 27, 2022 18:07

github-actions bot added A0-please_review Pull request needs code review. and removed A3-in_progress Pull request is in progress. No review needed at this stage. labels Nov 27, 2022

melekes reviewed Nov 28, 2022

View reviewed changes

acatangiu approved these changes Nov 28, 2022

View reviewed changes

client/network/src/config.rs Show resolved Hide resolved

altonen approved these changes Nov 28, 2022

View reviewed changes

dmitry-markin added 3 commits November 29, 2022 23:21

minor: fix comment

8ce510c

Merge remote-tracking branch 'origin/master' into dm-upgrade-tokio-an…

8184a9a

…d-libp2p-primitives

Replace async-std with tokio in MMR tests

3e53487

paritytech-processbot bot merged commit 2bde8c1 into master Dec 5, 2022

paritytech-processbot bot deleted the dm-upgrade-tokio-and-libp2p-primitives branch December 5, 2022 08:18

dmitry-markin mentioned this pull request Dec 5, 2022

upgrade libp2p to 0.50.0 #12734

Merged

joao-paulo-parity mentioned this pull request Dec 5, 2022

Rename some crates for publishing to crates.io #12837

Merged

dmitry-markin mentioned this pull request Dec 5, 2022

Extract syncing protocol from sc-network #12828

Merged

dmitry-markin linked an issue Dec 7, 2022 that may be closed by this pull request

Try switch sockets to tokio #5762

Closed

dmitry-markin mentioned this pull request Dec 7, 2022

Try switch sockets to tokio #5762

Closed

dmitry-markin mentioned this pull request Dec 15, 2022

Clean up tests from runtime.block_on() and moving around Runtime and Handle #12941

Merged

AurevoirXavier mentioned this pull request Jan 10, 2023

To polkadot-v0.9.36 darwinia-network/darwinia-2.0#193

Closed

kacperzuk-neti mentioned this pull request Jan 23, 2023

Update to polkadot 0.9.34 liberland/liberland_substrate#225

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade tokio to 1.22.0 and replace async-std with tokio #12646

Upgrade tokio to 1.22.0 and replace async-std with tokio #12646

dmitry-markin commented Nov 8, 2022 •

edited

dmitry-markin commented Nov 8, 2022 •

edited

altonen left a comment

koute commented Nov 10, 2022

dmitry-markin commented Nov 18, 2022

dmitry-markin commented Nov 27, 2022

melekes commented Nov 28, 2022

melekes left a comment

melekes Nov 28, 2022

dmitry-markin Nov 29, 2022

altonen Nov 28, 2022

dmitry-markin commented Dec 5, 2022

Upgrade tokio to 1.22.0 and replace async-std with tokio #12646

Upgrade tokio to 1.22.0 and replace async-std with tokio #12646

Conversation

dmitry-markin commented Nov 8, 2022 • edited

dmitry-markin commented Nov 8, 2022 • edited

altonen left a comment

Choose a reason for hiding this comment

koute commented Nov 10, 2022

dmitry-markin commented Nov 18, 2022

dmitry-markin commented Nov 27, 2022

melekes commented Nov 28, 2022

melekes left a comment

Choose a reason for hiding this comment

melekes Nov 28, 2022

Choose a reason for hiding this comment

dmitry-markin Nov 29, 2022

Choose a reason for hiding this comment

altonen Nov 28, 2022

Choose a reason for hiding this comment

dmitry-markin commented Dec 5, 2022

dmitry-markin commented Nov 8, 2022 •

edited

dmitry-markin commented Nov 8, 2022 •

edited