fs: use chunks in fs::read_dir
#5309
Merged
+73
−18
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
icedrocket
changed the title
fs::read_dirfs::read_dir
|
Ready for review. Just ignore the force pushes above. |
Darksonn
reviewed
Dec 27, 2022
Darksonn
reviewed
Dec 27, 2022
|
The fmt job failed because I forgot to run rustfmt. Sorry for the frequent force-pushes. |
crapStone
pushed a commit
to Calciumdibromid/CaBr2
that referenced
this pull request
Jan 8, 2023
This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [tokio](https://tokio.rs) ([source](https://github.com/tokio-rs/tokio)) | dependencies | minor | `1.23.0` -> `1.24.1` | | [tokio](https://tokio.rs) ([source](https://github.com/tokio-rs/tokio)) | dev-dependencies | minor | `1.23.0` -> `1.24.1` | --- ### Release Notes <details> <summary>tokio-rs/tokio</summary> ### [`v1.24.1`](https://github.com/tokio-rs/tokio/releases/tag/tokio-1.24.1): Tokio v1.24.1 [Compare Source](tokio-rs/tokio@tokio-1.24.0...tokio-1.24.1) This release fixes a compilation failure on targets without `AtomicU64` when using rustc older than 1.63. ([#​5356]) [#​5356]: tokio-rs/tokio#5356 ### [`v1.24.0`](https://github.com/tokio-rs/tokio/releases/tag/tokio-1.24.0): Tokio v1.24.0 [Compare Source](tokio-rs/tokio@tokio-1.23.1...tokio-1.24.0) The highlight of this release is the reduction of lock contention for all I/O operations ([#​5300](tokio-rs/tokio#5300)). We have received reports of up to a 20% improvement in CPU utilization and increased throughput for real-world I/O heavy applications. ##### Fixed - rt: improve native `AtomicU64` support detection ([#​5284]) ##### Added - rt: add configuration option for max number of I/O events polled from the OS per tick ([#​5186]) - rt: add an environment variable for configuring the default number of worker threads per runtime instance ([#​4250]) ##### Changed - sync: reduce MPSC channel stack usage ([#​5294]) - io: reduce lock contention in I/O operations ([#​5300]) - fs: speed up `read_dir()` by chunking operations ([#​5309]) - rt: use internal `ThreadId` implementation ([#​5329]) - test: don't auto-advance time when a `spawn_blocking` task is running ([#​5115]) [#​5186]: tokio-rs/tokio#5186 [#​5294]: tokio-rs/tokio#5294 [#​5284]: tokio-rs/tokio#5284 [#​4250]: tokio-rs/tokio#4250 [#​5300]: tokio-rs/tokio#5300 [#​5329]: tokio-rs/tokio#5329 [#​5115]: tokio-rs/tokio#5115 [#​5309]: tokio-rs/tokio#5309 ### [`v1.23.1`](https://github.com/tokio-rs/tokio/releases/tag/tokio-1.23.1): Tokio v1.23.1 [Compare Source](tokio-rs/tokio@tokio-1.23.0...tokio-1.23.1) This release forward ports changes from 1.18.4. ##### Fixed - net: fix Windows named pipe server builder to maintain option when toggling pipe mode ([#​5336]). [#​5336]: tokio-rs/tokio#5336 </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNC44NC4xIiwidXBkYXRlZEluVmVyIjoiMzQuODQuMSJ9--> Co-authored-by: cabr2-bot <cabr2.help@gmail.com> Reviewed-on: https://codeberg.org/Calciumdibromid/CaBr2/pulls/1703 Reviewed-by: crapStone <crapstone@noreply.codeberg.org> Co-authored-by: Calciumdibromid Bot <cabr2_bot@noreply.codeberg.org> Co-committed-by: Calciumdibromid Bot <cabr2_bot@noreply.codeberg.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
When using
fs::read_dir,spawn_blockingis called too often, resulting in poor performance.Solution
Use chunks to reduce frequent
spawn_blockingcalls and improve performance.I implemented
walkdirusing tokio and benchmarked it, and it was 3 to 5 times faster than the existing implementation.Related Issues: #3664