Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fs: use chunks in fs::read_dir #5309

Merged
merged 1 commit into from Dec 28, 2022
Merged

fs: use chunks in fs::read_dir #5309

merged 1 commit into from Dec 28, 2022

Conversation

icedrocket
Copy link
Contributor

@icedrocket icedrocket commented Dec 23, 2022

Motivation

When using fs::read_dir, spawn_blocking is called too often, resulting in poor performance.

Solution

Use chunks to reduce frequent spawn_blocking calls and improve performance.

I implemented walkdir using tokio and benchmarked it, and it was 3 to 5 times faster than the existing implementation.

Related Issues: #3664

@Darksonn Darksonn added A-tokio Area: The main tokio crate M-fs Module: tokio/fs labels Dec 23, 2022
@icedrocket icedrocket changed the title fs: use a buffer in fs::read_dir fs: use chunks in fs::read_dir Dec 24, 2022
@icedrocket
Copy link
Contributor Author

Ready for review. Just ignore the force pushes above.

Copy link
Contributor

@Darksonn Darksonn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks reasonable enough.

The CI issue isn't your fault. You don't have to do anything about it.

tokio/src/fs/read_dir.rs Outdated Show resolved Hide resolved
tokio/src/fs/read_dir.rs Outdated Show resolved Hide resolved
@icedrocket
Copy link
Contributor Author

icedrocket commented Dec 27, 2022

The fmt job failed because I forgot to run rustfmt. Sorry for the frequent force-pushes.

Copy link
Contributor

@Darksonn Darksonn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@Darksonn Darksonn merged commit 4a4f80c into tokio-rs:master Dec 28, 2022
crapStone pushed a commit to Calciumdibromid/CaBr2 that referenced this pull request Jan 8, 2023
This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [tokio](https://tokio.rs) ([source](https://github.com/tokio-rs/tokio)) | dependencies | minor | `1.23.0` -> `1.24.1` |
| [tokio](https://tokio.rs) ([source](https://github.com/tokio-rs/tokio)) | dev-dependencies | minor | `1.23.0` -> `1.24.1` |

---

### Release Notes

<details>
<summary>tokio-rs/tokio</summary>

### [`v1.24.1`](https://github.com/tokio-rs/tokio/releases/tag/tokio-1.24.1): Tokio v1.24.1

[Compare Source](tokio-rs/tokio@tokio-1.24.0...tokio-1.24.1)

This release fixes a compilation failure on targets without `AtomicU64` when using rustc older than 1.63. ([#&#8203;5356])

[#&#8203;5356]: tokio-rs/tokio#5356

### [`v1.24.0`](https://github.com/tokio-rs/tokio/releases/tag/tokio-1.24.0): Tokio v1.24.0

[Compare Source](tokio-rs/tokio@tokio-1.23.1...tokio-1.24.0)

The highlight of this release is the reduction of lock contention for all I/O operations ([#&#8203;5300](tokio-rs/tokio#5300)). We have received reports of up to a 20% improvement in CPU utilization and increased throughput for real-world I/O heavy applications.

##### Fixed

-   rt: improve native `AtomicU64` support detection ([#&#8203;5284])

##### Added

-   rt: add configuration option for max number of I/O events polled from the OS
    per tick ([#&#8203;5186])
-   rt: add an environment variable for configuring the default number of worker
    threads per runtime instance ([#&#8203;4250])

##### Changed

-   sync: reduce MPSC channel stack usage ([#&#8203;5294])
-   io: reduce lock contention in I/O operations  ([#&#8203;5300])
-   fs: speed up `read_dir()` by chunking operations ([#&#8203;5309])
-   rt: use internal `ThreadId` implementation ([#&#8203;5329])
-   test: don't auto-advance time when a `spawn_blocking` task is running ([#&#8203;5115])

[#&#8203;5186]: tokio-rs/tokio#5186

[#&#8203;5294]: tokio-rs/tokio#5294

[#&#8203;5284]: tokio-rs/tokio#5284

[#&#8203;4250]: tokio-rs/tokio#4250

[#&#8203;5300]: tokio-rs/tokio#5300

[#&#8203;5329]: tokio-rs/tokio#5329

[#&#8203;5115]: tokio-rs/tokio#5115

[#&#8203;5309]: tokio-rs/tokio#5309

### [`v1.23.1`](https://github.com/tokio-rs/tokio/releases/tag/tokio-1.23.1): Tokio v1.23.1

[Compare Source](tokio-rs/tokio@tokio-1.23.0...tokio-1.23.1)

This release forward ports changes from 1.18.4.

##### Fixed

-   net: fix Windows named pipe server builder to maintain option when toggling
    pipe mode ([#&#8203;5336]).

[#&#8203;5336]: tokio-rs/tokio#5336

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about these updates again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNC44NC4xIiwidXBkYXRlZEluVmVyIjoiMzQuODQuMSJ9-->

Co-authored-by: cabr2-bot <cabr2.help@gmail.com>
Reviewed-on: https://codeberg.org/Calciumdibromid/CaBr2/pulls/1703
Reviewed-by: crapStone <crapstone@noreply.codeberg.org>
Co-authored-by: Calciumdibromid Bot <cabr2_bot@noreply.codeberg.org>
Co-committed-by: Calciumdibromid Bot <cabr2_bot@noreply.codeberg.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-tokio Area: The main tokio crate M-fs Module: tokio/fs
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants