chore: test more features with Miri #4397

taiki-e · 2022-01-12T14:20:05Z

~~depend on #4396.~~

Thanks to rust-lang/miri#1952, more tests can be run with Miri.

The following tests still do not work.

Tests that call epoll, kqueue, etc. This is because Miri does not support those operations.
Trait objects with receivers other than self,&self,&mut self (e.g., Pin<&mut Self>). Implement arbitrary-self dynamic method receivers rust-lang/miri#1038

taiki-e · 2022-01-12T16:00:44Z

stacked borrows violation with -Zmiri-tag-raw-pointers:

MIRIFLAGS="-Zmiri-disable-isolation -Zmiri-tag-raw-pointers" cargo miri test -p tokio --features full --lib -- runtime::tests::queue::overflow

output

error: Undefined Behavior: trying to reborrow for SharedReadWrite at alloc4551509+0x30, but parent tag <13554726> does not have an appropriate item in the borrow stack
   --> /home/runner/.rustup/toolchains/nightly-2022-01-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/non_null.rs:327:18
    |
327 |         unsafe { &*self.as_ptr() }
    |                  ^^^^^^^^^^^^^^^ trying to reborrow for SharedReadWrite at alloc4551509+0x30, but parent tag <13554726> does not have an appropriate item in the borrow stack
    |
    = help: this indicates a potential bug in the program: it performed an invalid operation, but the rules it violated are still experimental
    = help: see https://github.com/rust-lang/unsafe-code-guidelines/blob/master/wip/stacked-borrows.md for further information
            
    = note: inside `std::ptr::NonNull::<runtime::task::core::Cell<std::future::from_generator::GenFuture<[static generator@tokio/src/runtime/tests/queue.rs:29:46: 29:48]>, runtime::blocking::schedule::NoopSchedule>>::as_ref` at /home/runner/.rustup/toolchains/nightly-2022-01-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/non_null.rs:327:18
note: inside `runtime::task::harness::Harness::<std::future::from_generator::GenFuture<[static generator@tokio/src/runtime/tests/queue.rs:29:46: 29:48]>, runtime::blocking::schedule::NoopSchedule>::trailer` at tokio/src/runtime/task/harness.rs:34:19
   --> tokio/src/runtime/task/harness.rs:34:19
    |
34  |         unsafe { &self.cell.as_ref().trailer }
    |                   ^^^^^^^^^^^^^^^^^^
note: inside `runtime::task::harness::Harness::<std::future::from_generator::GenFuture<[static generator@tokio/src/runtime/tests/queue.rs:29:46: 29:48]>, runtime::blocking::schedule::NoopSchedule>::dealloc` at tokio/src/runtime/task/harness.rs:148:9
   --> tokio/src/runtime/task/harness.rs:148:9
    |
148 |         self.trailer().waker.with_mut(drop);
    |         ^^^^^^^^^^^^^^
note: inside `runtime::task::raw::dealloc::<std::future::from_generator::GenFuture<[static generator@tokio/src/runtime/tests/queue.rs:29:46: 29:48]>, runtime::blocking::schedule::NoopSchedule>` at tokio/src/runtime/task/raw.rs:118:5
   --> tokio/src/runtime/task/raw.rs:118:5
    |
118 |     harness.dealloc();
    |     ^^^^^^^^^^^^^^^^^
note: inside `runtime::task::raw::RawTask::dealloc` at tokio/src/runtime/task/raw.rs:76:13
   --> tokio/src/runtime/task/raw.rs:76:13
    |
76  |             (vtable.dealloc)(self.ptr);
    |             ^^^^^^^^^^^^^^^^^^^^^^^^^^
note: inside `<runtime::task::Task<runtime::blocking::schedule::NoopSchedule> as std::ops::Drop>::drop` at tokio/src/runtime/task/mod.rs:394:13
   --> tokio/src/runtime/task/mod.rs:394:13
    |
394 |             self.raw.dealloc();
    |             ^^^^^^^^^^^^^^^^^^
    = note: inside `std::ptr::drop_in_place::<runtime::task::Task<runtime::blocking::schedule::NoopSchedule>> - shim(Some(runtime::task::Task<runtime::blocking::schedule::NoopSchedule>))` at /home/runner/.rustup/toolchains/nightly-2022-01-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/mod.rs:188:1
    = note: inside `std::ptr::drop_in_place::<runtime::task::Notified<runtime::blocking::schedule::NoopSchedule>> - shim(Some(runtime::task::Notified<runtime::blocking::schedule::NoopSchedule>))` at /home/runner/.rustup/toolchains/nightly-2022-01-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/mod.rs:188:1
    = note: inside `std::ptr::drop_in_place::<std::option::Option<runtime::task::Notified<runtime::blocking::schedule::NoopSchedule>>> - shim(Some(std::option::Option<runtime::task::Notified<runtime::blocking::schedule::NoopSchedule>>))` at /home/runner/.rustup/toolchains/nightly-2022-01-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/mod.rs:188:1
note: inside `runtime::tests::queue::overflow` at tokio/src/runtime/tests/queue.rs:35:32
   --> tokio/src/runtime/tests/queue.rs:35:32
    |
35  |     while inject.pop().is_some() {
    |                                ^
note: inside closure at tokio/src/runtime/tests/queue.rs:24:1
   --> tokio/src/runtime/tests/queue.rs:24:1
    |
23  |   #[test]
    |   ------- in this procedural macro expansion
24  | / fn overflow() {
25  | |     let (_, mut local) = queue::local();
26  | |     let inject = Inject::new();
27  | |
...   |
43  | |     assert_eq!(n, 257);
44  | | }
    | |_^
    = note: inside `<[closure@tokio/src/runtime/tests/queue.rs:24:1: 44:2] as std::ops::FnOnce<()>>::call_once - shim` at /home/runner/.rustup/toolchains/nightly-2022-01-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227:5
    = note: inside `<fn() as std::ops::FnOnce<()>>::call_once - shim(fn())` at /home/runner/.rustup/toolchains/nightly-2022-01-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227:5
    = note: inside `test::__rust_begin_short_backtrace::<fn()>` at /home/runner/.rustup/toolchains/nightly-2022-01-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:574:5
    = note: inside closure at /home/runner/.rustup/toolchains/nightly-2022-01-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:565:30
    = note: inside `<[closure@test::run_test::{closure#1}] as std::ops::FnOnce<()>>::call_once - shim(vtable)` at /home/runner/.rustup/toolchains/nightly-2022-01-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227:5
    = note: inside `<std::boxed::Box<dyn std::ops::FnOnce() + std::marker::Send> as std::ops::FnOnce<()>>::call_once` at /home/runner/.rustup/toolchains/nightly-2022-01-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1854:9
    = note: inside `<std::panic::AssertUnwindSafe<std::boxed::Box<dyn std::ops::FnOnce() + std::marker::Send>> as std::ops::FnOnce<()>>::call_once` at /home/runner/.rustup/toolchains/nightly-2022-01-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panic/unwind_safe.rs:271:9
    = note: inside `std::panicking::r#try::do_call::<std::panic::AssertUnwindSafe<std::boxed::Box<dyn std::ops::FnOnce() + std::marker::Send>>, ()>` at /home/runner/.rustup/toolchains/nightly-2022-01-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:406:40
    = note: inside `std::panicking::r#try::<(), std::panic::AssertUnwindSafe<std::boxed::Box<dyn std::ops::FnOnce() + std::marker::Send>>>` at /home/runner/.rustup/toolchains/nightly-2022-01-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:370:19
    = note: inside `std::panic::catch_unwind::<std::panic::AssertUnwindSafe<std::boxed::Box<dyn std::ops::FnOnce() + std::marker::Send>>, ()>` at /home/runner/.rustup/toolchains/nightly-2022-01-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:133:14
    = note: inside `test::run_test_in_process` at /home/runner/.rustup/toolchains/nightly-2022-01-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:597:18
    = note: inside closure at /home/runner/.rustup/toolchains/nightly-2022-01-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:491:39
    = note: inside `test::run_test::run_test_inner` at /home/runner/.rustup/toolchains/nightly-2022-01-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:529:13
    = note: inside `test::run_test` at /home/runner/.rustup/toolchains/nightly-2022-01-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:561:28
    = note: inside `test::run_tests::<[closure@test::run_tests_console::{closure#2}]>` at /home/runner/.rustup/toolchains/nightly-2022-01-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:304:17
    = note: inside `test::run_tests_console` at /home/runner/.rustup/toolchains/nightly-2022-01-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/console.rs:290:5
    = note: inside `test::test_main` at /home/runner/.rustup/toolchains/nightly-2022-01-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:115:15
    = note: inside `test::test_main_static` at /home/runner/.rustup/toolchains/nightly-2022-01-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/lib.rs:134:5
    = note: inside `main`
    = note: inside `<fn() as std::ops::FnOnce<()>>::call_once - shim(fn())` at /home/runner/.rustup/toolchains/nightly-2022-01-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227:5
    = note: inside `std::sys_common::backtrace::__rust_begin_short_backtrace::<fn(), ()>` at /home/runner/.rustup/toolchains/nightly-2022-01-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:123:18
    = note: inside closure at /home/runner/.rustup/toolchains/nightly-2022-01-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:145:18
    = note: inside `std::ops::function::impls::<impl std::ops::FnOnce<()> for &dyn std::ops::Fn() -> i32 + std::marker::Sync + std::panic::RefUnwindSafe>::call_once` at /home/runner/.rustup/toolchains/nightly-2022-01-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:259:13
    = note: inside `std::panicking::r#try::do_call::<&dyn std::ops::Fn() -> i32 + std::marker::Sync + std::panic::RefUnwindSafe, i32>` at /home/runner/.rustup/toolchains/nightly-2022-01-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:406:40
    = note: inside `std::panicking::r#try::<i32, &dyn std::ops::Fn() -> i32 + std::marker::Sync + std::panic::RefUnwindSafe>` at /home/runner/.rustup/toolchains/nightly-2022-01-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:370:19
    = note: inside `std::panic::catch_unwind::<&dyn std::ops::Fn() -> i32 + std::marker::Sync + std::panic::RefUnwindSafe, i32>` at /home/runner/.rustup/toolchains/nightly-2022-01-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:133:14
    = note: inside closure at /home/runner/.rustup/toolchains/nightly-2022-01-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:128:48
    = note: inside `std::panicking::r#try::do_call::<[closure@std::rt::lang_start_internal::{closure#2}], isize>` at /home/runner/.rustup/toolchains/nightly-2022-01-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:406:40
    = note: inside `std::panicking::r#try::<isize, [closure@std::rt::lang_start_internal::{closure#2}]>` at /home/runner/.rustup/toolchains/nightly-2022-01-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:370:19
    = note: inside `std::panic::catch_unwind::<[closure@std::rt::lang_start_internal::{closure#2}], isize>` at /home/runner/.rustup/toolchains/nightly-2022-01-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:133:14
    = note: inside `std::rt::lang_start_internal` at /home/runner/.rustup/toolchains/nightly-2022-01-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:128:20
    = note: inside `std::rt::lang_start::<()>` at /home/runner/.rustup/toolchains/nightly-2022-01-12-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:144:17
    = note: this error originates in the attribute macro `test` (in Nightly builds, run with -Z macro-backtrace for more info)

If run without -Zmiri-tag-raw-pointers, the test succeeds.

taiki-e · 2022-01-12T18:15:37Z

~~Hopefully, I know how to fix that SB violation.~~

Nah, it seems so complicated...

Darksonn · 2022-01-28T15:59:07Z

.github/workflows/ci.yml

-        run: cargo miri test --features rt,rt-multi-thread,sync task
+        # Many of tests in tokio/tests and doctests use #[tokio::test] or
+        # #[tokio::main] that calls epoll_create1 that Miri does not support.
+        run: cargo miri test --features fs,io-util,io-std,net,parking_lot,rt,rt-multi-thread,sync --lib --no-fail-fast


This is opt-ing out of some things that use tokio::sync::Notified. Getting intrusive linked lists on the stack to pass with miri will probably be difficult.

Or maybe not .. it seems like the problem was elsewhere entirely.

Not sure how to fix the timer, but Notified works now.

Haha, I figured it out: The issue for timers was futures::executor::block_on.

The issue for timers was futures::executor::block_on.

Ah, futures has a few unreleased fixes of stacked borrows violation with -Zmiri-tag-raw-pointers: rust-lang/futures-rs#2550

hawkw

as far as my understanding of stacked borrows goes, this looks legit. i didn't notice any serious issues with these changes.

i commented on a couple of minor style nits. also, i did notice that in some cases, some safety invariants seem to have changed, now that we access all linked list pointers via pointer casts. it would be nice if this was added to the existing safety documentation.

tokio/src/runtime/tests/queue.rs

tokio/src/sync/notify.rs

tokio/src/time/driver/entry.rs

tokio/src/signal/registry.rs

tokio/src/util/slab.rs

tokio/src/runtime/task/waker.rs

# 1.16.2 (February 15, 2022) This release updates the minimum supported Rust version (MSRV) to 1.49, the `mio` dependency to v0.8, and the (optional) `parking_lot` dependency to v0.12. Additionally, it contains several bug fixes, as well as internal refactoring and performance improvements. ### Fixed - time: prevent panicking in `sleep` with large durations ([#4495]) - time: eliminate potential panics in `Instant` arithmetic on platforms where `Instant::now` is not monotonic ([#4461]) - io: fix `DuplexStream` not participating in cooperative yielding ([#4478]) - rt: fix potential double panic when dropping a `JoinHandle` ([#4430]) ### Changed - update minimum supported Rust version to 1.49 ([#4457]) - update `parking_lot` dependency to v0.12.0 ([#4459]) - update `mio` dependency to v0.8 ([#4449]) - rt: remove an unnecessary lock in the blocking pool ([#4436]) - rt: remove an unnecessary enum in the basic scheduler ([#4462]) - time: use bit manipulation instead of modulo to improve performance ([#4480]) - net: use `std::future::Ready` instead of our own `Ready` future ([#4271]) - replace deprecated `atomic::spin_loop_hint` with `hint::spin_loop` ([#4491]) - fix miri failures in intrusive linked lists ([#4397]) ### Documented - io: add an example for `tokio::process::ChildStdin` ([#4479]) ### Unstable The following changes only apply when building with `--cfg tokio_unstable`: - task: fix missing location information in `tracing` spans generated by `spawn_local` ([#4483]) - task: add `JoinSet` for managing sets of tasks ([#4335]) - metrics: fix compilation error on MIPS ([#4475]) - metrics: fix compilation error on arm32v7 ([#4453]) [#4495]: #4495 [#4461]: #4461 [#4478]: #4478 [#4430]: #4430 [#4457]: #4457 [#4459]: #4459 [#4449]: #4449 [#4462]: #4462 [#4436]: #4436 [#4480]: #4480 [#4271]: #4271 [#4491]: #4491 [#4397]: #4397 [#4479]: #4479 [#4483]: #4483 [#4335]: #4335 [#4475]: #4475 [#4453]: #4453

# 1.17.0 (February 16, 2022) This release updates the minimum supported Rust version (MSRV) to 1.49, the `mio` dependency to v0.8, and the (optional) `parking_lot` dependency to v0.12. Additionally, it contains several bug fixes, as well as internal refactoring and performance improvements. ### Fixed - time: prevent panicking in `sleep` with large durations ([#4495]) - time: eliminate potential panics in `Instant` arithmetic on platforms where `Instant::now` is not monotonic ([#4461]) - io: fix `DuplexStream` not participating in cooperative yielding ([#4478]) - rt: fix potential double panic when dropping a `JoinHandle` ([#4430]) ### Changed - update minimum supported Rust version to 1.49 ([#4457]) - update `parking_lot` dependency to v0.12.0 ([#4459]) - update `mio` dependency to v0.8 ([#4449]) - rt: remove an unnecessary lock in the blocking pool ([#4436]) - rt: remove an unnecessary enum in the basic scheduler ([#4462]) - time: use bit manipulation instead of modulo to improve performance ([#4480]) - net: use `std::future::Ready` instead of our own `Ready` future ([#4271]) - replace deprecated `atomic::spin_loop_hint` with `hint::spin_loop` ([#4491]) - fix miri failures in intrusive linked lists ([#4397]) ### Documented - io: add an example for `tokio::process::ChildStdin` ([#4479]) ### Unstable The following changes only apply when building with `--cfg tokio_unstable`: - task: fix missing location information in `tracing` spans generated by `spawn_local` ([#4483]) - task: add `JoinSet` for managing sets of tasks ([#4335]) - metrics: fix compilation error on MIPS ([#4475]) - metrics: fix compilation error on arm32v7 ([#4453]) [#4495]: #4495 [#4461]: #4461 [#4478]: #4478 [#4430]: #4430 [#4457]: #4457 [#4459]: #4459 [#4449]: #4449 [#4462]: #4462 [#4436]: #4436 [#4480]: #4480 [#4271]: #4271 [#4491]: #4491 [#4397]: #4397 [#4479]: #4479 [#4483]: #4483 [#4335]: #4335 [#4475]: #4475 [#4453]: #4453

ci: upgrade to new nightly

bd93557

taiki-e added the A-ci Area: The continuous integration setup label Jan 12, 2022

taiki-e force-pushed the taiki-e/miri branch from 78b0d66 to 07ab34c Compare January 12, 2022 14:54

chore: test more features with Miri

67e7d72

taiki-e force-pushed the taiki-e/miri branch from 07ab34c to 67e7d72 Compare January 12, 2022 15:24

disable -Zmiri-tag-raw-pointers

8734ac1

Make tests run

ddcf1a3

github-actions bot added the R-loom Run loom tests on this PR label Jan 28, 2022

Merge branch 'master' into taiki-e/miri

719ec45

Darksonn reviewed Jan 28, 2022

View reviewed changes

Darksonn added 7 commits January 28, 2022 16:00

Remove commented out code

850600f

Get Notified (and signal) to work

90a9e52

Disable test on wasm

939266e

rustfmt

78c4bba

Fix time tests

45f31d3

rustfmt

72402b6

Remove comment

50739c8

taiki-e changed the title ~~[WIP] chore: test more features with Miri~~ chore: test more features with Miri Jan 29, 2022

taiki-e marked this pull request as ready for review January 29, 2022 04:37

Darksonn requested a review from hawkw January 30, 2022 11:49

hawkw approved these changes Jan 30, 2022

View reviewed changes

Darksonn added 2 commits February 2, 2022 15:55

Address reviews

56ec538

Merge remote-tracking branch 'origin/master' into taiki-e/miri

88571b0

This was referenced Feb 4, 2022

UB in the batch_semaphore linked list #3399

Open

Unsoundness in tokio-util's LinkedList/ListNode APIs #3101

Closed

Darksonn merged commit 1be8e9d into master Feb 9, 2022

Darksonn deleted the taiki-e/miri branch February 9, 2022 10:11

hawkw mentioned this pull request Feb 15, 2022

chore: prepare Tokio v1.17.0 release #4504

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: test more features with Miri #4397

chore: test more features with Miri #4397

taiki-e commented Jan 12, 2022 •

edited

taiki-e commented Jan 12, 2022 •

edited

taiki-e commented Jan 12, 2022 •

edited

Darksonn Jan 28, 2022

Darksonn Jan 28, 2022 •

edited

Darksonn Jan 28, 2022

Darksonn Jan 28, 2022

taiki-e Jan 28, 2022

hawkw left a comment

chore: test more features with Miri #4397

chore: test more features with Miri #4397

Conversation

taiki-e commented Jan 12, 2022 • edited

taiki-e commented Jan 12, 2022 • edited

taiki-e commented Jan 12, 2022 • edited

Darksonn Jan 28, 2022

Choose a reason for hiding this comment

Darksonn Jan 28, 2022 • edited

Choose a reason for hiding this comment

Darksonn Jan 28, 2022

Choose a reason for hiding this comment

Darksonn Jan 28, 2022

Choose a reason for hiding this comment

taiki-e Jan 28, 2022

Choose a reason for hiding this comment

hawkw left a comment

Choose a reason for hiding this comment

taiki-e commented Jan 12, 2022 •

edited

taiki-e commented Jan 12, 2022 •

edited

taiki-e commented Jan 12, 2022 •

edited

Darksonn Jan 28, 2022 •

edited