Thread 'tokio-runtime-worker' panicked at 'SinkImpl::poll_flush called after error.'

[GuruKathryn]

Panic output below. This occurred at 107613 blocks. Full log file attached.
- BUG - tokio-runtime-worker - Sun Oct 16 2022.pdf

====================

Version: 3.0.0-dev-unknown

0: backtrace::capture::Backtrace::new
1: sp_panic_handler::set::{{closure}}
2: std::panicking::rust_panic_with_hook
at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/panicking.rs:702:17
3: std::panicking::begin_panic::{{closure}}
4: std::sys_common::backtrace::__rust_end_short_backtrace
5: std::panicking::begin_panic
6: <quicksink::SinkImpl<S,F,T,A,E> as futures_sink::Sink>::poll_flush
7: <multistream_select::negotiated::Negotiated as futures_io::if_std::AsyncWrite>::poll_flush
8: <libp2p_noise::io::NoiseOutput as futures_io::if_std::AsyncWrite>::poll_flush
9: multistream_select::negotiated::Negotiated::poll
10: <multistream_select::negotiated::Negotiated as futures_io::if_std::AsyncWrite>::poll_close
11: <core::future::from_generator::GenFuture as core::future::future::Future>::poll
12: ::try_poll_next
13: <futures_util::stream::try_stream::ErrInto<St,E> as futures_core::stream::Stream>::poll_next
14: <libp2p_yamux::Yamux as libp2p_core::muxing::StreamMuxer>::poll_close
15: <libp2p_core::either::EitherOutput<A,B> as libp2p_core::muxing::StreamMuxer>::poll_close
16: <libp2p_core::muxing::boxed::Wrap as libp2p_core::muxing::StreamMuxer>::poll_close
17: <futures_util::future::poll_fn::PollFn as core::future::future::Future>::poll
18: std::panicking::try
19: <futures_util::future::select::Select<A,B> as core::future::future::Future>::poll
20: <core::future::from_generator::GenFuture as core::future::future::Future>::poll
21: tokio::loom::std::unsafe_cell::UnsafeCell::with_mut
22: tokio::runtime::task::core::CoreStage::poll
23: tokio::runtime::task::harness::poll_future
24: tokio::runtime::task::harness::Harness<T,S>::poll
25: std::thread::local::LocalKey::with
26: tokio::runtime::thread_pool::worker::Context::run_task
27: tokio::runtime::thread_pool::worker::Context::run
28: tokio::macros::scoped_tls::ScopedKey::set
29: tokio::runtime::thread_pool::worker::run
30: <tokio::runtime::blocking::task::BlockingTask as core::future::future::Future>::poll
31: tokio::runtime::task::
- BUG - tokio-runtime-worker - Sun Oct 16 2022.pdf
core::CoreStage::poll
32: tokio::runtime::task::harness::Harness<T,S>::poll
33: tokio::runtime::blocking::pool::Inner::run
34: std::sys_common::backtrace::__rust_begin_short_backtrace
35: core::ops::function::FnOnce::call_once{{vtable.shim}}
36: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once
at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/alloc/src/boxed.rs:1951:9
<alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once
at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/alloc/src/boxed.rs:1951:9
std::sys::unix::thread::Thread::new::thread_start
at /rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/sys/unix/thread.rs:108:17
37: __pthread_deallocate

Thread 'tokio-runtime-worker' panicked at 'SinkImpl::poll_flush called after error.', /Users/thomasmessegee/.cargo/registry/src/github.com-1ecc6299db9ec823/quicksink-0.1.2/src/lib.rs:225

This is a bug. Please report it at:

https://github.com/paritytech/substrate/issues/new

[bkchr]

What Substrate version are you using? Could you try latest master?

It would also be nice to if you could write on what you have done.

[GuruKathryn]

We are using 0.9.26 (babe/grandpa). We played with 0.9.30 but hit some errors compiling. We will try that version again soon. We have not attempted to correct the bug. This is the only time we have seen this error. We have run dozens of node experiments with various configurations (all on this same 0.9.26) before and after this event, without seeing this bug. It was completely new and has not surfaced again since.

[bkchr]

Yeah, I think this is the same problem as here: paritytech/polkadot#5639

Running cargo update -p yamux should fix it.

[bkchr]

I'm going to close this for now, if the above "solution" doesn't work, please reopen!

[GuruKathryn]

Roger that! Thanks!