supplied instant is later than self

[AaronKutch]

I have had several events in the past where neon_evm panics because of system clock fluctuations causing a panic in duration_since. However, on neonlabsorg/evm_loader:v0.15.1 I am now consistently getting this panic.

2023-02-03 15:27:20 thread 'sol-sub-notif-0' panicked at 'supplied instant is later than self', library/std/src/time.rs:313:48
2023-02-03 15:27:20 stack backtrace:
2023-02-03 15:27:20 [2023-02-03T21:27:20.908908282Z DEBUG solana_runtime::message_processor::stable_log] Program Vote111111111111111111111111111111111111111 invoke [1]
2023-02-03 15:27:20 [2023-02-03T21:27:20.908971823Z DEBUG solana_runtime::message_processor::stable_log] Program Vote111111111111111111111111111111111111111 success
2023-02-03 15:27:20    0: rust_begin_unwind
2023-02-03 15:27:20              at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/std/src/panicking.rs:498:5
2023-02-03 15:27:20    1: core::panicking::panic_fmt
2023-02-03 15:27:20              at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/core/src/panicking.rs:116:14
2023-02-03 15:27:20    2: core::panicking::panic_display
2023-02-03 15:27:20              at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/core/src/panicking.rs:72:5
2023-02-03 15:27:20    3: core::panicking::panic_str
2023-02-03 15:27:20              at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/core/src/panicking.rs:56:5
2023-02-03 15:27:20    4: core::option::expect_failed
2023-02-03 15:27:20              at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/core/src/option.rs:1817:5
2023-02-03 15:27:20    5: core::option::Option<T>::expect
2023-02-03 15:27:20              at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/core/src/option.rs:692:21
2023-02-03 15:27:20    6: std::time::Instant::duration_since
2023-02-03 15:27:20              at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/std/src/time.rs:313:9
2023-02-03 15:27:20    7: <std::time::Instant as core::ops::arith::Sub>::sub
2023-02-03 15:27:20              at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/std/src/time.rs:441:9
2023-02-03 15:27:20    8: std::time::Instant::elapsed
2023-02-03 15:27:20              at /rustc/9d1b2106e23b1abd32fce1f17267604a5102f57a/library/std/src/time.rs:380:9
2023-02-03 15:27:20    9: solana_rpc::rpc_subscriptions::RpcSubscriptions::process_notifications
2023-02-03 15:27:20   10: rayon_core::thread_pool::ThreadPool::install::{{closure}}
2023-02-03 15:27:20   11: std::panicking::try
2023-02-03 15:27:20   12: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute
2023-02-03 15:27:20   13: rayon_core::registry::WorkerThread::wait_until_cold
2023-02-03 15:27:20   14: rayon_core::registry::ThreadBuilder::run
2023-02-03 15:27:20 note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
2023-02-03 15:27:21 + abort
2023-02-03 15:27:21 + set +e
2023-02-03 15:27:21 + kill 72 73
2023-02-03 15:27:21 ./solana-run.sh: line 93: kill: (73) - No such process
2023-02-03 15:27:21 + wait 73

This was an issue in Rust (rust-lang/rust#86470) that should have been fixed as of Rust 1.60 and tokio 1.17 (tokio-rs/tokio#4461). I think that neon evm must be getting compiled with an old version of Rust/tokio/hyper. Can you cargo update all the dependencies and make sure CIs are using the latest stable Rust? I also use cargo-tree to make sure there are not multiple versions of the same dependency inflating binary sizes

[afalaleev]

Solana uses the custom version of Rust compiler - https://github.com/solana-labs/rust - which supports compiling to BPF bytecode.
What version of the Solana node do use?
The problem is linked to the Solana node, not to Neon EVM.

[AaronKutch]

Ok thanks, I didn't know that existed.

[AaronKutch]

I think my problem was fixed by #881