Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

backtrace (of rust code) is incomplete on esp32c6 #275

Open
t-moe opened this issue Jan 9, 2024 · 17 comments
Open

backtrace (of rust code) is incomplete on esp32c6 #275

t-moe opened this issue Jan 9, 2024 · 17 comments

Comments

@t-moe
Copy link

t-moe commented Jan 9, 2024

I'm trying to obtain a backtrace for a simple rust program when it panics.

Code:

fn c() {
    panic!("oh no");
}

fn b() {
    c();
}

fn a() {
    b();
}

fn main() {
    a();
}

I'm using the esp32c6 here, together with espflash run --monitor.
I have found that the amount of info I can get from the backtrace varies a lot when using esp-idf-sys compared to when using esp-hal.

rust --version -v:

rustc 1.77.0-nightly (e51e98dde 2023-12-31)
binary: rustc
commit-hash: e51e98dde6a60637b6a71b8105245b629ac3fe77
commit-date: 2023-12-31
host: x86_64-unknown-linux-gnu
release: 1.77.0-nightly
LLVM version: 17.0.6

esp-hal // no_std Case

Based on cargo generate esp-rs/esp-template I have the following setup.

main.rs:

#![no_std]
#![no_main]

use esp32c6_hal::prelude::*;
use esp_backtrace as _;


fn c() {
    panic!("oh no");
}

fn b() {
    c();
}

 fn a() {
    b();
}


#[entry]
fn main() -> ! {
    a();
    loop {};
}

Cargo.toml:

[package]
#....

# Template adapted to use jtag-serial instead of uart
[dependencies]
esp32c6-hal = "0.7.0"
esp-backtrace = { version = "0.10.0", features = ["esp32c6", "panic-handler", "exception-handler", "print-jtag-serial"]}
esp-println = { version = "0.8.0", features = ["esp32c6", "log", "colors", "critical-section"], default-features = false }
`.cargo/config.toml`: As generated by the template
[target.riscv32imac-unknown-none-elf]
runner = "espflash flash --monitor"


[build]
rustflags = [
  "-C", "link-arg=-Tlinkall.x",

  # Required to obtain backtraces (e.g. when using the "esp-backtrace" crate.)
  # NOTE: May negatively impact performance of produced code
  "-C", "force-frame-pointers",
]

target = "riscv32imac-unknown-none-elf"

[unstable]
build-std = ["core"]

Invocation: cargo run

espflash enhanced output:

....

!! A panic occured in 'src/main.rs', at line 9, column 5

PanicInfo {
    payload: Any { .. },
    message: Some(
        oh no,
    ),
    location: Location {
        file: "src/main.rs",
        line: 9,
        col: 5,
    },
    can_unwind: true,
    force_no_backtrace: false,
}

Backtrace:

0x42000152
0x42000152 - backtrace_nostd::b
    at /tmp/backtrace-nostd/src/main.rs:12
0x42000162
0x42000162 - backtrace_nostd::b
    at /tmp/backtrace-nostd/src/main.rs:14
0x4200017a
0x4200017a - backtrace_nostd::a
    at /tmp/backtrace-nostd/src/main.rs:18
0x42000192
0x42000192 - main
    at /tmp/backtrace-nostd/src/main.rs:23
0x42000116
0x42000116 - _start_rust
    at ??:??


esp-idf-sys/ std Case

Based on cargo generate esp-rs/esp-idf-template I have the following setup.

main.rs:

fn c() {
    panic!("oh no");
}

fn b() {
    c();
}

fn a() {
    b();
}

fn main() {
    esp_idf_svc::sys::link_patches();
    esp_idf_svc::log::EspLogger::initialize_default();
    a();
}
`Cargo.toml`: As generated by the template
[package]
#....

[profile.release]
opt-level = "s"

[profile.dev]
debug = true    # Symbols are nice and they don't increase the size on Flash
opt-level = "z"

[features]
default = ["std", "embassy", "esp-idf-svc/native"]

pio = ["esp-idf-svc/pio"]
std = ["alloc", "esp-idf-svc/binstart", "esp-idf-svc/std"]
alloc = ["esp-idf-svc/alloc"]
nightly = ["esp-idf-svc/nightly"]
experimental = ["esp-idf-svc/experimental"]
embassy = ["esp-idf-svc/embassy-sync", "esp-idf-svc/critical-section", "esp-idf-svc/embassy-time-driver"]

[dependencies]
log = { version = "0.4", default-features = false }
esp-idf-svc = { version = "0.47.3", default-features = false }

[build-dependencies]
embuild = "0.31.3"

.cargo/config.toml:

[build]
target = "riscv32imac-esp-espidf"

[target.riscv32imac-esp-espidf]
linker = "ldproxy"
runner = "espflash flash --monitor"
rustflags = ["--cfg", "espidf_time64", "-C", "default-linker-libraries",
    # Added:
    "-C", "force-frame-pointers=yes", "-C","debuginfo=2", "-C", "force-unwind-tables=yes"]

[unstable]
build-std = ["std", "panic_abort"]

[env]
MCU="esp32c6"
ESP_IDF_VERSION = "master"

esp-idf version: v5.3-dev-422-ga7fbf452fa

sdkconfig-defaults:

# Rust often needs a bit of an extra main task stack size compared to C (the default is 3K)
CONFIG_ESP_MAIN_TASK_STACK_SIZE=8000

CONFIG_ESP_SYSTEM_PANIC_PRINT_REBOOT=n
CONFIG_ESP_SYSTEM_PANIC_PRINT_HALT=y

Invocation: cargo run

raw output(serial port monitor, without esp flash):

...

thread 'main' panicked at src/main.rs:2:5:
oh no
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

abort() was called at PC 0x42029be5 on core 0
Core  0 register dump:
MEPC    : 0x40800ecc  RA      : 0x40804152  SP      : 0x4080f330  GP      : 0x408096e0  
TP      : 0x407f0a54  T0      : 0x37363534  T1      : 0x7271706f  T2      : 0x33323130  
S0/FP   : 0x4080f36c  S1      : 0x4080f36c  A0      : 0x4080f36c  A1      : 0x4080f34e  
A2      : 0x00000000  A3      : 0x4080f399  A4      : 0x4080aebc  A5      : 0x4080aec4  
A6      : 0x00000000  A7      : 0x76757473  S2      : 0x4080f350  S3      : 0x4080f488  
S4      : 0x00000001  S5      : 0x00000000  S6      : 0x4205013c  S7      : 0x00000001  
S8      : 0x00000000  S9      : 0x00000000  S10     : 0x00000000  S11     : 0x00000000  
T3      : 0x6e6d6c6b  T4      : 0x6a696867  T5      : 0x66656463  T6      : 0x62613938  
MSTATUS : 0x00001881  MTVEC   : 0x40800001  MCAUSE  : 0x00000007  MTVAL   : 0x00000000  
MHARTID : 0x00000000  

Stack memory:
4080f330: 0x00000000 0x42053c04 0x4080f36c 0x40807e80 0x00000003 0x00000000 0x00000001 0x40800030
4080f350: 0x32303234 0x35656239 0x40810100 0x40809f8c 0x4080f350 0x40809fa8 0x4080f34c 0x726f6261
4080f370: 0x20292874 0x20736177 0x6c6c6163 0x61206465 0x43502074 0x34783020 0x39323032 0x20356562
4080f390: 0x63206e6f 0x2065726f 0x00000030 0x40800000 0x42053d68 0x4080a194 0x4080f3c0 0x42029be8
4080f3b0: 0x00000001 0x4081011c 0x4080f3d0 0x42029bd8 0x40810120 0x4081011c 0x4080f410 0x4200ab2c
4080f3d0: 0x42053d68 0x4080a194 0x4080f3f0 0x4200b2d2 0x00000001 0x4081011c 0x4080f400 0x4200abe0
4080f3f0: 0x42053d68 0x4080a194 0x4080f410 0x42014af0 0x42053d68 0x4080a194 0x4080f480 0x4200aaf2
4080f410: 0x00000804 0x00000054 0x00000000 0x00000000 0x00000000 0x4080f488 0x42053d58 0x4080f4f0
4080f430: 0x4205013c 0x00000001 0x00000000 0x00000000 0x00000000 0x00000000 0x40810090 0x00000000
4080f450: 0x4080f4a8 0x4080f4d4 0x4080f4a0 0x00000000 0x00000000 0x00000000 0x00000000 0x4205013c
4080f470: 0x4080f4f0 0x4080f4d4 0x4080f4a0 0x4200a8b2 0x00000005 0x00000005 0x42050120 0x00000005
4080f490: 0x4080f554 0x4080f4d4 0x4080f4b0 0x42009f9e 0x4080ff60 0x00000005 0x4080f4d0 0x4200a6c0
4080f4b0: 0x4080f4f0 0x4080f4d4 0x4205013c 0x00000000 0x4080f554 0x4205014c 0x4080f4f0 0x4203b7e0
4080f4d0: 0x4080a260 0x420649c0 0x420649c0 0x4080f4f0 0x4205013c 0x40800001 0x4080f510 0x42000082
4080f4f0: 0x42050128 0x00000001 0x42050120 0x00000000 0x00000000 0x00000000 0x4080f520 0x4200002a
4080f510: 0x00000000 0x4205014c 0x4080f530 0x420000ea 0x4080f564 0x4080ff28 0x4080f550 0x4201d7bc
4080f530: 0x4080f564 0x00000000 0x420504ec 0x4204a500 0x00000000 0x00000000 0x4080f560 0x420000d0
4080f550: 0x420694e0 0x42000032 0x00000000 0x4204a5e4 0x00000000 0x00001388 0x00000001 0x00000000
4080f570: 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
4080f590: 0x00000000 0xa5a5a5a5 0xa5a5a5a5 0xa5a5a5a5 0xa5a5a5a5 0xa5a5a5a5 0xa5a5a5a5 0xa5a5a5a5
4080f5b0: 0x00000154 0x4080f170 0x00000000 0x4080a410 0x4080a410 0x4080f5b4 0x4080a408 0x00000018
4080f5d0: 0x00000000 0x00000000 0x4080f5b4 0x00000000 0x00000001 0x4080d470 0x6e69616d 0x00000000
4080f5f0: 0x00000000 0x00000000 0x00000000 0x4080f5a0 0x00000001 0x00000000 0x408100b4 0x4203dbb4
4080f610: 0x00000000 0x4080bcd0 0x4080bd38 0x4080bda0 0x00000000 0x00000000 0x00000001 0x00000000
4080f630: 0x00000000 0x00000000 0x42041e48 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
4080f650: 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
4080f670: 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
4080f690: 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
4080f6b0: 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
4080f6d0: 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
4080f6f0: 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000600 0xa5a5a5a5
4080f710: 0xa5a5a5a5 0xa5a5a5a5 0xa5a5a5a5 0xa5a5a5a5 0xa5a5a5a5 0xa5a5a5a5 0xa5a5a5a5 0xa5a5a5a5



ELF file SHA256: 000000000

CPU halted.

`espflash run --monitor` enhanced output: contains main, but neither a, b or c and is really verbose
thread 'main' panicked at src/main.rs:2:5:
oh no
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

abort() was called at PC 0x42029be5 on core 0
0x42029be5 - panic_abort::__rust_start_panic::abort
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/panic_abort/src/lib.rs:43
Core  0 register dump:
MEPC    : 0x40800ecc  RA      : 0x40804152  SP      : 0x4080f330  GP      : 0x408096e0  
0x40800ecc - panic_abort
    at /home/timo/esp/esp-idf/components/esp_system/panic.c:472
0x40804152 - __ubsan_include
    at /home/timo/esp/esp-idf/components/esp_system/ubsan.c:313
0x4080f330 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x408096e0 - coex_pti_tab
    at ??:??
TP      : 0x407f0a54  T0      : 0x37363534  T1      : 0x7271706f  T2      : 0x33323130  
0x407f0a54 - ??
    at ??:??
0x37363534 - ??
    at ??:??
0x7271706f - s_rtc_timer_retain_mem
    at ??:??
0x33323130 - ??
    at ??:??
S0/FP   : 0x4080f36c  S1      : 0x4080f36c  A0      : 0x4080f36c  A1      : 0x4080f34e  
0x4080f36c - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f36c - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f36c - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f34e - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
A2      : 0x00000000  A3      : 0x4080f399  A4      : 0x4080aebc  A5      : 0x4080aec4  
0x00000000 - ??
    at ??:??
0x4080f399 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080aebc - coex_schm_env
    at ??:??
0x4080aec4 - __sf
    at ??:??
A6      : 0x00000000  A7      : 0x76757473  S2      : 0x4080f350  S3      : 0x4080f488  
0x00000000 - ??
    at ??:??
0x76757473 - s_rtc_timer_retain_mem
    at ??:??
0x4080f350 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f488 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
S4      : 0x00000001  S5      : 0x00000000  S6      : 0x4205013c  S7      : 0x00000001  
0x00000001 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x4205013c - esp_app_desc
    at ??:??
0x00000001 - ??
    at ??:??
S8      : 0x00000000  S9      : 0x00000000  S10     : 0x00000000  S11     : 0x00000000  
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
T3      : 0x6e6d6c6b  T4      : 0x6a696867  T5      : 0x66656463  T6      : 0x62613938  
0x6e6d6c6b - s_rtc_timer_retain_mem
    at ??:??
0x6a696867 - s_rtc_timer_retain_mem
    at ??:??
0x66656463 - s_rtc_timer_retain_mem
    at ??:??
0x62613938 - s_rtc_timer_retain_mem
    at ??:??
MSTATUS : 0x00001881  MTVEC   : 0x40800001  MCAUSE  : 0x00000007  MTVAL   : 0x00000000  
0x00001881 - ??
    at ??:??
0x40800001 - _vector_table
    at ??:??
0x00000007 - ??
    at ??:??
0x00000000 - ??
    at ??:??
MHARTID : 0x00000000  
0x00000000 - ??
    at ??:??

Stack memory:
4080f330: 0x00000000 0x42053c04 0x4080f36c 0x40807e80 0x00000003 0x00000000 0x00000001 0x40800030
0x00000000 - ??
    at ??:??
0x42053c04 - anon.6f8d54fb548d5ff1abf9d01d3f110dc8.67.llvm.17408016719535564914
    at ??:??
0x4080f36c - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x40807e80 - __assert_func
    at /home/timo/esp/esp-idf/components/newlib/assert.c:34
0x00000003 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000001 - ??
    at ??:??
0x40800030 - _vector_table
    at ??:??
4080f350: 0x32303234 0x35656239 0x40810100 0x40809f8c 0x4080f350 0x40809fa8 0x4080f34c 0x726f6261
0x32303234 - ??
    at ??:??
0x35656239 - ??
    at ??:??
0x40810100 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x40809f8c - __func__.0
    at ??:??
0x4080f350 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x40809fa8 - __func__.0
    at ??:??
0x4080f34c - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x726f6261 - s_rtc_timer_retain_mem
    at ??:??
4080f370: 0x20292874 0x20736177 0x6c6c6163 0x61206465 0x43502074 0x34783020 0x39323032 0x20356562
0x20292874 - ??
    at ??:??
0x20736177 - ??
    at ??:??
0x6c6c6163 - s_rtc_timer_retain_mem
    at ??:??
0x61206465 - s_rtc_timer_retain_mem
    at ??:??
0x43502074 - esp_system_init_fn_init_components0
    at ??:??
0x34783020 - ??
    at ??:??
0x39323032 - ??
    at ??:??
0x20356562 - ??
    at ??:??
4080f390: 0x63206e6f 0x2065726f 0x00000030 0x40800000 0x42053d68 0x4080a194 0x4080f3c0 0x42029be8
0x63206e6f - s_rtc_timer_retain_mem
    at ??:??
0x2065726f - ??
    at ??:??
0x00000030 - ??
    at ??:??
0x40800000 - _vector_table
    at ??:??
0x42053d68 - anon.6f8d54fb548d5ff1abf9d01d3f110dc8.67.llvm.17408016719535564914
    at ??:??
0x4080a194 - _ZN3std9panicking4HOOK17hbc435311aff44f7fE
    at ??:??
0x4080f3c0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x42029be8 - <core::option::Option<T> as object::read::ReadError<T>>::read_error
    at /home/timo/.cargo/registry/src/index.crates.io-6f17d22bba15001f/object-0.32.1/src/read/mod.rs:101
4080f3b0: 0x00000001 0x4081011c 0x4080f3d0 0x42029bd8 0x40810120 0x4081011c 0x4080f410 0x4200ab2c
0x00000001 - ??
    at ??:??
0x4081011c - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f3d0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x42029bd8 - panic_abort::__rust_start_panic::abort
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/panic_abort/src/lib.rs:42
0x40810120 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4081011c - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f410 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4200ab2c - rust_panic
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:831
4080f3d0: 0x42053d68 0x4080a194 0x4080f3f0 0x4200b2d2 0x00000001 0x4081011c 0x4080f400 0x4200abe0
0x42053d68 - anon.6f8d54fb548d5ff1abf9d01d3f110dc8.67.llvm.17408016719535564914
    at ??:??
0x4080a194 - _ZN3std9panicking4HOOK17hbc435311aff44f7fE
    at ??:??
0x4080f3f0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4200b2d2 - std::sys::unix::thread_local_key::get
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread_local_key.rs:22
0x00000001 - ??
    at ??:??
0x4081011c - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f400 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4200abe0 - std::sys::common::thread_local::os_local::Key<T>::get
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/common/thread_local/os_local.rs:117
4080f3f0: 0x42053d68 0x4080a194 0x4080f410 0x42014af0 0x42053d68 0x4080a194 0x4080f480 0x4200aaf2
0x42053d68 - anon.6f8d54fb548d5ff1abf9d01d3f110dc8.67.llvm.17408016719535564914
    at ??:??
0x4080a194 - _ZN3std9panicking4HOOK17hbc435311aff44f7fE
    at ??:??
0x4080f410 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x42014af0 - std::thread::local::LocalKey<T>::try_with
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/local.rs:269
0x42053d68 - anon.6f8d54fb548d5ff1abf9d01d3f110dc8.67.llvm.17408016719535564914
    at ??:??
0x4080a194 - _ZN3std9panicking4HOOK17hbc435311aff44f7fE
    at ??:??
0x4080f480 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4200aaf2 - core::fmt::Arguments::new_const
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/fmt/mod.rs:325
4080f410: 0x00000804 0x00000054 0x00000000 0x00000000 0x00000000 0x4080f488 0x42053d58 0x4080f4f0
0x00000804 - ??
    at ??:??
0x00000054 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x4080f488 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x42053d58 - anon.6f8d54fb548d5ff1abf9d01d3f110dc8.67.llvm.17408016719535564914
    at ??:??
0x4080f4f0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
4080f430: 0x4205013c 0x00000001 0x00000000 0x00000000 0x00000000 0x00000000 0x40810090 0x00000000
0x4205013c - esp_app_desc
    at ??:??
0x00000001 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x40810090 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000000 - ??
    at ??:??
4080f450: 0x4080f4a8 0x4080f4d4 0x4080f4a0 0x00000000 0x00000000 0x00000000 0x00000000 0x4205013c
0x4080f4a8 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f4d4 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f4a0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x4205013c - esp_app_desc
    at ??:??
4080f470: 0x4080f4f0 0x4080f4d4 0x4080f4a0 0x4200a8b2 0x00000005 0x00000005 0x42050120 0x00000005
0x4080f4f0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f4d4 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f4a0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4200a8b2 - std::panicking::rust_panic_with_hook
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:734
0x00000005 - ??
    at ??:??
0x00000005 - ??
    at ??:??
0x42050120 - esp_app_desc
    at ??:??
0x00000005 - ??
    at ??:??
4080f490: 0x4080f554 0x4080f4d4 0x4080f4b0 0x42009f9e 0x4080ff60 0x00000005 0x4080f4d0 0x4200a6c0
0x4080f554 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f4d4 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f4b0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x42009f9e - std::sys_common::backtrace::output_filename
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:182
0x4080ff60 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000005 - ??
    at ??:??
0x4080f4d0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4200a6c0 - std::panicking::begin_panic_handler::FormatStringPayload::fill
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:604
4080f4b0: 0x4080f4f0 0x4080f4d4 0x4205013c 0x00000000 0x4080f554 0x4205014c 0x4080f4f0 0x4203b7e0
0x4080f4f0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f4d4 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4205013c - esp_app_desc
    at ??:??
0x00000000 - ??
    at ??:??
0x4080f554 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4205014c - anon.31793b6a68ab95f21ed38357dc39dbc2.0.llvm.17985776555296093392
    at ??:??
0x4080f4f0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4203b7e0 - core::panicking::panic_nounwind_fmt
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panicking.rs:86
4080f4d0: 0x4080a260 0x420649c0 0x420649c0 0x4080f4f0 0x4205013c 0x40800001 0x4080f510 0x42000082
0x4080a260 - s_log_cache
    at ??:??
0x420649c0 - str.3
    at ??:??
0x420649c0 - str.3
    at ??:??
0x4080f4f0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4205013c - esp_app_desc
    at ??:??
0x40800001 - _vector_table
    at ??:??
0x4080f510 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x42000082 - main
    at ??:??
4080f4f0: 0x42050128 0x00000001 0x42050120 0x00000000 0x00000000 0x00000000 0x4080f520 0x4200002a
0x42050128 - esp_app_desc
    at ??:??
0x00000001 - ??
    at ??:??
0x42050120 - esp_app_desc
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x4080f520 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4200002a - std::sys_common::backtrace::__rust_begin_short_backtrace
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:161
4080f510: 0x00000000 0x4205014c 0x4080f530 0x420000ea 0x4080f564 0x4080ff28 0x4080f550 0x4201d7bc
0x00000000 - ??
    at ??:??
0x4205014c - anon.31793b6a68ab95f21ed38357dc39dbc2.0.llvm.17985776555296093392
    at ??:??
0x4080f530 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x420000ea - std::rt::lang_start::{{closure}}
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:166
0x4080f564 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080ff28 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f550 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4201d7bc - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:284
4080f530: 0x4080f564 0x00000000 0x420504ec 0x4204a500 0x00000000 0x00000000 0x4080f560 0x420000d0
0x4080f564 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000000 - ??
    at ??:??
0x420504ec - anon.4c80ffe46e56309831a0959317c5d78b.174.llvm.13294485414787255219
    at ??:??
0x4204a500 - s_sar_power_release
    at /home/timo/esp/esp-idf/components/esp_hw_support/port/esp32c6/sar_periph_ctrl.c:74
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x4080f560 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x420000d0 - std::rt::lang_start
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:172
4080f550: 0x420694e0 0x42000032 0x00000000 0x4204a5e4 0x00000000 0x00001388 0x00000001 0x00000000
0x420694e0 - anon.d29e10f3af975f47e7d468438dff114e.33.llvm.12020091497156391816
    at ??:??
0x42000032 - backtrace_std::main
    at /tmp/backtrace-std/src/main.rs:13
0x00000000 - ??
    at ??:??
0x4204a5e4 - main_task
    at /home/timo/esp/esp-idf/components/freertos/app_startup.c:209
0x00000000 - ??
    at ??:??
0x00001388 - ??
    at ??:??
0x00000001 - ??
    at ??:??
0x00000000 - ??
    at ??:??
4080f570: 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
4080f590: 0x00000000 0xa5a5a5a5 0xa5a5a5a5 0xa5a5a5a5 0xa5a5a5a5 0xa5a5a5a5 0xa5a5a5a5 0xa5a5a5a5
0x00000000 - ??
    at ??:??
0xa5a5a5a5 - s_rtc_timer_retain_mem
    at ??:??
0xa5a5a5a5 - s_rtc_timer_retain_mem
    at ??:??
0xa5a5a5a5 - s_rtc_timer_retain_mem
    at ??:??
0xa5a5a5a5 - s_rtc_timer_retain_mem
    at ??:??
0xa5a5a5a5 - s_rtc_timer_retain_mem
    at ??:??
0xa5a5a5a5 - s_rtc_timer_retain_mem
    at ??:??
0xa5a5a5a5 - s_rtc_timer_retain_mem
    at ??:??
4080f5b0: 0x00000154 0x4080f170 0x00000000 0x4080a410 0x4080a410 0x4080f5b4 0x4080a408 0x00000018
0x00000154 - ??
    at ??:??
0x4080f170 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000000 - ??
    at ??:??
0x4080a410 - pxReadyTasksLists
    at ??:??
0x4080a410 - pxReadyTasksLists
    at ??:??
0x4080f5b4 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080a408 - pxReadyTasksLists
    at ??:??
0x00000018 - ??
    at ??:??
4080f5d0: 0x00000000 0x00000000 0x4080f5b4 0x00000000 0x00000001 0x4080d470 0x6e69616d 0x00000000
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x4080f5b4 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000000 - ??
    at ??:??
0x00000001 - ??
    at ??:??
0x4080d470 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x6e69616d - s_rtc_timer_retain_mem
    at ??:??
0x00000000 - ??
    at ??:??
4080f5f0: 0x00000000 0x00000000 0x00000000 0x4080f5a0 0x00000001 0x00000000 0x408100b4 0x4203dbb4
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x4080f5a0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000001 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x408100b4 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4203dbb4 - pthread_cleanup_thread_specific_data_callback
    at /home/timo/esp/esp-idf/components/pthread/pthread_local_storage.c:126
4080f610: 0x00000000 0x4080bcd0 0x4080bd38 0x4080bda0 0x00000000 0x00000000 0x00000001 0x00000000
0x00000000 - ??
    at ??:??
0x4080bcd0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080bd38 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080bda0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000001 - ??
    at ??:??
0x00000000 - ??
    at ??:??
4080f630: 0x00000000 0x00000000 0x42041e48 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x42041e48 - esp_cleanup_r
    at /home/timo/esp/esp-idf/components/newlib/newlib_init.c:60
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
4080f650: 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
4080f670: 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
4080f690: 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
4080f6b0: 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
4080f6d0: 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
4080f6f0: 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000600 0xa5a5a5a5
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000600 - ??
    at ??:??
0xa5a5a5a5 - s_rtc_timer_retain_mem
    at ??:??
4080f710: 0xa5a5a5a5 0xa5a5a5a5 0xa5a5a5a5 0xa5a5a5a5 0xa5a5a5a5 0xa5a5a5a5 0xa5a5a5a5 0xa5a5a5a5
0xa5a5a5a5 - s_rtc_timer_retain_mem
    at ??:??
0xa5a5a5a5 - s_rtc_timer_retain_mem
    at ??:??
0xa5a5a5a5 - s_rtc_timer_retain_mem
    at ??:??
0xa5a5a5a5 - s_rtc_timer_retain_mem
    at ??:??
0xa5a5a5a5 - s_rtc_timer_retain_mem
    at ??:??
0xa5a5a5a5 - s_rtc_timer_retain_mem
    at ??:??
0xa5a5a5a5 - s_rtc_timer_retain_mem
    at ??:??
0xa5a5a5a5 - s_rtc_timer_retain_mem
    at ??:??


Conclusion / Summary

The output of the esp-idf-version does not contain any of my functions (a,b,c). It does contain main, buried under a lot of noise.

I dont care a lot about the noise, but I would really like to see my functions (a,b,c).

@t-moe
Copy link
Author

t-moe commented Jan 9, 2024

Experiment 1: Add esp_backtrace to get more/alternative output.

New dependencies in Cargo.toml:
esp-backtrace = { version = "0.10.0", features = ["print-jtag-serial", "esp32c6"] }

Modified function c in main.rs:

fn c() {
    let bt = esp_backtrace::arch::backtrace();
    for a in bt {
        if let Some(a) = a {
            println!("esp_backtrace: 0x{:x}", a);
        }
    }

    panic!("oh no");
}

Additional output compared to before (raw, without espflash monitor):

esp_backtrace: 0x4200028a
esp_backtrace: 0x4201e1ae
esp_backtrace: 0x42000270

espflash run --monitor enhanced:

esp_backtrace: 0x4200028a
0x4200028a - std::rt::lang_start::{{closure}}
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:166
esp_backtrace: 0x4201e1ae
0x4201e1ae - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:284
esp_backtrace: 0x42000270
0x42000270 - std::rt::lang_start
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:172

This does not really add much value on its own. The functions in question are still missing.

@t-moe
Copy link
Author

t-moe commented Jan 9, 2024

Experiment 2: Add #[inline(never)]

I've annotated my functions a,b,c with #[inline(never)] , the rest is the same as in Experiment 1.

main.rs
#[inline(never)]
fn c() {
    let bt = esp_backtrace::arch::backtrace();
    for a in bt {
        if let Some(a) = a {
            println!("esp_backtrace: 0x{:x}", a);
        }
    }

    panic!("oh no");
}

#[inline(never)]
fn b() {
    c();
}
#[inline(never)]
fn a() {
    b();
}


fn main() {
    esp_idf_svc::sys::link_patches();
    esp_idf_svc::log::EspLogger::initialize_default();
    a();
}
espflash enhanced output:
esp_backtrace: 0x42000232
0x42000232 - backtrace_std::main
    at /tmp/backtrace-std/src/main.rs:24
esp_backtrace: 0x42000252
0x42000252 - main
    at ??:??
esp_backtrace: 0x42000040
0x42000040 - std::sys_common::backtrace::__rust_begin_short_backtrace
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:161
esp_backtrace: 0x420002ba
0x420002ba - std::rt::lang_start::{{closure}}
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:166
esp_backtrace: 0x4201e1de
0x4201e1de - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:284
esp_backtrace: 0x420002a0
0x420002a0 - std::rt::lang_start
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:172
thread 'main' panicked at src/main.rs:11:5:
oh no
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

abort() was called at PC 0x4202ac81 on core 0
0x4202ac81 - panic_abort::__rust_start_panic::abort
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/panic_abort/src/lib.rs:43
Core  0 register dump:
MEPC    : 0x40800ecc  RA      : 0x40804152  SP      : 0x4080f280  GP      : 0x408096e0  
0x40800ecc - panic_abort
    at /home/timo/esp/esp-idf/components/esp_system/panic.c:472
0x40804152 - __ubsan_include
    at /home/timo/esp/esp-idf/components/esp_system/ubsan.c:313
0x4080f280 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x408096e0 - coex_pti_tab
    at ??:??
TP      : 0x407efe44  T0      : 0x37363534  T1      : 0x7271706f  T2      : 0x33323130  
0x407efe44 - ??
    at ??:??
0x37363534 - ??
    at ??:??
0x7271706f - s_rtc_timer_retain_mem
    at ??:??
0x33323130 - ??
    at ??:??
S0/FP   : 0x4080f2bc  S1      : 0x4080f2bc  A0      : 0x4080f2bc  A1      : 0x4080f29e  
0x4080f2bc - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f2bc - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f2bc - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f29e - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
A2      : 0x00000000  A3      : 0x4080f2e9  A4      : 0x4080aebc  A5      : 0x4080aec4  
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f2e9 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080aebc - coex_schm_env
    at ??:??
0x4080aec4 - __sf
    at ??:??
A6      : 0x00000000  A7      : 0x76757473  S2      : 0x4080f2a0  S3      : 0x4080f3d8  
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x76757473 - s_rtc_timer_retain_mem
    at ??:??
0x4080f2a0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f3d8 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
S4      : 0x00000001  S5      : 0x00000000  S6      : 0x42050314  S7      : 0x00000001  
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x42050314 - anon.6511dfaaa557e14bcbd1dca894a2b50d.10.llvm.10280922575641377505
    at ??:??
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
S8      : 0x00000000  S9      : 0x00000000  S10     : 0x00000000  S11     : 0x00000000  
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
T3      : 0x6e6d6c6b  T4      : 0x6a696867  T5      : 0x66656463  T6      : 0x62613938  
0x6e6d6c6b - s_rtc_timer_retain_mem
    at ??:??
0x6a696867 - s_rtc_timer_retain_mem
    at ??:??
0x66656463 - s_rtc_timer_retain_mem
    at ??:??
0x62613938 - s_rtc_timer_retain_mem
    at ??:??
MSTATUS : 0x00001881  MTVEC   : 0x40800001  MCAUSE  : 0x00000007  MTVAL   : 0x00000000  
0x00001881 - ??
    at ??:??
0x40800001 - _vector_table
    at ??:??
0x00000007 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
MHARTID : 0x00000000  
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??

Stack memory:
4080f280: 0x00000000 0x42054004 0x4080f2bc 0x40807e80 0x00000003 0x00000000 0x00000001 0x40800030
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x42054004 - anon.6f8d54fb548d5ff1abf9d01d3f110dc8.67.llvm.17408016719535564914
    at ??:??
0x4080f2bc - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x40807e80 - __assert_func
    at /home/timo/esp/esp-idf/components/newlib/assert.c:34
0x00000003 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x40800030 - _vector_table
    at ??:??
4080f2a0: 0x32303234 0x31386361 0x40810600 0x40809f8c 0x4080f2a0 0x40809fa8 0x4080f29c 0x726f6261
0x32303234 - ??
    at ??:??
0x31386361 - ??
    at ??:??
0x40810600 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x40809f8c - __func__.0
    at ??:??
0x4080f2a0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x40809fa8 - __func__.0
    at ??:??
0x4080f29c - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x726f6261 - s_rtc_timer_retain_mem
    at ??:??
4080f2c0: 0x20292874 0x20736177 0x6c6c6163 0x61206465 0x43502074 0x34783020 0x61323032 0x20313863
0x20292874 - ??
    at ??:??
0x20736177 - ??
    at ??:??
0x6c6c6163 - s_rtc_timer_retain_mem
    at ??:??
0x61206465 - s_rtc_timer_retain_mem
    at ??:??
0x43502074 - esp_system_init_fn_init_components0
    at ??:??
0x34783020 - ??
    at ??:??
0x61323032 - s_rtc_timer_retain_mem
    at ??:??
0x20313863 - ??
    at ??:??
4080f2e0: 0x63206e6f 0x2065726f 0x00000030 0x40800000 0x42054100 0x4080a194 0x4080f310 0x4202ac84
0x63206e6f - s_rtc_timer_retain_mem
    at ??:??
0x2065726f - ??
    at ??:??
0x00000030 - ??
    at ??:??
0x40800000 - _vector_table
    at ??:??
0x42054100 - anon.6f8d54fb548d5ff1abf9d01d3f110dc8.67.llvm.17408016719535564914
    at ??:??
0x4080a194 - _ZN3std9panicking4HOOK17hbc435311aff44f7fE
    at ??:??
0x4080f310 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4202ac84 - <core::option::Option<T> as object::read::ReadError<T>>::read_error
    at /home/timo/.cargo/registry/src/index.crates.io-6f17d22bba15001f/object-0.32.1/src/read/mod.rs:101
4080f300: 0x00000001 0x408105b4 0x4080f320 0x4202ac74 0x408105b8 0x408105b4 0x4080f360 0x4200b2fe
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x408105b4 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f320 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4202ac74 - panic_abort::__rust_start_panic::abort
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/panic_abort/src/lib.rs:42
0x408105b8 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x408105b4 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f360 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4200b2fe - rust_panic
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:831
4080f320: 0x42054100 0x4080a194 0x4080f340 0x4200baa4 0x00000001 0x408105b4 0x4080f350 0x4200b3b2
0x42054100 - anon.6f8d54fb548d5ff1abf9d01d3f110dc8.67.llvm.17408016719535564914
    at ??:??
0x4080a194 - _ZN3std9panicking4HOOK17hbc435311aff44f7fE
    at ??:??
0x4080f340 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4200baa4 - std::sys::unix::thread_local_key::get
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread_local_key.rs:22
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x408105b4 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f350 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4200b3b2 - std::sys::common::thread_local::os_local::Key<T>::get
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/common/thread_local/os_local.rs:117
4080f340: 0x42054100 0x4080a194 0x4080f360 0x42015372 0x42054100 0x4080a194 0x4080f3d0 0x4200b2c4
0x42054100 - anon.6f8d54fb548d5ff1abf9d01d3f110dc8.67.llvm.17408016719535564914
    at ??:??
0x4080a194 - _ZN3std9panicking4HOOK17hbc435311aff44f7fE
    at ??:??
0x4080f360 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x42015372 - std::thread::local::LocalKey<T>::try_with
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/local.rs:269
0x42054100 - anon.6f8d54fb548d5ff1abf9d01d3f110dc8.67.llvm.17408016719535564914
    at ??:??
0x4080a194 - _ZN3std9panicking4HOOK17hbc435311aff44f7fE
    at ??:??
0x4080f3d0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4200b2c4 - core::fmt::Arguments::new_const
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/fmt/mod.rs:325
4080f360: 0x00000001 0x4080f3f0 0x40810518 0x00000000 0x4080f4a0 0x4080f3d8 0x420540f0 0x4080f444
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f3f0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x40810518 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f4a0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f3d8 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x420540f0 - anon.6f8d54fb548d5ff1abf9d01d3f110dc8.67.llvm.17408016719535564914
    at ??:??
0x4080f444 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
4080f380: 0x42050314 0x40800001 0x4080f3b0 0x00000000 0x4080f404 0x00000002 0x4080f3bc 0x42038270
0x42050314 - anon.6511dfaaa557e14bcbd1dca894a2b50d.10.llvm.10280922575641377505
    at ??:??
0x40800001 - _vector_table
    at ??:??
0x4080f3b0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f404 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000002 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f3bc - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x42038270 - core::fmt::num::<impl core::fmt::LowerHex for i32>::fmt
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/fmt/num.rs:158
4080f3a0: 0x4080f4a0 0x4080f3f0 0x4080f3d0 0x00000000 0x4080f4b8 0x00000002 0x42050338 0x42050314
0x4080f4a0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f3f0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f3d0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f4b8 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000002 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x42050338 - anon.6511dfaaa557e14bcbd1dca894a2b50d.10.llvm.10280922575641377505
    at ??:??
0x42050314 - anon.6511dfaaa557e14bcbd1dca894a2b50d.10.llvm.10280922575641377505
    at ??:??
4080f3c0: 0x4080f444 0x4080f424 0x4080f3f0 0x4200b084 0x42000040 0x4080f3fc 0x420502f8 0x00000005
0x4080f444 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f424 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f3f0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4200b084 - std::panicking::rust_panic_with_hook
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:734
0x42000040 - std::sys_common::backtrace::__rust_begin_short_backtrace
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:161
0x4080f3fc - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x420502f8 - anon.6511dfaaa557e14bcbd1dca894a2b50d.10.llvm.10280922575641377505
    at ??:??
0x00000005 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
4080f3e0: 0x4201e1de 0x00000001 0x4080f400 0x4200a770 0x00001804 0x00000000 0x4080f420 0x4200ae92
0x4201e1de - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:284
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f400 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4200a770 - std::sys_common::backtrace::output_filename
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:182
0x00001804 - ??
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f420 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4200ae92 - std::panicking::begin_panic_handler::FormatStringPayload::fill
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:604
4080f400: 0x4080f444 0x4080f424 0x42050314 0x42038270 0x4080f49c 0x00000001 0x4080f440 0x4203ca00
0x4080f444 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f424 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x42050314 - anon.6511dfaaa557e14bcbd1dca894a2b50d.10.llvm.10280922575641377505
    at ??:??
0x42038270 - core::fmt::num::<impl core::fmt::LowerHex for i32>::fmt
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/fmt/num.rs:158
0x4080f49c - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f440 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4203ca00 - core::panicking::panic_nounwind_fmt
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panicking.rs:86
4080f420: 0x00000000 0x420655d0 0x420655d0 0x4080f444 0x42050314 0x00000001 0x4080f4e0 0x42000212
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x420655d0 - str.3
    at ??:??
0x420655d0 - str.3
    at ??:??
0x4080f444 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x42050314 - anon.6511dfaaa557e14bcbd1dca894a2b50d.10.llvm.10280922575641377505
    at ??:??
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f4e0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x42000212 - backtrace_std::b
    at /tmp/backtrace-std/src/main.rs:15
4080f440: 0x00000000 0x42050300 0x00000001 0x420502f8 0x00000000 0x00000000 0x42000040 0x00000001
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x42050300 - anon.6511dfaaa557e14bcbd1dca894a2b50d.10.llvm.10280922575641377505
    at ??:??
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x420502f8 - anon.6511dfaaa557e14bcbd1dca894a2b50d.10.llvm.10280922575641377505
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x42000040 - std::sys_common::backtrace::__rust_begin_short_backtrace
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:161
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
4080f460: 0x420002ba 0x00000001 0x4201e1de 0x00000001 0x420002a0 0x00000000 0x00001800 0x00000000
0x420002ba - std::rt::lang_start::{{closure}}
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:166
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4201e1de - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:284
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x420002a0 - std::rt::lang_start
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:172
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00001800 - ??
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
4080f480: 0x4080f440 0x00000000 0x00000000 0x00000000 0x4080f440 0x0000000a 0x0000000a 0x420002a0
0x4080f440 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f440 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x0000000a - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x0000000a - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x420002a0 - std::rt::lang_start
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:172
4080f4a0: 0x42050338 0x00000002 0x4080f4b8 0x00000001 0x00000000 0x4080ff70 0x4080f49c 0x42038270
0x42050338 - anon.6511dfaaa557e14bcbd1dca894a2b50d.10.llvm.10280922575641377505
    at ??:??
0x00000002 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f4b8 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080ff70 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f49c - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x42038270 - core::fmt::num::<impl core::fmt::LowerHex for i32>::fmt
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/fmt/num.rs:158
4080f4c0: 0x00000000 0x00000000 0x00000000 0x00000000 0x4080f554 0x42050348 0x4080f4f0 0x42000222
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f554 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x42050348 - anon.c1dd869abbd1a5a08a4177ed47952a72.0.llvm.94216436206020396
    at ??:??
0x4080f4f0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x42000222 - backtrace_std::a
    at /tmp/backtrace-std/src/main.rs:19
4080f4e0: 0x00000003 0x4080f54c 0x4080f500 0x42000232 0x00000000 0x00000000 0x4080f510 0x42000252
0x00000003 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f54c - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f500 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x42000232 - backtrace_std::main
    at /tmp/backtrace-std/src/main.rs:24
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f510 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x42000252 - main
    at ??:??
4080f500: 0x00000000 0x00000000 0x4080f520 0x42000040 0x00000000 0x42050348 0x4080f530 0x420002ba
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f520 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x42000040 - std::sys_common::backtrace::__rust_begin_short_backtrace
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:161
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x42050348 - anon.c1dd869abbd1a5a08a4177ed47952a72.0.llvm.94216436206020396
    at ??:??
0x4080f530 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x420002ba - std::rt::lang_start::{{closure}}
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:166
4080f520: 0x4080f564 0x4080ff28 0x4080f550 0x4201e1de 0x4080f564 0x00000000 0x420506e8 0x4204b800
0x4080f564 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080ff28 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f550 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4201e1de - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:284
0x4080f564 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x420506e8 - anon.4c80ffe46e56309831a0959317c5d78b.174.llvm.13294485414787255219
    at ??:??
0x4204b800 - s_sar_power_release
    at /home/timo/esp/esp-idf/components/esp_hw_support/port/esp32c6/sar_periph_ctrl.c:80
4080f540: 0x00000000 0x00000000 0x4080f560 0x420002a0 0x4206a0f0 0x42000232 0x00000000 0x4204b8a0
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f560 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x420002a0 - std::rt::lang_start
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:172
0x4206a0f0 - anon.d29e10f3af975f47e7d468438dff114e.33.llvm.12020091497156391816
    at ??:??
0x42000232 - backtrace_std::main
    at /tmp/backtrace-std/src/main.rs:24
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4204b8a0 - main_task
    at /home/timo/esp/esp-idf/components/freertos/app_startup.c:209
4080f560: 0x00000000 0x00001388 0x00000001 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00001388 - ??
    at ??:??
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
4080f580: 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0xa5a5a5a5 0xa5a5a5a5 0xa5a5a5a5
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0xa5a5a5a5 - s_rtc_timer_retain_mem
    at ??:??
0xa5a5a5a5 - s_rtc_timer_retain_mem
    at ??:??
0xa5a5a5a5 - s_rtc_timer_retain_mem
    at ??:??
4080f5a0: 0xa5a5a5a5 0xa5a5a5a5 0xa5a5a5a5 0xa5a5a5a5 0x00000154 0x4080f0c0 0x00000000 0x4080a410
0xa5a5a5a5 - s_rtc_timer_retain_mem
    at ??:??
0xa5a5a5a5 - s_rtc_timer_retain_mem
    at ??:??
0xa5a5a5a5 - s_rtc_timer_retain_mem
    at ??:??
0xa5a5a5a5 - s_rtc_timer_retain_mem
    at ??:??
0x00000154 - ??
    at ??:??
0x4080f0c0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080a410 - pxReadyTasksLists
    at ??:??
4080f5c0: 0x4080a410 0x4080f5b4 0x4080a408 0x00000018 0x00000000 0x00000000 0x4080f5b4 0x00000000
0x4080a410 - pxReadyTasksLists
    at ??:??
0x4080f5b4 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080a408 - pxReadyTasksLists
    at ??:??
0x00000018 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f5b4 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
4080f5e0: 0x00000001 0x4080d470 0x6e69616d 0x00000000 0x00000000 0x00000000 0x00000000 0x4080f5a0
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080d470 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x6e69616d - s_rtc_timer_retain_mem
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f5a0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
4080f600: 0x00000001 0x00000000 0x408100b4 0x4203ee70 0x00000000 0x4080bcd0 0x4080bd38 0x4080bda0
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x408100b4 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4203ee70 - pthread_cleanup_thread_specific_data_callback
    at /home/timo/esp/esp-idf/components/pthread/pthread_local_storage.c:126
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080bcd0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080bd38 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080bda0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
4080f620: 0x00000000 0x00000000 0x00000001 0x00000000 0x00000000 0x00000000 0x42043104 0x00000000
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x42043104 - esp_cleanup_r
    at /home/timo/esp/esp-idf/components/newlib/newlib_init.c:60
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
4080f640: 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
4080f660: 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??

esp_backtrace contains main now, but not a,b,c
panic output contains main, a and b but is still very noisy and not really helpful

@t-moe
Copy link
Author

t-moe commented Jan 9, 2024

Experiment 3: Set RUST_BACKTRACE=1

I've added std::env::set_var("RUST_BACKTRACE", "1"); to main.rs, the rest is the same as in Experiment 2.

main.rs
#[inline(never)]
fn c() {
    let bt = esp_backtrace::arch::backtrace();
    for a in bt {
        if let Some(a) = a {
            println!("esp_backtrace: 0x{:x}", a);
        }
    }

    panic!("oh no");
}

#[inline(never)]
fn b() {
    c();
}
#[inline(never)]
fn a() {
    b();
}


fn main() {
    esp_idf_svc::sys::link_patches();
    esp_idf_svc::log::EspLogger::initialize_default();
    std::env::set_var("RUST_BACKTRACE", "1");
    a();
}
espflash enhanced output:
esp_backtrace: 0x42000232
0x42000232 - backtrace_std::main
    at /tmp/backtrace-std/src/main.rs:24
esp_backtrace: 0x4200026e
0x4200026e - main
    at ??:??
esp_backtrace: 0x42000040
0x42000040 - std::sys_common::backtrace::__rust_begin_short_backtrace
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:161
esp_backtrace: 0x420002ec
0x420002ec - std::rt::lang_start::{{closure}}
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:166
esp_backtrace: 0x4201e8e8
0x4201e8e8 - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:284
esp_backtrace: 0x420002d2
0x420002d2 - std::rt::lang_start
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:172
thread 'main' panicked at src/main.rs:11:5:
oh no
stack backtrace:

abort() was called at PC 0x420444b1 on core 0
0x420444b1 - abort_return<int>
    at /home/timo/esp/esp-idf/components/cxx/cxx_exception_stubs.cpp:33
Core  0 register dump:
MEPC    : 0x40800ecc  RA      : 0x40804152  SP      : 0x4080f0b0  GP      : 0x408096e0  
0x40800ecc - panic_abort
    at /home/timo/esp/esp-idf/components/esp_system/panic.c:472
0x40804152 - __ubsan_include
    at /home/timo/esp/esp-idf/components/esp_system/ubsan.c:313
0x4080f0b0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x408096e0 - coex_pti_tab
    at ??:??
TP      : 0x407ef894  T0      : 0x37363534  T1      : 0x7271706f  T2      : 0x33323130  
0x407ef894 - ??
    at ??:??
0x37363534 - ??
    at ??:??
0x7271706f - s_rtc_timer_retain_mem
    at ??:??
0x33323130 - ??
    at ??:??
S0/FP   : 0x4080f0ec  S1      : 0x4080f0ec  A0      : 0x4080f0ec  A1      : 0x4080f0ce  
0x4080f0ec - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f0ec - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f0ec - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f0ce - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
A2      : 0x00000000  A3      : 0x4080f119  A4      : 0x4080aebc  A5      : 0x4080aec4  
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f119 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080aebc - coex_schm_env
    at ??:??
0x4080aec4 - __sf
    at ??:??
A6      : 0x00000000  A7      : 0x76757473  S2      : 0x4080f0d0  S3      : 0x00000001  
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x76757473 - s_rtc_timer_retain_mem
    at ??:??
0x4080f0d0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
S4      : 0x00000000  S5      : 0x4080f25c  S6      : 0x00000000  S7      : 0x00000001  
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f25c - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
S8      : 0x00000000  S9      : 0x00000000  S10     : 0x00000000  S11     : 0x00000000  
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
T3      : 0x6e6d6c6b  T4      : 0x6a696867  T5      : 0x66656463  T6      : 0x62613938  
0x6e6d6c6b - s_rtc_timer_retain_mem
    at ??:??
0x6a696867 - s_rtc_timer_retain_mem
    at ??:??
0x66656463 - s_rtc_timer_retain_mem
    at ??:??
0x62613938 - s_rtc_timer_retain_mem
    at ??:??
MSTATUS : 0x00001881  MTVEC   : 0x40800001  MCAUSE  : 0x00000007  MTVAL   : 0x00000000  
0x00001881 - ??
    at ??:??
0x40800001 - _vector_table
    at ??:??
0x00000007 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
MHARTID : 0x00000000  
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??

Stack memory:
4080f0b0: 0x4080b8e0 0x00000001 0x4080f0ec 0x40807e80 0x00000000 0x420539dc 0x00000001 0x00000030
0x4080b8e0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f0ec - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x40807e80 - __assert_func
    at /home/timo/esp/esp-idf/components/newlib/assert.c:34
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x420539dc - anon.6f8d54fb548d5ff1abf9d01d3f110dc8.4.llvm.17408016719535564914
    at ??:??
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000030 - ??
    at ??:??
4080f0d0: 0x34303234 0x31623434 0x4080f100 0x40809f8c 0x4080f0d0 0x40809fa8 0x4080f0cc 0x726f6261
0x34303234 - ??
    at ??:??
0x31623434 - ??
    at ??:??
0x4080f100 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x40809f8c - __func__.0
    at ??:??
0x4080f0d0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x40809fa8 - __func__.0
    at ??:??
0x4080f0cc - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x726f6261 - s_rtc_timer_retain_mem
    at ??:??
4080f0f0: 0x20292874 0x20736177 0x6c6c6163 0x61206465 0x43502074 0x34783020 0x34343032 0x20316234
0x20292874 - ??
    at ??:??
0x20736177 - ??
    at ??:??
0x6c6c6163 - s_rtc_timer_retain_mem
    at ??:??
0x61206465 - s_rtc_timer_retain_mem
    at ??:??
0x43502074 - esp_system_init_fn_init_components0
    at ??:??
0x34783020 - ??
    at ??:??
0x34343032 - ??
    at ??:??
0x20316234 - ??
    at ??:??
4080f110: 0x63206e6f 0x2065726f 0x00000030 0x42050000 0x4080f1c4 0x00000001 0x4080f150 0x420444b4
0x63206e6f - s_rtc_timer_retain_mem
    at ??:??
0x2065726f - ??
    at ??:??
0x00000030 - ??
    at ??:??
0x42050000 - intr_handler_set
    at ??:??
0x4080f1c4 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f150 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x420444b4 - __wrap___gxx_personality_v0
    at /home/timo/esp/esp-idf/components/cxx/cxx_exception_stubs.cpp:184
4080f130: 0x00000000 0x4080f25c 0x00000000 0x4201edb0 0x4080f18c 0x42059b20 0x4080f1c0 0x4200ab32
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f25c - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4201edb0 - std::backtrace_rs::backtrace::trace_unsynchronized
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/../../backtrace/src/backtrace/mod.rs:67
0x4080f18c - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x42059b20 - anon.bdfacf5b35ca691d5ec60ab94749bceb.70.llvm.15151070200704552323
    at ??:??
0x4080f1c0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4200ab32 - std::sys_common::backtrace::_print_fmt
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:135
4080f150: 0x42050314 0x00000000 0x00000001 0x40810858 0x00000001 0x40810700 0x4080f1c4 0x4080f158
0x42050314 - anon.6511dfaaa557e14bcbd1dca894a2b50d.10.llvm.10280922575641377505
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x40810858 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x40810700 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f1c4 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f158 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
4080f170: 0x42054044 0x00000000 0x40810700 0x00000000 0x0080b034 0x00000000 0x0001f1c0 0x4080f157
0x42054044 - anon.6f8d54fb548d5ff1abf9d01d3f110dc8.58.llvm.17408016719535564914
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x40810700 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x0080b034 - ??
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x0001f1c0 - ??
    at ??:??
0x4080f157 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
4080f190: 0x4080f17c 0x4080f18b 0x4080f184 0x4080f18a 0x4080f168 0x4080f183 0x00000001 0x00000001
0x4080f17c - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f18b - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f184 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f18a - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f168 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f183 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
4080f1b0: 0x42054020 0x00000008 0x4080f210 0x42038d76 0x4080f21c 0x00000000 0x4080f1e0 0x00000000
0x42054020 - anon.6f8d54fb548d5ff1abf9d01d3f110dc8.58.llvm.17408016719535564914
    at ??:??
0x00000008 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f210 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x42038d76 - core::fmt::write
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/fmt/mod.rs:1120
0x4080f21c - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f1e0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
4080f1d0: 0x00000003 0x00000020 0x4080f210 0x4205a390 0x00000000 0x4080f203 0x00000000 0x00000001
0x00000003 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000020 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f210 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4205a390 - anon.9ac532135757da3aea0395e7f4074be1.49.llvm.9578683146214607830
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f203 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
4080f1f0: 0x4080f238 0x00000000 0x4080f2ac 0x4080f34f 0x00000004 0x4080f2ac 0x4080f230 0x42020416
0x4080f238 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f2ac - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f34f - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000004 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f2ac - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f230 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x42020416 - std::io::Write::write_fmt
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/io/mod.rs:1810
4080f210: 0x40810004 0x00000000 0x4080f34f 0x00000000 0x42054184 0x4080b034 0x4080f280 0x4200aa10
0x40810004 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f34f - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x42054184 - anon.6f8d54fb548d5ff1abf9d01d3f110dc8.67.llvm.17408016719535564914
    at ??:??
0x4080b034 - _ZN3std10sys_common9backtrace4lock4LOCK17he7fba721b3476408E.llvm.17408016719535564914
    at ??:??
0x4080f280 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4200aa10 - <std::sync::mutex::MutexGuard<T> as core::ops::drop::Drop>::drop
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sync/mutex.rs:528
4080f230: 0x00000000 0x4080b034 0x00000001 0x42054020 0x00000001 0x4080f254 0x00000001 0x00000000
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080b034 - _ZN3std10sys_common9backtrace4lock4LOCK17he7fba721b3476408E.llvm.17408016719535564914
    at ??:??
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x42054020 - anon.6f8d54fb548d5ff1abf9d01d3f110dc8.58.llvm.17408016719535564914
    at ??:??
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f254 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
4080f250: 0x00000004 0x4080f25f 0x4200aa3c 0x00020440 0x42050314 0x00000003 0x420203ea 0x42054184
0x00000004 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f25f - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4200aa3c - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:43
0x00020440 - ??
    at ??:??
0x42050314 - anon.6511dfaaa557e14bcbd1dca894a2b50d.10.llvm.10280922575641377505
    at ??:??
0x00000003 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x420203ea - std::io::Write::write_fmt
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/io/mod.rs:1789
0x42054184 - anon.6f8d54fb548d5ff1abf9d01d3f110dc8.67.llvm.17408016719535564914
    at ??:??
4080f270: 0x4080f34f 0x4080f328 0x4080f2e0 0x4200b028 0xcd1e7bc1 0xc1a2c89c 0x00b1ef64 0xfdbc1604
0x4080f34f - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f328 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f2e0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4200b028 - core::mem::drop
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/mem/mod.rs:992
0xcd1e7bc1 - s_rtc_timer_retain_mem
    at ??:??
0xc1a2c89c - s_rtc_timer_retain_mem
    at ??:??
0x00b1ef64 - ??
    at ??:??
0xfdbc1604 - s_rtc_timer_retain_mem
    at ??:??
4080f290: 0x40880000 0x42054228 0x00000004 0x4080f2ac 0x00000003 0x00000000 0x4080f2c0 0x4080f320
0x40880000 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x42054228 - anon.6f8d54fb548d5ff1abf9d01d3f110dc8.67.llvm.17408016719535564914
    at ??:??
0x00000004 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f2ac - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000003 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f2c0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f320 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
4080f2b0: 0x42039c30 0x4080f310 0x4201af8a 0x4080f314 0x42039c30 0x00000000 0x00000001 0x4080f3e8
0x42039c30 - <&T as core::fmt::Display>::fmt
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/fmt/mod.rs:2294
0x4080f310 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4201af8a - <&T as core::fmt::Display>::fmt
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/fmt/mod.rs:2294
0x4080f314 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x42039c30 - <&T as core::fmt::Display>::fmt
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/fmt/mod.rs:2294
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f3e8 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
4080f2d0: 0x4080f3e8 0x4080f2f0 0x4080f370 0x4200aeea 0x408106e8 0x408106e4 0x40810784 0x4080485a
0x4080f3e8 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f2f0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f370 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4200aeea - std::panicking::default_hook
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:294
0x408106e8 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x408106e4 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x40810784 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080485a - xQueueCreateMutex
    at /home/timo/esp/esp-idf/components/freertos/FreeRTOS-Kernel/queue.c:673
4080f2f0: 0x408106e8 0x00000000 0x4081071c 0x40804190 0x00000000 0x00000000 0x4081071c 0x008046a8
0x408106e8 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4081071c - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x40804190 - prvCopyDataToQueue
    at /home/timo/esp/esp-idf/components/freertos/FreeRTOS-Kernel/queue.c:2470
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4081071c - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x008046a8 - ??
    at ??:??
4080f310: 0x42050314 0x420502f8 0x00000005 0x4080ff80 0x4080ff70 0x00000004 0x4080f320 0x4080f310
0x42050314 - anon.6511dfaaa557e14bcbd1dca894a2b50d.10.llvm.10280922575641377505
    at ??:??
0x420502f8 - anon.6511dfaaa557e14bcbd1dca894a2b50d.10.llvm.10280922575641377505
    at ??:??
0x00000005 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080ff80 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080ff70 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000004 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f320 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f310 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
4080f330: 0x4080f314 0x4080f30f 0x00000001 0x4080f3e8 0x00000001 0x408106e4 0x4081070c 0x4203f85e
0x4080f314 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f30f - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f3e8 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x408106e4 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4081070c - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4203f85e - pthread_mutex_unlock
    at /home/timo/esp/esp-idf/components/pthread/pthread.c:703
4080f350: 0x42050314 0x00000000 0x00000001 0x4080f3e8 0x420542d0 0x4080a194 0x4080f3e0 0x4200b66c
0x42050314 - anon.6511dfaaa557e14bcbd1dca894a2b50d.10.llvm.10280922575641377505
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f3e8 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x420542d0 - anon.6f8d54fb548d5ff1abf9d01d3f110dc8.67.llvm.17408016719535564914
    at ??:??
0x4080a194 - _ZN3std9panicking4HOOK17hbc435311aff44f7fE
    at ??:??
0x4080f3e0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4200b66c - std::panicking::rust_panic_with_hook
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:779
4080f370: 0x00000001 0x4080f400 0x40810648 0x00000000 0x4080f4b0 0x4080f3e8 0x420542c0 0x4080f454
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f400 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x40810648 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f4b0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f3e8 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x420542c0 - anon.6f8d54fb548d5ff1abf9d01d3f110dc8.67.llvm.17408016719535564914
    at ??:??
0x4080f454 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
4080f390: 0x42050314 0x40800001 0x4080f3c0 0x00000000 0x4080f404 0x00000002 0x4080f3cc 0x4203897a
0x42050314 - anon.6511dfaaa557e14bcbd1dca894a2b50d.10.llvm.10280922575641377505
    at ??:??
0x40800001 - _vector_table
    at ??:??
0x4080f3c0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f404 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000002 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f3cc - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4203897a - core::fmt::num::<impl core::fmt::LowerHex for i32>::fmt
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/fmt/num.rs:158
4080f3b0: 0x4080f4b0 0x4080f400 0x4080f3e0 0x00000000 0x4080f4c8 0x00000002 0x42050338 0x42050314
0x4080f4b0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f400 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f3e0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f4c8 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000002 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x42050338 - anon.6511dfaaa557e14bcbd1dca894a2b50d.10.llvm.10280922575641377505
    at ??:??
0x42050314 - anon.6511dfaaa557e14bcbd1dca894a2b50d.10.llvm.10280922575641377505
    at ??:??
4080f3d0: 0x4080f454 0x4080f434 0x4080f400 0x4200b4e0 0x42000040 0x4080f40c 0x420502f8 0x00000005
0x4080f454 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f434 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f400 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4200b4e0 - std::panicking::rust_panic_with_hook
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:734
0x42000040 - std::sys_common::backtrace::__rust_begin_short_backtrace
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:161
0x4080f40c - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x420502f8 - anon.6511dfaaa557e14bcbd1dca894a2b50d.10.llvm.10280922575641377505
    at ??:??
0x00000005 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
4080f3f0: 0x4201e8e8 0x00000001 0x4080f410 0x4200abcc 0x00000004 0x00000000 0x4080f430 0x4200b2ee
0x4201e8e8 - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:284
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f410 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4200abcc - std::sys_common::backtrace::output_filename
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:182
0x00000004 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f430 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4200b2ee - std::panicking::begin_panic_handler::FormatStringPayload::fill
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:604
4080f410: 0x4080f454 0x4080f434 0x42050314 0x4203897a 0x4080f4ac 0x00000001 0x4080f450 0x4203d742
0x4080f454 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080f434 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x42050314 - anon.6511dfaaa557e14bcbd1dca894a2b50d.10.llvm.10280922575641377505
    at ??:??
0x4203897a - core::fmt::num::<impl core::fmt::LowerHex for i32>::fmt
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/fmt/num.rs:158
0x4080f4ac - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f450 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4203d742 - core::panicking::panic_nounwind_fmt
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panicking.rs:86
4080f430: 0x00000000 0x42065b90 0x42065b90 0x4080f454 0x42050314 0x00000001 0x4080f4f0 0x42000212
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x42065b90 - str.3
    at ??:??
0x42065b90 - str.3
    at ??:??
0x4080f454 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x42050314 - anon.6511dfaaa557e14bcbd1dca894a2b50d.10.llvm.10280922575641377505
    at ??:??
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f4f0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x42000212 - backtrace_std::b
    at /tmp/backtrace-std/src/main.rs:15
4080f450: 0x00000031 0x42050300 0x00000001 0x420502f8 0x00000000 0x00000000 0x42000040 0x00000001
0x00000031 - ??
    at ??:??
0x42050300 - anon.6511dfaaa557e14bcbd1dca894a2b50d.10.llvm.10280922575641377505
    at ??:??
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x420502f8 - anon.6511dfaaa557e14bcbd1dca894a2b50d.10.llvm.10280922575641377505
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x42000040 - std::sys_common::backtrace::__rust_begin_short_backtrace
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:161
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
4080f470: 0x420002ec 0x00000001 0x4201e8e8 0x00000001 0x420002d2 0x00000000 0x00000000 0x00000000
0x420002ec - std::rt::lang_start::{{closure}}
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:166
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4201e8e8 - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:284
0x00000001 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x420002d2 - std::rt::lang_start
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:172
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
4080f490: 0x4080f420 0x00000000 0x4080f4c8 0x00000000 0x4080f4b0 0x0000000a 0x0000000a 0x420002d2
0x4080f420 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f4c8 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000000 - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x4080f4b0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x0000000a - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x0000000a - core::array::iter::_$LT$impl$u20$core..iter..traits..collect..IntoIterator$u20$for$u20$$u5b$T$u3b$$u20$N$u5d$$GT$::into_iter::h1f11533074ef3cf2
    at ??:??
0x420002d2 - std::rt::lang_start
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:172

Same results as in experiment 2.

@t-moe
Copy link
Author

t-moe commented Jan 9, 2024

Experiment 4: Enable DWARF by setting CONFIG_ESP_SYSTEM_USE_EH_FRAME=y

I've added CONFIG_ESP_SYSTEM_USE_EH_FRAME=y to sdkconfig.defaults, the rest ist the same as in experiment 3.

sdkconfig.defautls
# Rust often needs a bit of an extra main task stack size compared to C (the default is 3K)
CONFIG_ESP_MAIN_TASK_STACK_SIZE=8000

CONFIG_ESP_SYSTEM_PANIC_PRINT_REBOOT=n
CONFIG_ESP_SYSTEM_PANIC_PRINT_HALT=y

CONFIG_ESP_SYSTEM_USE_EH_FRAME=y

Note: this needs "-C", "force-unwind-tables=yes" rustc flags to work properly. Thanks @Vollbrecht

raw serial monitor output:
esp_backtrace: 0x42000232
esp_backtrace: 0x4200026e
esp_backtrace: 0x42000040
esp_backtrace: 0x420002ec
esp_backtrace: 0x4201e8e8
esp_backtrace: 0x420002d2
thread 'main' panicked at src/main.rs:11:5:
oh no
stack backtrace:

abort() was called at PC 0x42044d2d on core 0
Backtrace: 0x40800eba:0x4080f1a0 0x4080413e:0x4080f1a0 0x40807e6c:0x4080f1b0 0x42044d2e:0x4080f220 0x4201edae:0x4080f230 0x4200ab30:0x4080f240 0x42038d74:0x4080f2b0 0x42020414:0x4080f300 0x4200aa0e:0x4080f320 0x4200b026:0x4080f370 0x4200aee8:0x4080f3d0 0x4200b66a:0x4080f460 0x4200b4de:0x4080f4d0 0x4200abca:0x4080f4f0 0x4200b2ec:0x4080f500 0x4203d740:0x4080f520 0x42000210:0x4080f540 0x42000220:0x4080f5e0 0x42000230:0x4080f5f0 0x4200026c:0x4080f600 0x4200003e:0x4080f610 0x420002ea:0x4080f620 0x4201e8e6:0x4080f630 0x420002d0:0x4080f650 0x4204d0e8:0x4080f660 0xfffffffe:0x4080f680
Backtrace ended abruptly: cannot find DWARF information for instruction at address 0xfffffffe

espflash enhanced output:
esp_backtrace: 0x42000232
0x42000232 - backtrace_std::main
    at /tmp/backtrace-std/src/main.rs:24
esp_backtrace: 0x4200026e
0x4200026e - main
    at ??:??
esp_backtrace: 0x42000040
0x42000040 - std::sys_common::backtrace::__rust_begin_short_backtrace
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:161
esp_backtrace: 0x420002ec
0x420002ec - std::rt::lang_start::{{closure}}
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:166
esp_backtrace: 0x4201e8e8
0x4201e8e8 - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:284
esp_backtrace: 0x420002d2
0x420002d2 - std::rt::lang_start
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:172
thread 'main' panicked at src/main.rs:11:5:
oh no
stack backtrace:

abort() was called at PC 0x42044d2d on core 0
0x42044d2d - abort_return<int>
    at /home/timo/esp/esp-idf/components/cxx/cxx_exception_stubs.cpp:33
Backtrace: 0x40800eba:0x4080f1a0 0x4080413e:0x4080f1a0 0x40807e6c:0x4080f1b0 0x42044d2e:0x4080f220 0x4201edae:0x4080f230 0x4200ab30:0x4080f240 0x42038d74:0x4080f2b0 0x42020414:0x4080f300 0x4200aa0e:0x4080f320 0x4200b026:0x4080f370 0x4200aee8:0x4080f3d0 0x4200b66a:0x4080f460 0x4200b4de:0x4080f4d0 0x4200abca:0x4080f4f0 0x4200b2ec:0x4080f500 0x4203d740:0x4080f520 0x42000210:0x4080f540 0x42000220:0x4080f5e0 0x42000230:0x4080f5f0 0x4200026c:0x4080f600 0x4200003e:0x4080f610 0x420002ea:0x4080f620 0x4201e8e6:0x4080f630 0x420002d0:0x4080f650 0x4204d0e8:0x4080f660 0xfffffffe:0x4080f680
0x40800eba - panic_abort
    at /home/timo/esp/esp-idf/components/esp_system/panic.c:472
0x4080f1a0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080413e - esp_system_abort
    at /home/timo/esp/esp-idf/components/esp_system/port/esp_system_chip.c:93
0x4080f1a0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x40807e6c - abort
    at /home/timo/esp/esp-idf/components/newlib/abort.c:38
0x4080f1b0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x42044d2e - abort_return<int>
    at /home/timo/esp/esp-idf/components/cxx/cxx_exception_stubs.cpp:33
0x4080f220 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4201edae - std::backtrace_rs::backtrace::libunwind::trace
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/../../backtrace/src/backtrace/libunwind.rs:104
0x4080f230 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4200ab30 - std::sys_common::backtrace::_print_fmt
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:68
0x4080f240 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x42038d74 - core::fmt::rt::Argument::fmt
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/fmt/rt.rs:142
0x4080f2b0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x42020414 - std::io::Write::write_fmt
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/io/mod.rs:1810
0x4080f300 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4200aa0e - std::sys_common::backtrace::_print
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:47
0x4080f320 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4200b026 - std::panicking::default_hook::{{closure}}
    at ??:??
0x4080f370 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4200aee8 - std::panicking::default_hook
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:292
0x4080f3d0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4200b66a - std::panicking::rust_panic_with_hook
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:779
0x4080f460 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4200b4de - std::panicking::begin_panic_handler::{{closure}}
    at ??:??
0x4080f4d0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4200abca - std::sys_common::backtrace::__rust_end_short_backtrace
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:171
0x4080f4f0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4200b2ec - rust_begin_unwind
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:645
0x4080f500 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4203d740 - core::panicking::panic_fmt
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panicking.rs:72
0x4080f520 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x42000210 - backtrace_std::c
    at /tmp/backtrace-std/src/main.rs:11
0x4080f540 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x42000220 - backtrace_std::b
    at /tmp/backtrace-std/src/main.rs:16
0x4080f5e0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x42000230 - backtrace_std::a
    at /tmp/backtrace-std/src/main.rs:20
0x4080f5f0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4200026c - backtrace_std::main
    at /tmp/backtrace-std/src/main.rs:28
0x4080f600 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4200003e - core::ops::function::FnOnce::call_once
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:250
0x4080f610 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x420002ea - std::rt::lang_start::{{closure}}
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:166
0x4080f620 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4201e8e6 - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:284
0x4080f630 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x420002d0 - std::rt::lang_start
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:165
0x4080f650 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4204d0e8 - main_task
    at /home/timo/esp/esp-idf/components/freertos/app_startup.c:208
0x4080f660 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0xfffffffe - s_rtc_timer_retain_mem
    at ??:??
0x4080f680 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
Backtrace ended abruptly: cannot find DWARF information for instruction at address 0xfffffffe
0xfffffffe - s_rtc_timer_retain_mem
    at ??:??

the panic output now contains main, a, b and c, and is much more concise. 👍

(note: It ends with the error Backtrace ended abruptly: cannot find DWARF information for instruction at address 0xfffffffe)

@t-moe
Copy link
Author

t-moe commented Jan 9, 2024

Conclusion

Coming from esp-hal, I find the backtraces on esp-idf-sys extremely painful to handle. They contain a lot of noise, and unless i enable DWARF generation, they will not contain any of my functions in question or only some of them. Surely there must be a way to get a clear and concise backtrace?

How can we improve the situation?

When printing with esp_backtrace, why dont I see my rust functions a,b and c, when using esp-idf-sys instead of esp-hal. Have my frame pointers suddenly disappeared?

Thank you!

@ivmarkov
Copy link
Collaborator

ivmarkov commented Jan 9, 2024

Your original experiment, in the original issue report:

espflash run --monitor enhanced output: contains main, but neither a, b or c and is really verbose

This is because the panic output from ESP IDF contains a stack memory dump, that - unluckily - is not formatted with regular byte-oriented hexdump i.e. 0a 0c 22 34 but with clusters of 4 bytes each where each such cluster looks like (but isn't necessarily) a 32bit value.

My guess is that espflash blindly applies stacktrace decoding on the memory dump, assuming that each such 4-byte cluster is a properly decoded 32 bit word. Which seems not to be the case. Hence the noise.

@ivmarkov
Copy link
Collaborator

ivmarkov commented Jan 9, 2024

Experiment 1: Add esp_backtrace to get more/alternative output.
Experiment 2: Add #[inline(never)]

esp_backtrace contains main now, but not a,b,c

I don't have explanation for that. That's weird, because from LLVM's POV, the two targets look exactly the same.
Maybe following question: what are your optimization settings? (opt-level)? 0, 1, 2 or 3? It could be, that with opt-level != 0, even if you say "inline(never)", stuff is still sometimes inlined, though then I don't have an explanation why the b symbol then pops up in the stack memory dump below.
UPDATE: Seems you are running with level "z" (optimize for size, but still inline). What happens if you change this to 0?
UPDATE 2: Not clear from the no_std Cargo.toml, but it seems you are running with opt-level 0 (in the absence of anything else specified)? If so, this is not apples-to-apples comparison, and would explain the differences in the output from esp_backtrace

panic output contains main, a and b but is still very noisy and not really helpful

Sure. As per above, espflash crawls though the stack memory and blindly applies decoding to everything which looks like a 32 bit word. Including to stuff which is is definitely not an IP (instruction pointer), but data.

Suggestion for improvement: the noise might get reduced if the decoding is applied by espflash only on 32bit words which start from 0x4000000. If I'm not mistaken, ESP programs start from this address.

@ivmarkov
Copy link
Collaborator

ivmarkov commented Jan 9, 2024

Experiment 3: Set RUST_BACKTRACE=1

There IS a difference! :)

Snippet from experiment 2:

thread 'main' panicked at src/main.rs:11:5:
oh no
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

abort() was called at PC 0x4202ac81 on core 0

Snippet from experiment 3:

thread 'main' panicked at src/main.rs:11:5:
oh no
stack backtrace:

abort() was called at PC 0x420444b1 on core 0

Where originally it used to say

note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

now it says:

stack backtrace:

stack backtrace: is coming from the Rust stack walker in the panic crate. It is "just" that the stack trace generated by the panic crate is completely empty. Well, apparently the thing does not work with riscv frame pointers enabled (unlike esp_backtrace!). It does work with xtensa frame pointers enabled though. Weird. Maybe we need to upstream a patch into it!

@igrr
Copy link

igrr commented Jan 9, 2024

espflash crawls though the stack memory and blindly applies decoding to everything which looks like a 32 bit word. Including to stuff which is is definitely not an IP (instruction pointer), but data.

Another idea could be to run GDB (if installed) and actually decode the backtrace based on the stack and the registers, like https://github.com/espressif/esp-idf-panic-decoder/blob/master/esp_idf_panic_decoder/gdb_panic_server.py does.

@t-moe
Copy link
Author

t-moe commented Jan 9, 2024

UPDATE: Seems you are running with level "z" (optimize for size, but still inline). What happens if you change this to 0?
UPDATE 2: Not clear from the no_std Cargo.toml, but it seems you are running with opt-level 0 (in the absence of anything else specified)? If so, this is not apples-to-apples comparison, and would explain the differences in the output from esp_backtrace

Aaah, good catch. 🎉
This at least fixes the esp_backtrace case when used in combination with std::panic::set_hook

Experiment 1b

Instead of using
opt-level = "z"
lets use
opt-level = 0

and then invoke esp_backtrace in as a panic hook.

main.rs:

fn c() {
    panic!("oh no");
}

fn b() {
    c();
}
fn a() {
    b();
}


fn main() {
    esp_idf_svc::sys::link_patches();
    esp_idf_svc::log::EspLogger::initialize_default();
    std::panic::set_hook(Box::new(|info| {
        println!("panic occurred: {:?}", info);
            let bt = esp_backtrace::arch::backtrace();
            for a in bt {
                if let Some(a) = a {
                    println!("esp_backtrace: 0x{:x}", a);
                }
            }
    }));
    a();
}

espflash now outputs:

panic occurred: PanicInfo { payload: Any { .. }, message: Some(oh no), location: Location { file: "src/main.rs", line: 3, col: 5 }, can_unwind: true, force_no_backtrace: false }
esp_backtrace: 0x420ad352
0x420ad352 - std::panicking::rust_panic_with_hook
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:783
esp_backtrace: 0x420ad094
0x420ad094 - std::panicking::begin_panic_handler::{{closure}}
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:658
esp_backtrace: 0x4205c714
0x4205c714 - std::sys_common::backtrace::__rust_end_short_backtrace
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:174
esp_backtrace: 0x420ace32
0x420ace32 - std::panicking::begin_panic_handler::FormatStringPayload::new
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:600
esp_backtrace: 0x420f4e24
0x420f4e24 - core::panicking::panic_fmt
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panicking.rs:54
esp_backtrace: 0x42000230
0x42000230 - backtrace_std::b
    at /tmp/backtrace-std/src/main.rs:6
esp_backtrace: 0x42000240
0x42000240 - backtrace_std::b
    at /tmp/backtrace-std/src/main.rs:8
esp_backtrace: 0x42000258
0x42000258 - backtrace_std::a
    at /tmp/backtrace-std/src/main.rs:11
esp_backtrace: 0x4200029c
0x4200029c - backtrace_std::main
    at /tmp/backtrace-std/src/main.rs:27
esp_backtrace: 0x42000a1a
0x42000a1a - core::ops::function::FnOnce::call_once
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:250

abort() was called at PC 0x420ae48d on core 0

Full espflash enhanced output
panic occurred: PanicInfo { payload: Any { .. }, message: Some(oh no), location: Location { file: "src/main.rs", line: 3, col: 5 }, can_unwind: true, force_no_backtrace: false }
esp_backtrace: 0x420ad352
0x420ad352 - std::panicking::rust_panic_with_hook
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:783
esp_backtrace: 0x420ad094
0x420ad094 - std::panicking::begin_panic_handler::{{closure}}
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:658
esp_backtrace: 0x4205c714
0x4205c714 - std::sys_common::backtrace::__rust_end_short_backtrace
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:174
esp_backtrace: 0x420ace32
0x420ace32 - std::panicking::begin_panic_handler::FormatStringPayload::new
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:600
esp_backtrace: 0x420f4e24
0x420f4e24 - core::panicking::panic_fmt
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panicking.rs:54
esp_backtrace: 0x42000230
0x42000230 - backtrace_std::b
    at /tmp/backtrace-std/src/main.rs:6
esp_backtrace: 0x42000240
0x42000240 - backtrace_std::b
    at /tmp/backtrace-std/src/main.rs:8
esp_backtrace: 0x42000258
0x42000258 - backtrace_std::a
    at /tmp/backtrace-std/src/main.rs:11
esp_backtrace: 0x4200029c
0x4200029c - backtrace_std::main
    at /tmp/backtrace-std/src/main.rs:27
esp_backtrace: 0x42000a1a
0x42000a1a - core::ops::function::FnOnce::call_once
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:250

abort() was called at PC 0x420ae48d on core 0
0x420ae48d - panic_abort::__rust_start_panic::abort
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/panic_abort/src/lib.rs:43
Core  0 register dump:
MEPC    : 0x40800fde  RA      : 0x40804ad8  SP      : 0x408104c0  GP      : 0x4080ab20  
0x40800fde - panic_abort
    at /home/timo/esp/esp-idf/components/esp_system/panic.c:472
0x40804ad8 - __ubsan_include
    at /home/timo/esp/esp-idf/components/esp_system/ubsan.c:313
0x408104c0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080ab20 - coex_pti_tab
    at ??:??
TP      : 0x407eb664  T0      : 0x37363534  T1      : 0x7271706f  T2      : 0x33323130  
0x407eb664 - ??
    at ??:??
0x37363534 - ??
    at ??:??
0x7271706f - s_rtc_timer_retain_mem
    at ??:??
0x33323130 - ??
    at ??:??
S0/FP   : 0x00000004  S1      : 0x40810524  A0      : 0x408104ec  A1      : 0x40810522  
0x00000004 - ??
    at ??:??
0x40810524 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x408104ec - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x40810522 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
A2      : 0x00000000  A3      : 0x40810519  A4      : 0x4080bfce  A5      : 0x4080bfd6  
0x00000000 - ??
    at ??:??
0x40810519 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080bfce - xIsrStack
    at ??:??
0x4080bfd6 - xIsrStack
    at ??:??
A6      : 0x00000000  A7      : 0x76757473  S2      : 0x00000000  S3      : 0x00000000  
0x00000000 - ??
    at ??:??
0x76757473 - s_rtc_timer_retain_mem
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
S4      : 0x00000000  S5      : 0x00000000  S6      : 0x00000000  S7      : 0x00000000  
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
S8      : 0x00000000  S9      : 0x00000000  S10     : 0x00000000  S11     : 0x00000000  
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
T3      : 0x6e6d6c6b  T4      : 0x6a696867  T5      : 0x66656463  T6      : 0x62613938  
0x6e6d6c6b - s_rtc_timer_retain_mem
    at ??:??
0x6a696867 - s_rtc_timer_retain_mem
    at ??:??
0x66656463 - s_rtc_timer_retain_mem
    at ??:??
0x62613938 - s_rtc_timer_retain_mem
    at ??:??
MSTATUS : 0x00001881  MTVEC   : 0x40800001  MCAUSE  : 0x00000007  MTVAL   : 0x00000000  
0x00001881 - ??
    at ??:??
0x40800001 - _vector_table
    at ??:??
0x00000007 - ??
    at ??:??
0x00000000 - ??
    at ??:??
MHARTID : 0x00000000  
0x00000000 - ??
    at ??:??

Stack memory:
408104c0: 0x40811600 0x408116d4 0x40810520 0x4080916c 0x00000000 0x408116d4 0x00000001 0x4080b478
0x40811600 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x408116d4 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x40810520 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080916c - abort
    at /home/timo/esp/esp-idf/components/newlib/abort.c:35
0x00000000 - ??
    at ??:??
0x408116d4 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000001 - ??
    at ??:??
0x4080b478 - __func__.0
    at ??:??
408104e0: 0x40810524 0x4080b494 0x40810520 0x726f6261 0x20292874 0x20736177 0x6c6c6163 0x61206465
0x40810524 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080b494 - __func__.0
    at ??:??
0x40810520 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x726f6261 - s_rtc_timer_retain_mem
    at ??:??
0x20292874 - ??
    at ??:??
0x20736177 - ??
    at ??:??
0x6c6c6163 - s_rtc_timer_retain_mem
    at ??:??
0x61206465 - s_rtc_timer_retain_mem
    at ??:??
40810500: 0x43502074 0x34783020 0x65613032 0x20643834 0x63206e6f 0x2065726f 0x00000030 0x42060000
0x43502074 - esp_system_init_fn_init_components0
    at ??:??
0x34783020 - ??
    at ??:??
0x65613032 - s_rtc_timer_retain_mem
    at ??:??
0x20643834 - ??
    at ??:??
0x63206e6f - s_rtc_timer_retain_mem
    at ??:??
0x2065726f - ??
    at ??:??
0x00000030 - ??
    at ??:??
0x42060000 - core::ptr::unique::Unique<T>::cast
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/unique.rs:141
40810520: 0x40810030 0x61303234 0x64383465 0x40811600 0x00000001 0x00000000 0x40810550 0x420ae490
0x40810030 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x61303234 - s_rtc_timer_retain_mem
    at ??:??
0x64383465 - s_rtc_timer_retain_mem
    at ??:??
0x40811600 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000001 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x40810550 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x420ae490 - <core::ops::range::RangeTo<usize> as core::slice::index::SliceIndex<[T]>>::get
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/slice/index.rs:418
40810540: 0x4080a32c 0x00000000 0x40810560 0x420ae480 0x40810728 0x4211dcf4 0x408105c0 0x420ad430
0x4080a32c - _ZN3std9panicking11panic_count17LOCAL_PANIC_COUNT7__getit5__KEY17h0e7eca99d80651bbE
    at ??:??
0x00000000 - ??
    at ??:??
0x40810560 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x420ae480 - panic_abort::__rust_start_panic::abort
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/panic_abort/src/lib.rs:42
0x40810728 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4211dcf4 - str.1
    at ??:??
0x408105c0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x420ad430 - rust_panic
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:831
40810560: 0x408116d4 0x00000001 0x408105a0 0x42039422 0x408104f8 0x408115d8 0x000e6bec 0x0080b684
0x408116d4 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000001 - ??
    at ??:??
0x408105a0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x42039422 - std::thread::local::LocalKey<T>::try_with
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/local.rs:270
0x408104f8 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x408115d8 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x000e6bec - ??
    at ??:??
0x0080b684 - ??
    at ??:??
40810580: 0x42115578 0x00000046 0x421154bf 0x4211000c 0x421154bf 0x00000046 0x408105b0 0x40810728
0x42115578 - str.6
    at ??:??
0x00000046 - ??
    at ??:??
0x421154bf - str.6
    at ??:??
0x4211000c - intr_handler_set
    at ??:??
0x421154bf - str.6
    at ??:??
0x00000046 - ??
    at ??:??
0x408105b0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x40810728 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
408105a0: 0x4211dcf4 0x408105b0 0x408105c0 0x420699de 0x4080b68c 0x4080b684 0x40810700 0x420ad3d0
0x4211dcf4 - str.1
    at ??:??
0x408105b0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x408105c0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x420699de - std::panicking::panic_count::finished_panic_hook
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:412
0x4080b68c - _ZN3std9panicking4HOOK17h2008636bd958d835E
    at ??:??
0x4080b684 - _ZN3std9panicking4HOOK17h2008636bd958d835E
    at ??:??
0x40810700 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x420ad3d0 - std::panicking::rust_panic_with_hook
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:797
408105c0: 0x408115d8 0x408115d8 0x40810650 0x00000001 0x40810650 0x00000001 0x40810728 0x40810678
0x408115d8 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x408115d8 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x40810650 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000001 - ??
    at ??:??
0x40810650 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000001 - ??
    at ??:??
0x40810728 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x40810678 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
408105e0: 0x00000000 0x00000000 0x00000000 0x40022498 0x40810694 0x4080b68c 0x4211021c 0x408107d0
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x40022498 - ??
    at ??:??
0x40810694 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080b68c - _ZN3std9panicking4HOOK17h2008636bd958d835E
    at ??:??
0x4211021c - esp_app_desc
    at ??:??
0x408107d0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
40810600: 0x4211dcf4 0x40810728 0x00000000 0x00000001 0x00000000 0x00000002 0x40811664 0x408051c0
0x4211dcf4 - str.1
    at ??:??
0x40810728 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000000 - ??
    at ??:??
0x00000001 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x00000002 - ??
    at ??:??
0x40811664 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x408051c0 - xQueueGenericSend
    at /home/timo/esp/esp-idf/components/freertos/FreeRTOS-Kernel/queue.c:1057
40810620: 0x00000000 0x00000000 0x40811664 0x00000000 0x00000804 0x00000054 0x40811654 0x00000000
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x40811664 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000000 - ??
    at ??:??
0x00000804 - ??
    at ??:??
0x00000054 - ??
    at ??:??
0x40811654 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000000 - ??
    at ??:??
40810640: 0x00000000 0x00000000 0x408115fc 0x40804bf6 0x00000000 0x00000000 0x408115fc 0x408051c0
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x408115fc - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x40804bf6 - prvCopyDataToQueue
    at /home/timo/esp/esp-idf/components/freertos/FreeRTOS-Kernel/queue.c:2470
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x408115fc - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x408051c0 - xQueueGenericSend
    at /home/timo/esp/esp-idf/components/freertos/FreeRTOS-Kernel/queue.c:1057
40810660: 0x408115c8 0x00000004 0x40811654 0x00000000 0x408115c8 0x4080b684 0x40810728 0x4211dce4
0x408115c8 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000004 - ??
    at ??:??
0x40811654 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000000 - ??
    at ??:??
0x408115c8 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4080b684 - _ZN3std9panicking4HOOK17h2008636bd958d835E
    at ??:??
0x40810728 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4211dce4 - str.1
    at ??:??
40810680: 0x408107d0 0x4211021c 0x00000001 0x4080b68c 0x4080b684 0x00000000 0x4080b68c 0x4080b684
0x408107d0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4211021c - esp_app_desc
    at ??:??
0x00000001 - ??
    at ??:??
0x4080b68c - _ZN3std9panicking4HOOK17h2008636bd958d835E
    at ??:??
0x4080b684 - _ZN3std9panicking4HOOK17h2008636bd958d835E
    at ??:??
0x00000000 - ??
    at ??:??
0x4080b68c - _ZN3std9panicking4HOOK17h2008636bd958d835E
    at ??:??
0x4080b684 - _ZN3std9panicking4HOOK17h2008636bd958d835E
    at ??:??
408106a0: 0x408115c8 0x40810678 0x00000000 0x420fdc7c 0x00000000 0x00000000 0x40810710 0x420adbce
0x408115c8 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x40810678 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000000 - ??
    at ??:??
0x420fdc7c - pthread_rwlock_unlock
    at /home/timo/esp/esp-idf/components/pthread/pthread_rwlock.c:311
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x40810710 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x420adbce - std::sys::unix::locks::pthread_rwlock::AllocatedRwLock::raw_unlock
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/locks/pthread_rwlock.rs:47
408106c0: 0x408115d8 0x408115d8 0x408106d0 0x40810728 0x4211dcf4 0x408107d0 0x4211021c 0x00010005
0x408115d8 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x408115d8 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x408106d0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x40810728 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4211dcf4 - str.1
    at ??:??
0x408107d0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4211021c - esp_app_desc
    at ??:??
0x00010005 - ??
    at ??:??
408106e0: 0x4080b68c 0x00000001 0x421108f0 0x00000000 0x408107d0 0x408107a0 0x40810750 0x420ad094
0x4080b68c - _ZN3std9panicking4HOOK17h2008636bd958d835E
    at ??:??
0x00000001 - ??
    at ??:??
0x421108f0 - str.0
    at ??:??
0x00000000 - ??
    at ??:??
0x408107d0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x408107a0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x40810750 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x420ad094 - std::panicking::begin_panic_handler::{{closure}}
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:658
40810700: 0x42110de4 0x408115d8 0x40810770 0x420ae158 0x408107d0 0x4211021c 0x00000001 0x40810770
0x42110de4 - str.1
    at ??:??
0x408115d8 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x40810770 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x420ae158 - std::sys::unix::locks::pthread_rwlock::RwLock::write_unlock
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/locks/pthread_rwlock.rs:194
0x408107d0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4211021c - esp_app_desc
    at ??:??
0x00000001 - ??
    at ??:??
0x40810770 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
40810720: 0x42110200 0x00000005 0x42110200 0x00000005 0x00000000 0x4000b68c 0x40810750 0x4204ef12
0x42110200 - esp_app_desc
    at ??:??
0x00000005 - ??
    at ??:??
0x42110200 - esp_app_desc
    at ??:??
0x00000005 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x4000b68c - ??
    at ??:??
0x40810750 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4204ef12 - std::thread::panicking
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:799
40810740: 0x42110200 0x00000005 0x40810760 0x4205c714 0x408107d0 0x408107d0 0x40810790 0x420ace32
0x42110200 - esp_app_desc
    at ??:??
0x00000005 - ??
    at ??:??
0x40810760 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4205c714 - std::sys_common::backtrace::__rust_end_short_backtrace
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:174
0x408107d0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x408107d0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x40810790 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x420ace32 - std::panicking::begin_panic_handler::FormatStringPayload::new
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:600
40810760: 0x408115e4 0x408115e4 0x408107a0 0x4211021c 0x408107d0 0x408107a0 0x4211021c 0x408107a0
0x408115e4 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x408115e4 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x408107a0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4211021c - esp_app_desc
    at ??:??
0x408107d0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x408107a0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4211021c - esp_app_desc
    at ??:??
0x408107a0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
40810780: 0x4211021c 0x408107d0 0x408107c0 0x420f4e24 0x00000000 0x408107a0 0x4211021c 0x408107d0
0x4211021c - esp_app_desc
    at ??:??
0x408107d0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x408107c0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x420f4e24 - core::panicking::panic_fmt
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panicking.rs:54
0x00000000 - ??
    at ??:??
0x408107a0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4211021c - esp_app_desc
    at ??:??
0x408107d0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
408107a0: 0x4212b23c 0x4212b23c 0x408107d0 0x4211021c 0x42110001 0x408107d0 0x408107f0 0x42000230
0x4212b23c - str.0
    at ??:??
0x4212b23c - str.0
    at ??:??
0x408107d0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4211021c - esp_app_desc
    at ??:??
0x42110001 - intr_handler_set
    at ??:??
0x408107d0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x408107f0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x42000230 - backtrace_std::b
    at /tmp/backtrace-std/src/main.rs:6
408107c0: 0x00000001 0x00000001 0x40810810 0x408107d0 0x42110208 0x00000001 0x421108f0 0x00000000
0x00000001 - ??
    at ??:??
0x00000001 - ??
    at ??:??
0x40810810 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x408107d0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x42110208 - esp_app_desc
    at ??:??
0x00000001 - ??
    at ??:??
0x421108f0 - str.0
    at ??:??
0x00000000 - ??
    at ??:??
408107e0: 0x00000000 0x420e0c94 0x40810800 0x42000240 0x4080b684 0x00000000 0x40810810 0x42000258
0x00000000 - ??
    at ??:??
0x420e0c94 - core::ptr::non_null::NonNull<[T]>::as_mut_ptr
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/non_null.rs:1612
0x40810800 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x42000240 - backtrace_std::b
    at /tmp/backtrace-std/src/main.rs:8
0x4080b684 - _ZN3std9panicking4HOOK17h2008636bd958d835E
    at ??:??
0x00000000 - ??
    at ??:??
0x40810810 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x42000258 - backtrace_std::a
    at /tmp/backtrace-std/src/main.rs:11
40810800: 0x00000000 0x00000000 0x40810820 0x4200029c 0x408108a4 0x408108a4 0x40810830 0x42000a1a
0x00000000 - ??
    at ??:??
0x00000000 - ??
    at ??:??
0x40810820 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4200029c - backtrace_std::main
    at /tmp/backtrace-std/src/main.rs:27
0x408108a4 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x408108a4 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x40810830 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x42000a1a - core::ops::function::FnOnce::call_once
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:250
40810820: 0x421154bf 0x42000260 0x40810840 0x420003e8 0x01811468 0x42000260 0x40810850 0x4200102c
0x421154bf - str.6
    at ??:??
0x42000260 - backtrace_std::main
    at /tmp/backtrace-std/src/main.rs:14
0x40810840 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x420003e8 - std::sys_common::backtrace::__rust_begin_short_backtrace
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:158
0x01811468 - ??
    at ??:??
0x42000260 - backtrace_std::main
    at /tmp/backtrace-std/src/main.rs:14
0x40810850 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4200102c - std::rt::lang_start::{{closure}}
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:166
40810840: 0x40810944 0x40810a00 0x40810870 0x4208f1e8 0x408108a4 0x408108a4 0x40810870 0x4208ded4
0x40810944 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x40810a00 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x40810870 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4208f1e8 - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:285
0x408108a4 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x408108a4 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x40810870 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x4208ded4 - core::mem::manually_drop::ManuallyDrop<T>::take
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/mem/manually_drop.rs:114
40810860: 0x40810a00 0x42110748 0x408108a0 0x420acd2c 0xa5a5a5a5 0x00000000 0x408108a4 0x408108a4
0x40810a00 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x42110748 - esp_app_desc
    at ??:??
0x408108a0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x420acd2c - std::panicking::try::do_call
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:552
0xa5a5a5a5 - s_rtc_timer_retain_mem
    at ??:??
0x00000000 - ??
    at ??:??
0x408108a4 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x408108a4 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
40810880: 0x408108a4 0x408108a4 0x40810a00 0x42110748 0x40810944 0x00000004 0x408108e0 0x420acb08
0x408108a4 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x408108a4 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x40810a00 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x42110748 - esp_app_desc
    at ??:??
0x40810944 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x00000004 - ??
    at ??:??
0x408108e0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x420acb08 - std::panicking::try
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:516
408108a0: 0x408108f8 0x40810a00 0x42110748 0x40810a00 0x42110748 0x408108a4 0x408108e0 0x42099f48
0x408108f8 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x40810a00 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x42110748 - esp_app_desc
    at ??:??
0x40810a00 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x42110748 - esp_app_desc
    at ??:??
0x408108a4 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x408108e0 - _ZN12_GLOBAL__N_110eh_globalsE
    at ??:??
0x42099f48 - core::intrinsics::is_aligned_and_not_null
    at /home/timo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/intrinsics.rs:2564

@ivmarkov
Copy link
Collaborator

ivmarkov commented Jan 9, 2024

Experiment 4: Enable DWARF by setting CONFIG_ESP_SYSTEM_USE_EH_FRAME=y

For me, this works. The output from espflash is still very noisy, because espflash seems to apply IP decoding on everything. So e.g. in Backtrace: 0x40800eba:0x4080f1a0 ... it decodes not just 0x40800eba but also 0x4080f1a0 (what is this?) which is wrong and results in noisiness.

@ivmarkov
Copy link
Collaborator

ivmarkov commented Jan 9, 2024

Conclusion
Coming from esp-hal, I find the backtraces on esp-idf-sys extremely painful to handle. They contain a lot of noise, and unless i enable DWARF generation, they will not contain any of my functions in question or only some of them. Surely there must be a way to get a clear and concise backtrace?

Let's unpack this a bit for the sake of being precise:

  • Situation only an issue with riscv chips, not xtensa (ok, pretty bad as everything new is riscv anyway)
  • Noisiness comes from the unfortunate combination of espflash monitor eagerly applying decoding on everything which looks like a memory address + the raw memory dump generated by ESP IDF on riscv*. Not specific to ESP IDF, or esp-idf-sys per se

How can we improve the situation?

Let's put the "noisiness" problem aside for the moment.
Let's look at the three supported stack-walking schemes:

1. DWARF based (solved for me)

  • I think this works. Thanks again to @Vollbrecht for "-C", "force-unwind-tables=yes"

Action item: document this approach better somewhere

2. riscv frame-pointer based

  • I think that can be made to work for Rust code LLVM
  • I don't think we can make it work for GCC, as I have been unable to force riscv frame pointers with GCC
  • I'm not sure what would happen with stack walking if you have a mixed Rust/C stack, where some frames have frame pointers, and others - don't. If this confuses the stack walker, "I think that can be made to work for Rust code" would be an exaggeration for code that calls C and then the C code calls back Rust code

Action item: We can try to upstream the esp_backtrace code in Rust's unwind crate. But only if "it is worth it", i.e. mixed Rust/C stacks can be at least partially decoded (at least the Rust part)

3. GDB decoding of the memory stack frame

  • Will "just work" for ESP IDF with default settings
  • Needs us to resurrect the project that wrapped the idf.py tooling (cargo espidf ...)
    OR
  • Needs us to enhance espflash with what amounts to a port of the Python code that @igrr mentioned above. Seems to me a non-trivial effort

Action item: Work on the first or second option

When printing with esp_backtrace, why dont I see my rust functions a,b and c, when using esp-idf-sys instead of esp-hal. Have my frame pointers suddenly disappeared?

Different optimization settings, as per above?

The noisiness problem

I think we should postpone this for a while, until we figure out whether we need to concentrate on (2) or (3).

@t-moe
Copy link
Author

t-moe commented Jan 9, 2024

Oops. I was a bit too eagerly creating esp-rs/espflash#543, seeing your conclusion here ^^.

Can you take over, and create the remaining issues/tasks based on your action items?

@Vollbrecht
Copy link
Collaborator

Vollbrecht commented Jan 9, 2024

  1. GDB decoding of the memory stack frame

Will "just work" for ESP IDF with default settings
Needs us to resurrect the project that wrapped the idf.py tooling (cargo espidf ...)
OR
Needs us to enhance espflash with what amounts to a port of the Python code that igrr mentioned above. Seems to me a non-trivial effort

i think for gdb and stuff as long as you are not talking about the gdb-stub over uart rather connecting external gdb server, we already have a working solution with probe-rs. So it think we don't need espflash for that

@ivmarkov
Copy link
Collaborator

ivmarkov commented Jan 9, 2024

  1. GDB decoding of the memory stack frame

Will "just work" for ESP IDF with default settings
Needs us to resurrect the project that wrapped the idf.py tooling (cargo espidf ...)
OR
Needs us to enhance espflash with what amounts to a port of the Python code that igrr mentioned above. Seems to me a non-trivial effort

i think for gdb and stuff as long as you are not talking about the gdb-stub over uart rather connecting external gdb server, we already have a working solution with probe-rs. So it think we don't need espflash for that

No it is not gdb-over-uart. It is about (ab)using GDB to do the stack walking for us when presented with
(a) The stack memory (3K or so)
(b) The registers as they were valid for the stack topmost frame
(c) The original ELF file (for DWARF tables & stuff)

Basically, we have to either find a way to run this script and the ESP IDF monitor in general
or
translate the above Python script (and maybe a few accompanying ones) to Rust, and then use from within the espflash monitor code.

@ivmarkov
Copy link
Collaborator

ivmarkov commented Jan 9, 2024

And given that the whole stack memory + registers' dump is so ESP IDF specific, perhaps it would be easier and more natural to just find a way to call idf.py monitor with a cargo sub-command like cargo espidf. Especially given that we have the ESP IDF and all its tooling (we can also force the GDB tool to always be auto-installed by esp-idf-sys) in the .embuild directory.

Which brings back the question/trick from some time ago which is that we have to have at least one successful build of esp-idf-sys so that we get as an output from it all build settings, including the location of ESP IDF and its tooling, as utilized by the project. This task was in the context of cargo espidf menuconfig but it is basically the same for the future cargo espidf monitor that we are discussing here.

@ivmarkov
Copy link
Collaborator

ivmarkov commented Jan 9, 2024

Oops. I was a bit too eagerly creating esp-rs/espflash#543, seeing your conclusion here ^^.

Can you take over, and create the remaining issues/tasks based on your action items?

I think we can keep it simple and have just this task/issue for now. I would only create a separate task for the documentation of the DWARF "trick" in the Rust ESP book.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants