Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

backtraces broken on the Android bot #17520

Open
thestinger opened this issue Sep 24, 2014 · 27 comments
Open

backtraces broken on the Android bot #17520

thestinger opened this issue Sep 24, 2014 · 27 comments
Labels
C-bug Category: This is a bug. O-android Operating system: Android

Comments

@thestinger
Copy link
Contributor

PIE is now required by Android (#17437) but it breaks backtrace support on the Android bot. It may just need to be updated to the current NDK / Android versions.
@thestinger thestinger added the O-android Operating system: Android label Sep 24, 2014
@tamird
Copy link
Contributor

tamird commented Apr 29, 2015

I've locally enabled PIE for android and tested this with the latest NDK (r10d) generated for android-21 and an android emulator running the latest API (android-22), and it still fails with:

test [run-pass] run-pass/backtrace.rs ... FAILED

failures:

---- [run-pass] run-pass/backtrace.rs stdout ----

error: test run failed!
status: exit code: 101
command: x86_64-apple-darwin/test/run-pass/backtrace.stage2-arm-linux-androideabi
stdout:
------------------------------------------

------------------------------------------
stderr:
------------------------------------------
thread '<main>' panicked at 'bad output: thread '<main>' panicked at 'explicit panic', /Users/tamird/src/rust/src/test/run-pass/backtrace.rs:23
stack backtrace:
   1: 0xb6c53da7 - <unknown>
   2: 0xb6c5ea8b - <unknown>
   3: 0xb6c0df37 - <unknown>
   4: 0xb6f87fb7 - <unknown>
   5: 0xb6f87d73 - <unknown>
   6: 0xb6f8b913 - <unknown>
   7: 0xb6d0dfe3 - <unknown>
   8: 0xb6d0dfbb - <unknown>
   9: 0xb6d0dfbb - <unknown>
  10: 0xb6d0dfbb - <unknown>
  11: 0xb6d0dfbb - <unknown>
  12: 0xb6d0dfbb - <unknown>
  13: 0xb6d0dfbb - <unknown>
  14: 0xb6d0dfbb - <unknown>
  15: 0xb6d0dfbb - <unknown>
  16: 0xb6d0dfbb - <unknown>
  17: 0xb6d0dfbb - <unknown>
  18: 0xb6d0dfbb - <unknown>
  19: 0xb6d0dfbb - <unknown>
  20: 0xb6d0dfbb - <unknown>
  21: 0xb6d0dfbb - <unknown>
  22: 0xb6d0dfbb - <unknown>
  23: 0xb6d0dfbb - <unknown>
  24: 0xb6d0dfbb - <unknown>
  25: 0xb6d0dfbb - <unknown>
  26: 0xb6d0dfbb - <unknown>
  27: 0xb6d0dfbb - <unknown>
  28: 0xb6d0dfbb - <unknown>
  29: 0xb6d0dfbb - <unknown>
  30: 0xb6d0dfbb - <unknown>
  31: 0xb6d0dfbb - <unknown>
  32: 0xb6d0dfbb - <unknown>
  33: 0xb6d0dfbb - <unknown>
  34: 0xb6d0dfbb - <unknown>
  35: 0xb6d0dfbb - <unknown>
  36: 0xb6d0dfbb - <unknown>
  37: 0xb6d0dfbb - <unknown>
  38: 0xb6d0dfbb - <unknown>
  39: 0xb6d0dfbb - <unknown>
  40: 0xb6d0dfbb - <unknown>
  41: 0xb6d0dfbb - <unknown>
  42: 0xb6d0dfbb - <unknown>
  43: 0xb6d0dfbb - <unknown>
  44: 0xb6d0dfbb - <unknown>
  45: 0xb6d0dfbb - <unknown>
  46: 0xb6d0dfbb - <unknown>
  47: 0xb6d0dfbb - <unknown>
  48: 0xb6d0dfbb - <unknown>
  49: 0xb6d0dfbb - <unknown>
  50: 0xb6d0dfbb - <unknown>
  51: 0xb6d0dfbb - <unknown>
  52: 0xb6d0dfbb - <unknown>
  53: 0xb6d0dfbb - <unknown>
  54: 0xb6d0dfbb - <unknown>
  55: 0xb6d0dfbb - <unknown>
  56: 0xb6d0dfbb - <unknown>
  57: 0xb6d0dfbb - <unknown>
  58: 0xb6d0dfbb - <unknown>
  59: 0xb6d0dfbb - <unknown>
  60: 0xb6d0dfbb - <unknown>
  61: 0xb6d0dfbb - <unknown>
  62: 0xb6d0dfbb - <unknown>
  63: 0xb6d0dfbb - <unknown>
  64: 0xb6d0dfbb - <unknown>
  65: 0xb6d0dfbb - <unknown>
  66: 0xb6d0dfbb - <unknown>
  67: 0xb6d0dfbb - <unknown>
  68: 0xb6d0dfbb - <unknown>
  69: 0xb6d0dfbb - <unknown>
  70: 0xb6d0dfbb - <unknown>
  71: 0xb6d0dfbb - <unknown>
  72: 0xb6d0dfbb - <unknown>
  73: 0xb6d0dfbb - <unknown>
  74: 0xb6d0dfbb - <unknown>
  75: 0xb6d0dfbb - <unknown>
  76: 0xb6d0dfbb - <unknown>
  77: 0xb6d0dfbb - <unknown>
  78: 0xb6d0dfbb - <unknown>
  79: 0xb6d0dfbb - <unknown>
  80: 0xb6d0dfbb - <unknown>
  81: 0xb6d0dfbb - <unknown>
  82: 0xb6d0dfbb - <unknown>
  83: 0xb6d0dfbb - <unknown>
  84: 0xb6d0dfbb - <unknown>
  85: 0xb6d0dfbb - <unknown>
  86: 0xb6d0dfbb - <unknown>
  87: 0xb6d0dfbb - <unknown>
  88: 0xb6d0dfbb - <unknown>
  89: 0xb6d0dfbb - <unknown>
  90: 0xb6d0dfbb - <unknown>
  91: 0xb6d0dfbb - <unknown>
  92: 0xb6d0dfbb - <unknown>
  93: 0xb6d0dfbb - <unknown>
  94: 0xb6d0dfbb - <unknown>
  95: 0xb6d0dfbb - <unknown>
  96: 0xb6d0dfbb - <unknown>
  97: 0xb6d0dfbb - <unknown>
  98: 0xb6d0dfbb - <unknown>
  99: 0xb6d0dfbb - <unknown>
  100: 0xb6d0dfbb - <unknown>
 ... <frames omitted>
', /Users/tamird/src/rust/src/test/run-pass/backtrace.rs:54

------------------------------------------

thread '[run-pass] run-pass/backtrace.rs' panicked at 'explicit panic', /Users/tamird/src/rust/src/compiletest/runtest.rs:1525

tamird added a commit to tamird/rust that referenced this issue Apr 29, 2015
@tamird
Use PIE on Android
9f36ec0 
This is OK to do given:
  - PIE is supported on Android starting with API 16.
  - The bots are running API 18.
  - API < 16 now has a 12.5% market share[0] as of 2015-04-29.

Unfortunately, this breaks backtrace support. See rust-lang#17520.

Closes rust-lang#17437.

[0] https://developer.android.com/about/dashboards/index.html
@tamird tamird mentioned this issue Apr 29, 2015
Merged
@steveklabnik
Copy link
Member

Triage: we're in the process of moving all of the bots to Travis/AppVeyor, so I'm not sure if this 1. is still true 2. will persist there.

@roblabla roblabla mentioned this issue Jun 21, 2017
Open
@Mark-Simulacrum Mark-Simulacrum added the C-bug Category: This is a bug. label Jul 22, 2017
@roblabla roblabla mentioned this issue Jul 22, 2017
Closed
@roblabla
Copy link
Contributor

So after a bit of investigation, I found that android internally uses libunwind to handle its unwinding. It might be a good idea to try having rust use that ? If it's a good idea, I might start implementing it.

@DanAlbert
Copy link
Contributor

It's an option. We only used libunwind for a handful of releases. We've started writing our own unwinder since none of the off the shelf options were reliable or performant enough for our needs. That's getting close to ready, and we will be including it in the NDK.

@roblabla
Copy link
Contributor

@DanAlbert Oh, OK. So, if I make FFI bindings to libunwinderstack (when it's in the NDK) and use them in libstd, would I need to statically link to it ?

@DanAlbert
Copy link
Contributor

DanAlbert commented Jul 22, 2017 •
edited

Either statically linked or included in the APK. The former is probably simpler.

@roblabla
Copy link
Contributor

roblabla commented Jul 22, 2017 •
edited

Right. Thanks for the insights :D. I'll try figuring out how libunwinderstack works and hopefully draft a PR up over the next few days.

EDIT: Just found out libunwinderstack is written in CPP. Hopefully that won't hinder me too much, but it is going to be a bit more painful that I had anticipated at first.

@roblabla
Copy link
Contributor

roblabla commented Jul 23, 2017 •
edited

So after some unfruitful attempts at making bindgen and libunwinderstack work together (I couldn't get bindgen to generate constructors for unwinderstack::Maps), I decided to investigate other options.

In doing so, I found out that backtrace-rs has no trouble getting a correct stacktrace. Furthermore, I figured that rust's tracing part seemed correct (when using RUST_BACKTRACE=full and comparing, the stack depth and pc seem correct), so the printing module seems to be at fault.

I'm going to be looking into the differences between backtrace-rs and the built-in backtracing of rust. I already have a few ideas.

@roblabla roblabla mentioned this issue Jul 23, 2017
Closed
@azdlowry
Copy link

azdlowry commented Nov 16, 2017 •
edited

I've been investigating this over the last few days and I think the issue is a missing dl_iterate_phdr function. It looks like that function was removed from android at some point.

@azdlowry
Copy link

Found this in the NDK which might be the issue:

#if __ANDROID_API__ >= 21
int dl_iterate_phdr(int (*)(struct dl_phdr_info*, size_t, void*), void*) __INTRODUCED_IN(21);
#endif /* __ANDROID_API__ >= 21 */

@DanAlbert
Copy link
Contributor

You're missing a small amount of scope there: that's wrapped in an #ifdef __arm__. It's more obvious what's going on if you look at the unprocessed source in bionic.

dl_iterate_phdr is not available for ARM Android (for whatever reason) until android-21 (Lollipop). I don't know the current state of libunwindstack, but the goal is for it to be usable for the NDK (so supporting ICS). If the dl_iterate_phdr call is coming from LLVM's libunwind, there was a patch that was submitted recently to add a fallback implementation if it's not available: https://reviews.llvm.org/D39468.

@azdlowry
Copy link

Thanks. I was trying to work out exactly where that function came from.

I was ignoring non-arm androids, as they are not that common atm.

Is there anything we need to do to test and then consume this fix? Or is just a case of waiting and re-enabling the tests once the change propagates to rust?

@DanAlbert
Copy link
Contributor

DanAlbert commented Nov 17, 2017 •
edited

I was ignoring non-arm androids, as they are not that common atm.

idk about rust developers, but most of the time Android developers will use x86 emulators since they are much faster than an arm emulator.

Is there anything we need to do to test and then consume this fix? Or is just a case of waiting and re-enabling the tests once the change propagates to rust?

Should be automatic once you get the update. Do you periodically update from upstream, or do you track AOSP's copy (a fair number of projects do that for Android libraries) or something? If you're tracking AOSP lmk and I'll get it updated.

@azdlowry
Copy link

We have our own fork of AOSP that we need to bring back towards the official. We would need to bring any change into our own fork.

@DanAlbert
Copy link
Contributor

Okay, I'll try to find some time this week to get AOSP's unwinder updated so you at least get this fix when you sync up with upstream.

@DanAlbert
Copy link
Contributor

Two weeks late, but better than never: https://android-review.googlesource.com/c/platform/external/libunwind_llvm/+/567913

DennySPB pushed a commit to AEXmod/platform_external_libunwind_llvm that referenced this issue Feb 14, 2018
@DanAlbert @DennySPB
libunwind_llvm: Merge to upstream r320528.
3cf0455 
Gets us https://reviews.llvm.org/D39468 which lets us update the
NDK's version of the unwinder.

Test: make native
Test: external/libcxx/run_tests.py  # no regressions
Bug: rust-lang/rust#17520
Change-Id: I1284fb151c00f1ae4350c73ad9033246720d5190
Signed-off-by: mydongistiny <jaysonedson@gmail.com>
miodragdinic pushed a commit to MIPS/external-libunwind_llvm that referenced this issue Mar 14, 2018
@DanAlbert
Merge to upstream r320528.
9ca0ad7 
Gets us https://reviews.llvm.org/D39468 which lets us update the
NDK's version of the unwinder.

Test: make native
Test: external/libcxx/run_tests.py  # no regressions
Bug: rust-lang/rust#17520
Change-Id: I1284fb151c00f1ae4350c73ad9033246720d5190
Hikari-no-Tenshi pushed a commit to MiracleDROID-HnT/android_external_libunwind_llvm that referenced this issue May 6, 2018
@DanAlbert @Hikari-no-Tenshi
libunwind_llvm: Merge to upstream r320528.
83b698d 
Gets us https://reviews.llvm.org/D39468 which lets us update the
NDK's version of the unwinder.

Test: make native
Test: external/libcxx/run_tests.py  # no regressions
Bug: rust-lang/rust#17520
Change-Id: I1284fb151c00f1ae4350c73ad9033246720d5190
Signed-off-by: mydongistiny <jaysonedson@gmail.com>
Eliminater74 pushed a commit to PureFusionOS/android_external_libunwind_llvm that referenced this issue Jun 29, 2018
@DanAlbert @Eliminater74
libunwind_llvm: Merge to upstream r320528.
03a0120 
Gets us https://reviews.llvm.org/D39468 which lets us update the
NDK's version of the unwinder.

Test: make native
Test: external/libcxx/run_tests.py  # no regressions
Bug: rust-lang/rust#17520
Change-Id: I1284fb151c00f1ae4350c73ad9033246720d5190
Signed-off-by: mydongistiny <jaysonedson@gmail.com>
Signed-off-by: Eliminater74 <eliminater74@gmail.com>
@artemii235 artemii235 mentioned this issue Jul 17, 2019
Closed
@katyo
Copy link

katyo commented Dec 24, 2019

Is any progress here?

@realcr
Copy link

realcr commented Mar 17, 2020

I Bumped into this issue myself today. I built the binary with this command:

cargo build --target armv7-linux-androideabi --release

Rust compiler version:

$ rustc --version
rustc 1.41.0 (5e1a79984 2020-01-27)

This is what the backtrace looks like:

I/flutter (22222): ⛔  main - stderr:
I/flutter (22222): thread '<unnamed>' panicked at 'internal error: entered unreachable code', components/funder/src/handler/canceler.rs:309:43
I/flutter (22222): stack backtrace:
I/flutter (22222):    0: <unknown>
I/flutter (22222):    1: <unknown>
I/flutter (22222):    2: <unknown>
I/flutter (22222):    3: <unknown>
I/flutter (22222):    4: <unknown>
I/flutter (22222):    5: <unknown>
I/flutter (22222):    6: <unknown>
I/flutter (22222):    7: <unknown>
I/flutter (22222):    8: <unknown>
I/flutter (22222):    9: <unknown>
I/flutter (22222):   10: <unknown>
I/flutter (22222):   11: <unknown>
I/flutter (22222):   12: <unknown>
I/flutter (22222):   13: <unknown>
I/flutter (22222):   14: <unknown>
I/flutter (22222):   15: <unknown>
I/flutter (22222):   16: <unknown>
I/flutter (22222):   17: <unknown>
I/flutter (22222):   18: <unknown>
I/flutter (22222):   19: <unknown>
I/flutter (22222): 
I/flutter (22222): ⛔  main - stderr:
I/flutter (22222):   20: <unknown>
I/flutter (22222):   21: <unknown>
I/flutter (22222): note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
I/flutter (22222): [2020-03-17T12:50:00Z ERROR stcompact::server_loop] node() error: IndexClientError(AppServerClosed)
I/flutter (22222):

@Mark-Simulacrum
Copy link
Member

Latest output inside the emulator. This seems somewhat promising, as at least some symbols are showing up. It's not clear to me why we don't have them for the test case but do for std.

thread 'main' panicked at 'bad output: thread 'main' panicked at 'explicit panic', /checkout/src/test/ui/backtrace.rs:17:9
stack backtrace:
   0: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
   1: core::fmt::write
   2: rust_metadata_std_1b375e3cf1aedcda5e72be7c39da1f61
   3: rust_metadata_std_1b375e3cf1aedcda5e72be7c39da1f61
   4: rust_metadata_std_1b375e3cf1aedcda5e72be7c39da1f61
   5: std::panicking::rust_panic_with_hook
   6: <unknown>
   7: <unknown>
   8: <unknown>
   9: <unknown>
  10: std::rt::lang_start_internal
  11: <unknown>
  12: __libc_init
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
', /checkout/src/test/ui/backtrace.rs:67:5

@s1341
Copy link

s1341 commented Apr 7, 2021

I'm encountering this issue on an Android 11 device... Any luck resolving it? I only get <unknown>.

@s1341
Copy link

s1341 commented Apr 7, 2021

backtrace-rs doesn't work either. unwind at least produces a list of addresses, but crashes with a segfault when it hits a bad address.

@fzyzcjy
Copy link

fzyzcjy commented Oct 12, 2021

Hi, after many years, is there any solution? This is a critical issue, because stack traces are quite important to debug issues in production environments!

I need to at least know a list of addresses (and do not segfault even if hit bad addr!), such that I can copy these addresses and symbolize on my computer.

This was referenced Oct 12, 2021
Closed
Open
Closed
@fzyzcjy
Copy link

fzyzcjy commented Oct 13, 2021

@s1341 Have you solved it? Thanks!

@s1341
Copy link

s1341 commented Oct 14, 2021 •
edited

I think backtraces are working for me... using backtrace-rs.

@fzyzcjy
Copy link

fzyzcjy commented Oct 14, 2021

@s1341 Hi do you use it in production environment, such as --release, or only with debug? I do face problems sometimes there...

@s1341
Copy link

s1341 commented Oct 14, 2021

I use in release. What issues are you facing? Can you be more specific about when you have problems?

@fzyzcjy
Copy link

fzyzcjy commented Oct 14, 2021

@s1341 Hi thanks for your reply! My issue: rust-lang/backtrace-rs#445

@ShootingKing-AM ShootingKing-AM mentioned this issue Apr 13, 2023
Closed
5 tasks
@chenyukang chenyukang mentioned this issue May 15, 2023
Merged
@RalfJung RalfJung mentioned this issue Aug 24, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-bug Category: This is a bug. O-android Operating system: Android
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix android backtraces roblabla/rust