Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Profiling causing tiflash-proxy crash on arm #3553

Closed
SchrodingerZhu opened this issue Nov 26, 2021 · 9 comments
Closed

Profiling causing tiflash-proxy crash on arm #3553

SchrodingerZhu opened this issue Nov 26, 2021 · 9 comments
Labels
affects-5.4 affects-6.0 affects-6.1 component/storage may-affects-4.0 may-affects-5.0 may-affects-5.1 may-affects-5.2 may-affects-5.3 severity/major type/bug Issue for bug

Comments

@SchrodingerZhu
Copy link
Contributor

Bug Report

This may be hard to reproduce. use br to import data of TPCH 100 and run profiling during the process.

image
@SchrodingerZhu SchrodingerZhu added the type/bug Issue for bug label Nov 26, 2021
@solotzg
Copy link
Contributor

solotzg commented Nov 26, 2021

Can you reproduce it after pprof-rs upgraded to 0.6.1 ？#3544

@SchrodingerZhu
Copy link
Contributor Author

I reproduced it again on aarch64 today

[2021/11/27 11:16:04.170 +08:00] [WARN] [<unknown>] ["Application:The configuration \"path\" is deprecated. Check [storage] section for new style."] [thread_id=1]
[2021/11/27 11:36:16.544 +08:00] [ERROR] [<unknown>] [BaseDaemon:########################################] [thread_id=48]
[2021/11/27 11:36:16.544 +08:00] [ERROR] [<unknown>] ["BaseDaemon:(from thread 47) Received signal Segmentation fault (11)."] [thread_id=48]
[2021/11/27 11:36:16.544 +08:00] [ERROR] [<unknown>] ["BaseDaemon:Address: 0x9"] [thread_id=48]
[2021/11/27 11:36:16.544 +08:00] [ERROR] [<unknown>] ["BaseDaemon:Address not mapped to object."] [thread_id=48]
[2021/11/27 11:36:16.544 +08:00] [ERROR] [<unknown>] ["BaseDaemon:0. /lib64/libgcc_s.so.1(+0xce0c) [0xffffa6fdce0c]"] [thread_id=48]
[2021/11/27 11:36:16.544 +08:00] [ERROR] [<unknown>] ["BaseDaemon:1. /lib64/libgcc_s.so.1(_Unwind_Backtrace+0x7c) [0xffffa6fddd6c]"] [thread_id=48]
[2021/11/27 11:36:16.544 +08:00] [ERROR] [<unknown>] ["BaseDaemon:2. /home/schzhu/tidb-deploy/tiflash-19999/bin/tiflash/libtiflash_proxy.so(perf_signal_handler+0xb0) [0xffffa7c7199c]"] [thread_id=48]
[2021/11/27 11:36:16.544 +08:00] [ERROR] [<unknown>] ["BaseDaemon:3. [0xffffa9dd066c]"] [thread_id=48]
[2021/11/27 11:36:16.544 +08:00] [ERROR] [<unknown>] ["BaseDaemon:4. /home/schzhu/tidb-deploy/tiflash-19999/bin/tiflash/libtiflash_proxy.so(+0xdffad8) [0xffffa82afad8]"] [thread_id=48]
[2021/11/27 11:36:16.544 +08:00] [ERROR] [<unknown>] ["BaseDaemon:5. [0x9]"] [thread_id=48]
[2021/11/27 11:36:42.634 +08:00] [WARN] [<unknown>] ["Application:The configuration \"path\" is deprecated. Check [storage] section for new style."] [thread_id=1]

@SchrodingerZhu
Copy link
Contributor Author

my guess tikv/pprof-rs@55d6f87#diff-d36b2d2be6e7c06c1b1c76dbdc82c6e3c15ebb6a02be2a6e7f119f34aa909dd4

@solotzg
Copy link
Contributor

solotzg commented Nov 28, 2021

I reproduced it again on aarch64 today

[2021/11/27 11:16:04.170 +08:00] [WARN] [<unknown>] ["Application:The configuration \"path\" is deprecated. Check [storage] section for new style."] [thread_id=1]
[2021/11/27 11:36:16.544 +08:00] [ERROR] [<unknown>] [BaseDaemon:########################################] [thread_id=48]
[2021/11/27 11:36:16.544 +08:00] [ERROR] [<unknown>] ["BaseDaemon:(from thread 47) Received signal Segmentation fault (11)."] [thread_id=48]
[2021/11/27 11:36:16.544 +08:00] [ERROR] [<unknown>] ["BaseDaemon:Address: 0x9"] [thread_id=48]
[2021/11/27 11:36:16.544 +08:00] [ERROR] [<unknown>] ["BaseDaemon:Address not mapped to object."] [thread_id=48]
[2021/11/27 11:36:16.544 +08:00] [ERROR] [<unknown>] ["BaseDaemon:0. /lib64/libgcc_s.so.1(+0xce0c) [0xffffa6fdce0c]"] [thread_id=48]
[2021/11/27 11:36:16.544 +08:00] [ERROR] [<unknown>] ["BaseDaemon:1. /lib64/libgcc_s.so.1(_Unwind_Backtrace+0x7c) [0xffffa6fddd6c]"] [thread_id=48]
[2021/11/27 11:36:16.544 +08:00] [ERROR] [<unknown>] ["BaseDaemon:2. /home/schzhu/tidb-deploy/tiflash-19999/bin/tiflash/libtiflash_proxy.so(perf_signal_handler+0xb0) [0xffffa7c7199c]"] [thread_id=48]
[2021/11/27 11:36:16.544 +08:00] [ERROR] [<unknown>] ["BaseDaemon:3. [0xffffa9dd066c]"] [thread_id=48]
[2021/11/27 11:36:16.544 +08:00] [ERROR] [<unknown>] ["BaseDaemon:4. /home/schzhu/tidb-deploy/tiflash-19999/bin/tiflash/libtiflash_proxy.so(+0xdffad8) [0xffffa82afad8]"] [thread_id=48]
[2021/11/27 11:36:16.544 +08:00] [ERROR] [<unknown>] ["BaseDaemon:5. [0x9]"] [thread_id=48]
[2021/11/27 11:36:42.634 +08:00] [WARN] [<unknown>] ["Application:The configuration \"path\" is deprecated. Check [storage] section for new style."] [thread_id=1]

tiflash version ?

@SchrodingerZhu
Copy link
Contributor Author

SchrodingerZhu commented Nov 28, 2021 •
edited

master at 239295a

@hehechen
Copy link
Contributor

pprof-rs will panic on arm, have you ever reproduced it on x86?

@SchrodingerZhu
Copy link
Contributor Author

no. i am running this on aarch64

@solotzg solotzg changed the title Profiling causing tiflash-proxy crash Profiling causing tiflash-proxy crash on arm Nov 29, 2021
@SchrodingerZhu SchrodingerZhu mentioned this issue Dec 3, 2021
Closed
@SchrodingerZhu
Copy link
Contributor Author

addressed by pingcap/tidb-engine-ext#64, pingcap/tidb-engine-ext#65

@JaySon-Huang
Copy link
Contributor

similar to #4172

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
affects-5.4 affects-6.0 affects-6.1 component/storage may-affects-4.0 may-affects-5.0 may-affects-5.1 may-affects-5.2 may-affects-5.3 severity/major type/bug Issue for bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@jebter @JaySon-Huang @VelocityLight @solotzg @hehechen @SchrodingerZhu @ti-chi-bot