Attach uretprobe causes Go Application crash

[mrproliu]

Hi, I found that when I attach an eBPF program to a Go application with uretprobe, then the Go application crashes. If I only attach uprobe it works well. The Go program only makes network requests, the eBPF program is an empty function and returns 0 only. So I'm not quite sure where the problem is.

Application Server Environment

uname -a: Linux han 5.15.0-43-generic #46~20.04.1-Ubuntu SMP Thu Jul 14 15:20:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
go version: go version go1.18.1 linux/amd64
eBPF framework version: v0.8.1

Program error log

runtime: unexpected return pc for crypto/tls.(*Conn).Read called from 0x7fffffffe000
stack: frame={sp:0xc0004e3c18, fp:0xc0004e3c88} stack=[0xc0004e3000,0xc0004e4000)
0x000000c0004e3b18:  0x0000000000000000  0x0000000000000000
0x000000c0004e3b28:  0x0000000000000000  0x0000000000000000
0x000000c0004e3b38:  0x0000000000000000  0x0000000000591366 <crypto/tls.(*Conn).handshakeContext+0x0000000000000066>
0x000000c0004e3b48:  0x00000000006e7610  0x000000c0000b4000
0x000000c0004e3b58:  0x0000000000000000  0x0000000000000000
0x000000c0004e3b68:  0x0000000000000000  0x0000000000000000
0x000000c0004e3b78:  0x0000000000000000  0x0000000000000000
0x000000c0004e3b88:  0x0000000000000000  0x0000000000000000
0x000000c0004e3b98:  0x0000000000000000  0x0000000000000000
0x000000c0004e3ba8:  0x0000000000000000  0x0000000000000000
0x000000c0004e3bb8:  0x0000000000000000  0x0000000000000000
0x000000c0004e3bc8:  0x0000000000000000  0x0000000000000000
0x000000c0004e3bd8:  0x0000000000000000  0x0000000000000000
0x000000c0004e3be8:  0x0000000000000000  0x0000000000000000
0x000000c0004e3bf8:  0x0000000000000000  0x0000000000000000
0x000000c0004e3c08:  0x000000c0004e3c78  0x0000000000590ba5 <crypto/tls.(*Conn).Read+0x0000000000000065>
0x000000c0004e3c18: <0x000000c000070000  0x00000000006e7610
0x000000c0004e3c28:  0x000000c0000b4000  0x0000000000000000
0x000000c0004e3c38:  0x0000000000000000  0x0000000000000000
0x000000c0004e3c48:  0x0000000000000000  0x0000000000000000
0x000000c0004e3c58:  0x0000000000000000  0x0000000000000000
0x000000c0004e3c68:  0x0000000000000000  0x0000000000000000
0x000000c0004e3c78:  0x000000c0004db4d8 !0x00007fffffffe000
0x000000c0004e3c88: >0x000000c000070000  0x000000c0004d2000
0x000000c0004e3c98:  0x0000000000001000  0x0000000000000000
0x000000c0004e3ca8:  0x0001000000000000  0x00007f85b4c1aaa8
0x000000c0004e3cb8:  0x0000000000000000  0x00007f85dc89d108
0x000000c0004e3cc8:  0x0000000000000060  0x000000c00003cc00
0x000000c0004e3cd8:  0x000000c0004db510  0x00000000004f95e3 <bufio.(*Reader).fill+0x0000000000000103>
0x000000c0004e3ce8:  0x000000c0000d65a0  0x000000c0004d2000
0x000000c0004e3cf8:  0x000000000040695d <runtime.makechan+0x000000000000011d>  0x0000000000000060
0x000000c0004e3d08:  0x0000000000000064  0x000000c0004db530
0x000000c0004e3d18:  0x00000000004f973d <bufio.(*Reader).Peek+0x000000000000005d>  0x000000c000510ba0
0x000000c0004e3d28:  0x000000000061152a <net/http.(*fakeLocker).Unlock+0x000000000000002a>  0x000000c0004db7b8
0x000000c0004e3d38:  0x0000000000609e0c <net/http.(*persistConn).readLoop+0x00000000000001ac>  0x000000c000510ba0
0x000000c0004e3d48:  0x0000000000000001  0x0000000000000000
0x000000c0004e3d58:  0x0000000000000000  0x0000000000000000
0x000000c0004e3d68:  0x0000000000000000  0x0000000000000000
0x000000c0004e3d78:  0x0000000000000000  0x0300000100000000
fatal error: unknown caller pc

runtime stack:
runtime.throw({0x676e7c?, 0x8157c0?})
	/usr/local/go/src/runtime/panic.go:992 +0x71
runtime.gentraceback(0x0?, 0x0?, 0x0?, 0x0?, 0x0, 0x0, 0x7fffffff, 0x6a0b78, 0x0?, 0x0)
	/usr/local/go/src/runtime/traceback.go:254 +0x1a36
runtime.copystack(0xc000083380, 0x800000002?)
	/usr/local/go/src/runtime/stack.go:930 +0x2f5
runtime.newstack()
	/usr/local/go/src/runtime/stack.go:1110 +0x497
runtime.morestack()
	/usr/local/go/src/runtime/asm_amd64.s:547 +0x8b

goroutine 8 [copystack]:
context.WithCancel({0x6e7610?, 0xc0000b4000?})
	/usr/local/go/src/context/context.go:232 +0x135 fp=0xc0004e3b48 sp=0xc0004e3b40 pc=0x4b3fb5
crypto/tls.(*Conn).handshakeContext(0xc000070000, {0x6e7610, 0xc0000b4000})
	/usr/local/go/src/crypto/tls/conn.go:1406 +0x66 fp=0xc0004e3c18 sp=0xc0004e3b48 pc=0x591366
runtime: unexpected return pc for crypto/tls.(*Conn).Read called from 0x7fffffffe000
stack: frame={sp:0xc0004e3c18, fp:0xc0004e3c88} stack=[0xc0004e3000,0xc0004e4000)
0x000000c0004e3b18:  0x0000000000000000  0x0000000000000000
0x000000c0004e3b28:  0x0000000000000000  0x0000000000000000
0x000000c0004e3b38:  0x0000000000000000  0x0000000000591366 <crypto/tls.(*Conn).handshakeContext+0x0000000000000066>
0x000000c0004e3b48:  0x00000000006e7610  0x000000c0000b4000
0x000000c0004e3b58:  0x0000000000000000  0x0000000000000000
0x000000c0004e3b68:  0x0000000000000000  0x0000000000000000
0x000000c0004e3b78:  0x0000000000000000  0x0000000000000000
0x000000c0004e3b88:  0x0000000000000000  0x0000000000000000
0x000000c0004e3b98:  0x0000000000000000  0x0000000000000000
0x000000c0004e3ba8:  0x0000000000000000  0x0000000000000000
0x000000c0004e3bb8:  0x0000000000000000  0x0000000000000000
0x000000c0004e3bc8:  0x0000000000000000  0x0000000000000000
0x000000c0004e3bd8:  0x0000000000000000  0x0000000000000000
0x000000c0004e3be8:  0x0000000000000000  0x0000000000000000
0x000000c0004e3bf8:  0x0000000000000000  0x0000000000000000
0x000000c0004e3c08:  0x000000c0004e3c78  0x0000000000590ba5 <crypto/tls.(*Conn).Read+0x0000000000000065>
0x000000c0004e3c18: <0x000000c000070000  0x00000000006e7610
0x000000c0004e3c28:  0x000000c0000b4000  0x0000000000000000
0x000000c0004e3c38:  0x0000000000000000  0x0000000000000000
0x000000c0004e3c48:  0x0000000000000000  0x0000000000000000
0x000000c0004e3c58:  0x0000000000000000  0x0000000000000000
0x000000c0004e3c68:  0x0000000000000000  0x0000000000000000
0x000000c0004e3c78:  0x000000c0004db4d8 !0x00007fffffffe000
0x000000c0004e3c88: >0x000000c000070000  0x000000c0004d2000
0x000000c0004e3c98:  0x0000000000001000  0x0000000000000000
0x000000c0004e3ca8:  0x0001000000000000  0x00007f85b4c1aaa8
0x000000c0004e3cb8:  0x0000000000000000  0x00007f85dc89d108
0x000000c0004e3cc8:  0x0000000000000060  0x000000c00003cc00
0x000000c0004e3cd8:  0x000000c0004db510  0x00000000004f95e3 <bufio.(*Reader).fill+0x0000000000000103>
0x000000c0004e3ce8:  0x000000c0000d65a0  0x000000c0004d2000
0x000000c0004e3cf8:  0x000000000040695d <runtime.makechan+0x000000000000011d>  0x0000000000000060
0x000000c0004e3d08:  0x0000000000000064  0x000000c0004db530
0x000000c0004e3d18:  0x00000000004f973d <bufio.(*Reader).Peek+0x000000000000005d>  0x000000c000510ba0
0x000000c0004e3d28:  0x000000000061152a <net/http.(*fakeLocker).Unlock+0x000000000000002a>  0x000000c0004db7b8
0x000000c0004e3d38:  0x0000000000609e0c <net/http.(*persistConn).readLoop+0x00000000000001ac>  0x000000c000510ba0
0x000000c0004e3d48:  0x0000000000000001  0x0000000000000000
0x000000c0004e3d58:  0x0000000000000000  0x0000000000000000
0x000000c0004e3d68:  0x0000000000000000  0x0000000000000000
0x000000c0004e3d78:  0x0000000000000000  0x0300000100000000
crypto/tls.(*Conn).HandshakeContext(...)
	/usr/local/go/src/crypto/tls/conn.go:1402
crypto/tls.(*Conn).Handshake(...)
	/usr/local/go/src/crypto/tls/conn.go:1386
crypto/tls.(*Conn).Read(0xc000070000, {0xc0004d2000, 0x1000, 0x0?})
	/usr/local/go/src/crypto/tls/conn.go:1271 +0x65 fp=0xc0004e3c88 sp=0xc0004e3c18 pc=0x590ba5
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1750 +0x173e

goroutine 1 [select]:
net/http.(*persistConn).roundTrip(0xc0000d65a0, 0xc0000e41c0)
	/usr/local/go/src/net/http/transport.go:2620 +0x974
net/http.(*Transport).roundTrip(0x843780, 0xc000106000)
	/usr/local/go/src/net/http/transport.go:594 +0x7c9
net/http.(*Transport).RoundTrip(0x7f85dc89d5b8?, 0x6e59a0?)
	/usr/local/go/src/net/http/roundtrip.go:17 +0x19
net/http.send(0xc000106000, {0x6e59a0, 0x843780}, {0x66ac20?, 0x1?, 0x0?})
	/usr/local/go/src/net/http/client.go:252 +0x5d8
net/http.(*Client).send(0x8490e0, 0xc000106000, {0x203000?, 0xc000106000?, 0x0?})
	/usr/local/go/src/net/http/client.go:176 +0x9b
net/http.(*Client).do(0x8490e0, 0xc000106000)
	/usr/local/go/src/net/http/client.go:725 +0x8f5
net/http.(*Client).Do(...)
	/usr/local/go/src/net/http/client.go:593
net/http.(*Client).Get(0x407699?, {0x6782de?, 0x401230?})
	/usr/local/go/src/net/http/client.go:480 +0x6a
net/http.Get(...)
	/usr/local/go/src/net/http/client.go:449
main.main()
	/root/my_ebpf_test/programs/curlhttps.go:12 +0x73

goroutine 9 [select]:
net/http.(*persistConn).writeLoop(0xc0000d65a0)
	/usr/local/go/src/net/http/transport.go:2392 +0xf5
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1751 +0x1791

Programs

I have created a Git gist for this, you can see the code here: https://gist.github.com/mrproliu/dff336fe9f3aa34ae8ed270e55b4a27a

[mmat11]

Uretprobes don't work well with Go, see iovisor/bcc#1320 and golang/go#22008.

AFAIK, there's a workaround which is to set N uprobes on RET instructions, as explained here iovisor/bcc#1320 (comment)

Some examples: 1, 2