Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

libbpfgo: Libbpf v1.0 migration #826

Closed
javierhonduco opened this issue Sep 21, 2022 · 4 comments
Closed

libbpfgo: Libbpf v1.0 migration #826

javierhonduco opened this issue Sep 21, 2022 · 4 comments
Milestone
v0.12.0

Comments

@javierhonduco
Copy link
Contributor

javierhonduco commented Sep 21, 2022 •
edited

Opening this issue to track the libbpf v1.0 migration:
@maxbrunet maxbrunet mentioned this issue Sep 21, 2022
Merged
@maxbrunet
Copy link
Member

maxbrunet commented Sep 21, 2022 •
edited

The update has only been partial so far, here's how to make a full one:

# Update Go module
go get github.com/aquasecurity/libbpfgo@v0.4.0-libbpf-1.0.0
go mod tidy

# Update Git submodule
cd 3rdparty/libbpf
git fetch --tags
git checkout v1.0.0

# Build
cd ../..
make clean
make build

# Run
sudo dist/parca-agent --node=test

Everything works until you try to run it, the failed to obtain profiles from eBPF maps message is replaced by a SIGSEGV: segmentation violation:

Full logs 
ooooooooo.                                                  .o.                                            .                                                                                                                                                   
`888   `Y88.                                               .888.                                         .o8                                                                                                                                                   
 888   .d88'  .oooo.   oooo d8b  .ooooo.   .oooo.         .8"888.      .oooooooo  .ooooo.  ooo. .oo.   .o888oo                                                                                                                                                 
 888ooo88P'  `P  )88b  `888""8P d88' `"Y8 `P  )88b       .8' `888.    888' `88b  d88' `88b `888P"Y88b    888                                                                                                                                                   
 888          .oP"888   888     888        .oP"888      .88ooo8888.   888   888  888ooo888  888   888    888                                                                                                                                                   
 888         d8(  888   888     888   .o8 d8(  888     .8'     `888.  `88bod8P'  888    .o  888   888    888 .                                                                                                                                                 
o888o        `Y888""8o d888b    `Y8bod8P' `Y888""8o   o88o     o8888o `8oooooo.  `Y8bod8P' o888o o888o   "888"                                                                                                                                                 
                                                                      d"     YD                                                                                                                                                                                
                                                                      "Y88888P'                                                                                                                                                                                

level=info name=parca-agent ts=2022-09-21T19:52:56.116950609Z caller=main.go:203 msg="local profile storage is enabled" dir=./tmp/profiles
name=parca-agent ts=2022-09-21T19:52:56.116976112Z caller=main.go:210 msg=starting... node=test store=
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x186246d]

runtime stack:
runtime.throw({0x1d96060?, 0x7fd0bc0008fa?})
	runtime/panic.go:992 +0x71
runtime.sigpanic()
	runtime/signal_unix.go:802 +0x389

goroutine 13 [syscall]:
runtime.cgocall(0x17c5be0, 0xc0000af858)
	runtime/cgocall.go:157 +0x5c fp=0xc0000af830 sp=0xc0000af7f8 pc=0x40525c
github.com/aquasecurity/libbpfgo._C2func_bpf_object__open_mem(0x7fd0bc000b80, 0x700, 0xc000306280)
	_cgo_gotypes.go:325 +0x57 fp=0xc0000af858 sp=0xc0000af830 pc=0x17a83b7
github.com/aquasecurity/libbpfgo.NewModuleFromBufferArgs.func3(0x0?, 0x0?, 0x1d6f465?)
	github.com/aquasecurity/libbpfgo@v0.4.0-libbpf-1.0.0/libbpfgo.go:472 +0x7e fp=0xc0000af8b0 sp=0xc0000af858 pc=0x17aca1e
github.com/aquasecurity/libbpfgo.NewModuleFromBufferArgs({{0x0, 0x0}, {0x1d6f465, 0x17}, {0x1d51a75, 0x5}, {0x0, 0x0}, {0x2dc0660, 0x700, ...}})
	github.com/aquasecurity/libbpfgo@v0.4.0-libbpf-1.0.0/libbpfgo.go:472 +0x1f6 fp=0xc0000af9a8 sp=0xc0000af8b0 pc=0x17ac636
github.com/parca-dev/parca-agent/pkg/profiler/cpu.(*CPU).Run(0xc000326e00, {0x2007be8?, 0xc00007a9c0})
	github.com/parca-dev/parca-agent/pkg/profiler/cpu/cpu.go:145 +0x1d8 fp=0xc0000afe28 sp=0xc0000af9a8 pc=0x17b9d58
main.run.func6()
	github.com/parca-dev/parca-agent/cmd/parca-agent/main.go:315 +0x388 fp=0xc0000aff88 sp=0xc0000afe28 pc=0x17c25e8
github.com/oklog/run.(*Group).Run.func1({0xc00007aa00?, 0xc0004dba10?})
	github.com/oklog/run@v1.1.0/group.go:38 +0x2f fp=0xc0000affc0 sp=0xc0000aff88 pc=0x99412f
github.com/oklog/run.(*Group).Run.func2()
	github.com/oklog/run@v1.1.0/group.go:39 +0x2e fp=0xc0000affe0 sp=0xc0000affc0 pc=0x9940ce
runtime.goexit()
	runtime/asm_amd64.s:1571 +0x1 fp=0xc0000affe8 sp=0xc0000affe0 pc=0x46cc81
created by github.com/oklog/run.(*Group).Run
	github.com/oklog/run@v1.1.0/group.go:37 +0x22a

goroutine 1 [chan receive]:
github.com/oklog/run.(*Group).Run(0xc000a1fab8)
	github.com/oklog/run@v1.1.0/group.go:43 +0x7c
main.run({0x1febda0?, 0xc0003065f0}, 0xc000306640, {{0xc00088b0c0, 0x4}, {0xc00088b0d0, 0x5}, {0xc00088b0b0, 0x4}, 0x2540be400, ...})
	github.com/parca-dev/parca-agent/cmd/parca-agent/main.go:338 +0x2827
main.main()
	github.com/parca-dev/parca-agent/cmd/parca-agent/main.go:120 +0x313

goroutine 26 [select]:
github.com/baidubce/bce-sdk-go/util/log.NewLogger.func1()
	github.com/baidubce/bce-sdk-go@v0.9.111/util/log/logger.go:375 +0xb9
created by github.com/baidubce/bce-sdk-go/util/log.NewLogger
	github.com/baidubce/bce-sdk-go@v0.9.111/util/log/logger.go:368 +0x136

goroutine 7 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc0001a6c00)
	go.opencensus.io@v0.23.0/stats/view/worker.go:276 +0xad
created by go.opencensus.io/stats/view.init.0
	go.opencensus.io@v0.23.0/stats/view/worker.go:34 +0x8d

goroutine 33 [chan receive]:
github.com/goburrow/cache.(*localCache).processEntries(0xc00023e000)
	github.com/goburrow/cache@v0.1.4/local.go:212 +0x8b
created by github.com/goburrow/cache.(*localCache).init
	github.com/goburrow/cache@v0.1.4/local.go:83 +0x25b

goroutine 87 [chan receive]:
github.com/goburrow/cache.(*localCache).processEntries(0xc00023e2c0)
	github.com/goburrow/cache@v0.1.4/local.go:212 +0x8b
created by github.com/goburrow/cache.(*localCache).init
	github.com/goburrow/cache@v0.1.4/local.go:83 +0x25b

goroutine 88 [select]:
github.com/parca-dev/parca-agent/pkg/discovery.(*SystemdDiscoverer).Run(0xc0004dafb0, {0x2007be8, 0xc00007a740}, 0xc000126088?)
	github.com/parca-dev/parca-agent/pkg/discovery/systemd.go:56 +0x145
github.com/parca-dev/parca-agent/pkg/discovery.(*Manager).startProvider.func1()
	github.com/parca-dev/parca-agent/pkg/discovery/discovery_manager.go:220 +0x52
created by github.com/parca-dev/parca-agent/pkg/discovery.(*Manager).startProvider
	github.com/parca-dev/parca-agent/pkg/discovery/discovery_manager.go:219 +0x38e

goroutine 89 [select]:
github.com/parca-dev/parca-agent/pkg/discovery.(*Manager).updater(0xc000128510, {0x2007be8, 0xc00007a740}, 0xc0000ce6e0, 0xc0002a0a20)
	github.com/parca-dev/parca-agent/pkg/discovery/discovery_manager.go:229 +0xd9
created by github.com/parca-dev/parca-agent/pkg/discovery.(*Manager).startProvider
	github.com/parca-dev/parca-agent/pkg/discovery/discovery_manager.go:224 +0x430

goroutine 90 [chan receive]:
github.com/goburrow/cache.(*localCache).processEntries(0xc00023e580)
	github.com/goburrow/cache@v0.1.4/local.go:212 +0x8b
created by github.com/goburrow/cache.(*localCache).init
	github.com/goburrow/cache@v0.1.4/local.go:83 +0x25b

goroutine 92 [chan receive]:
github.com/goburrow/cache.(*localCache).processEntries(0xc00023e6e0)
	github.com/goburrow/cache@v0.1.4/local.go:212 +0x8b
created by github.com/goburrow/cache.(*localCache).init
	github.com/goburrow/cache@v0.1.4/local.go:83 +0x25b

goroutine 95 [chan receive]:
github.com/goburrow/cache.(*localCache).processEntries(0xc00023e840)
	github.com/goburrow/cache@v0.1.4/local.go:212 +0x8b
created by github.com/goburrow/cache.(*localCache).init
	github.com/goburrow/cache@v0.1.4/local.go:83 +0x25b

goroutine 96 [chan receive]:
github.com/goburrow/cache.(*localCache).processEntries(0xc00023e9a0)
	github.com/goburrow/cache@v0.1.4/local.go:212 +0x8b
created by github.com/goburrow/cache.(*localCache).init
	github.com/goburrow/cache@v0.1.4/local.go:83 +0x25b

goroutine 97 [chan receive]:
github.com/goburrow/cache.(*localCache).processEntries(0xc00023eb00)
	github.com/goburrow/cache@v0.1.4/local.go:212 +0x8b
created by github.com/goburrow/cache.(*localCache).init
	github.com/goburrow/cache@v0.1.4/local.go:83 +0x25b

goroutine 98 [chan receive]:
github.com/goburrow/cache.(*localCache).processEntries(0xc00023ec60)
	github.com/goburrow/cache@v0.1.4/local.go:212 +0x8b
created by github.com/goburrow/cache.(*localCache).init
	github.com/goburrow/cache@v0.1.4/local.go:83 +0x25b

goroutine 114 [chan receive]:
github.com/parca-dev/parca-agent/pkg/discovery.(*Manager).Run(0xc000128510, {0x2007be8?, 0xc00007a500})
	github.com/parca-dev/parca-agent/pkg/discovery/discovery_manager.go:147 +0xaf
main.run.func3()
	github.com/parca-dev/parca-agent/cmd/parca-agent/main.go:244 +0x228
github.com/oklog/run.(*Group).Run.func1({0xc0006dc270?, 0xc0004db200?})
	github.com/oklog/run@v1.1.0/group.go:38 +0x2f
created by github.com/oklog/run.(*Group).Run
	github.com/oklog/run@v1.1.0/group.go:37 +0x22a

goroutine 115 [select]:
github.com/parca-dev/parca-agent/pkg/discovery.(*Manager).sender(0xc000128510, {0x2007be8, 0xc00007a500})
	github.com/parca-dev/parca-agent/pkg/discovery/discovery_manager.go:256 +0xe5
created by github.com/parca-dev/parca-agent/pkg/discovery.(*Manager).Run
	github.com/parca-dev/parca-agent/pkg/discovery/discovery_manager.go:146 +0x95

goroutine 14 [IO wait]:
internal/poll.runtime_pollWait(0x7fd0d2e321d8, 0x72)
	runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc000328a80?, 0x0?, 0x0)
	internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Accept(0xc000328a80)
	internal/poll/fd_unix.go:614 +0x22c
net.(*netFD).accept(0xc000328a80)
	net/fd_unix.go:172 +0x35
net.(*TCPListener).accept(0xc000674000)
	net/tcpsock_posix.go:139 +0x28
net.(*TCPListener).Accept(0xc000674000)
	net/tcpsock.go:288 +0x3d
net/http.(*Server).Serve(0xc000390540, {0x2006d38, 0xc000674000})
	net/http/server.go:3039 +0x385
net/http.Serve(...)
	net/http/server.go:2543
main.run.func8()
	github.com/parca-dev/parca-agent/cmd/parca-agent/main.go:331 +0x265
github.com/oklog/run.(*Group).Run.func1({0xc00078c030?, 0xc000674018?})
	github.com/oklog/run@v1.1.0/group.go:38 +0x2f
created by github.com/oklog/run.(*Group).Run
	github.com/oklog/run@v1.1.0/group.go:37 +0x22a

goroutine 15 [select]:
github.com/oklog/run.SignalHandler.func1()
	github.com/oklog/run@v1.1.0/actors.go:18 +0xe9
github.com/oklog/run.(*Group).Run.func1({0xc00078c060?, 0xc00053a080?})
	github.com/oklog/run@v1.1.0/group.go:38 +0x2f
created by github.com/oklog/run.(*Group).Run
	github.com/oklog/run@v1.1.0/group.go:37 +0x22a

goroutine 130 [syscall]:
os/signal.signal_recv()
	runtime/sigqueue.go:151 +0x2f
os/signal.loop()
	os/signal/signal_unix.go:23 +0x19
created by os/signal.Notify.func1.1
	os/signal/signal.go:151 +0x2a

If you update libbpfgo to include the fix from aquasecurity/libbpfgo#248:

go get github.com/aquasecurity/libbpfgo@v0.4.0-libbpf-1.0.0.0.20220921161443-245a1a13132e
go mod tidy

It seems to be a lot better, but what use to be a deprecation warning before is now an error in v1:

libbpf: elf: legacy map definitions in 'maps' section are not supported by libbpf v1.0+
Failed to open bpf object: Operation not supported
level=error name=parca-agent ts=2022-09-21T20:03:41.041680823Z caller=main.go:121 err="new bpf module: failed to open BPF object parca: operation not supported"

Looks like aya still does not support the new format, latest update I could find in aya-rs/aya#248 (comment):

Support in aya-bpf for BTF maps is sadly not as simple as the BTF generated for Rust programs by LLVM isn't correct.

@maxbrunet
Copy link
Member

Blocked by #294
Blocked by aya-rs/aya#351

@kakkoyun
Copy link
Member

kakkoyun commented Oct 5, 2022

No blockers after #869.

@kakkoyun kakkoyun added this to the v0.11.0 milestone Oct 10, 2022
@kakkoyun
Copy link
Member

Done by #901

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
v0.12.0
Development

No branches or pull requests
3 participants
@kakkoyun @javierhonduco @maxbrunet