Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Locally download and use libbpf dependencies #1602

Draft
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

kakkoyun
Copy link
Member

@kakkoyun kakkoyun commented May 4, 2023

Why?

After libbpf started requiring zstd as a dependency, my system can't build libbpf locally. That's why I have started changes. (Thanks @javierhonduco, for the pointers).

This is a first step towards to system-independent way of reproducibly building our binary. This will also simplify our CI dependencies. We need to enhance the script to handle cross-platform builds and configure goreleaser accordingly.

related: #1304

What?

馃 Generated by Copilot at f5b7e98

This pull request adds support for building the parca-agent binary with static libraries for libbpf and its dependencies. It introduces a new script scripts/download-and-build-libbpf-deps.sh and modifies the Makefile and the Tiltfile accordingly. This enables the parca-agent binary to run on systems without dynamic libraries for libbpf.

How?

馃 Generated by Copilot at f5b7e98

  • Define and use variables for libbpf dependencies in Makefile (link, link, link, link, link)
  • Add a new script scripts/download-and-build-libbpf-deps.sh to download and build the static libraries for libbpf dependencies (link)
  • Remove an empty line from Tiltfile (link)

Original work: https://github.com/javierhonduco/rbperf/blob/573ebc0a90f05ca0ef9b26ae7825d27468035cf1/tools/build_deps#L8

TODO

@maxbrunet
Copy link
Member

It'd be nice if C dependencies had their own package manager (I mean an other option from Bash or Make or Nix), like:

But none of them has support for our libraries 馃槥

@kakkoyun kakkoyun changed the title WIP: Locally download and use libbpf dependencies Locally download and use libbpf dependencies Jul 4, 2023
@kakkoyun kakkoyun marked this pull request as ready for review July 25, 2023 14:59
@kakkoyun kakkoyun requested a review from a team as a code owner July 25, 2023 14:59
Copy link
Contributor

@javierhonduco javierhonduco left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good -- Before we merge this, could we check that:

  • the generated arm64 artefact works fine in CI;
  • local "native" (without x-compilation) arm64 builds work;

Was the scripts/local-dev-cluster.sh file created by accident? 馃槃

Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>
Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>
kakkoyun added a commit that referenced this pull request Jan 9, 2024
Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>

> [!NOTE]
> This targets
https://github.com/parca-dev/parca-agent/tree/new_build_pipeline and
there will be a PR in the end targets the `main`.

From the users' perspective, this PR shouldn't change anything. If you
want to use specified dependencies please consider installing the devbox
and direnv (simply `make init`)

This brings the result of the experimentation for a new build pipeline
from parca-dev/testdata#27 to the agent.

At first, we will start by introducing the https://www.jetpack.io/devbox

> Now, we use https://www.jetpack.io/devbox for managing external and
tool dependencies, such as the Go toolchain or C compiler. This tool has
been chosen because it has all the benefits of
[Nix](https://nixos.org/), but we don't have to deal with all the
intricacies of Nix. It also has an excellent integration with
[direnv](https://direnv.net/) to load the environment automatically when
you enter the directory.

Moreover, this PR:
- Changes the directory structure a little bit to reduce the number of
top-level directories (I will introduce more) and group them.
- It also vendors in the dependencies of libbpf as git submodules and
compiles it. It's the same strategy as in
#1602 (that PR will be
deprecated). It is easier to manage than using some external tool.
Especially when we want to cross-compile them. (Makefile changes)
- The GH actions are updated to utilize the devbox.

### Test Plan
- See that CI is green.
- (Locally) Just run the make targets as before.
@kakkoyun kakkoyun added the area/build-pipeline Something to do with CI and build pipeline label Feb 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/build-pipeline Something to do with CI and build pipeline
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants