Reproducible builds #28987

holiman · 2024-02-14T09:41:33Z

Reproducible builds

This is a little investigation into "do we have reproducible builds in geth?".

A reproducible build means that one can replicate locally a build made on e.g. a build-server.
That is, produce an exact matching binary. This is very useful to verify the integrity
of the build-servers: any remote machine can be used to watch over the builds.

The Go compiler is, supposedly, reproducible. However, go-ethereum is not pure go

It uses a c compiler, thus we need to ensure the same compiler is used
It bundles some version-system artefacts at build-time,
Maybe other things. To be discovered.

Testing

First, I downloaded the latest build from our downloads-page. The downloads-page
lists the checksum as 8d5e138dc3eb7b08cde48966aee0ea79 (note: md5 is not a secure
cryptographic hash, but we also provide detached signatures, which offers much
better security in verifying integrity).

[user@work go-ethereum]$ md5sum geth-linux-amd64-1.13.13-unstable-fe91d476.tar.gz
8d5e138dc3eb7b08cde48966aee0ea79  geth-linux-amd64-1.13.13-unstable-fe91d476.tar.gz
[user@work go-ethereum]$ md5sum geth-linux-amd64-1.13.13-unstable-fe91d476/geth
1a372833c2a63c95a2f855524eb5fcd9  geth-linux-amd64-1.13.13-unstable-fe91d476/geth

I then tried to create a docker container replicating the enviromment used. Details gleaned from the downloaded file:

$ ./geth-linux-amd64-1.13.13-unstable-fe91d476/geth version
Geth
Version: 1.13.13-unstable
Git Commit: fe91d476ba3e29316b6dc99b6efd4a571481d888
Git Commit Date: 20240213
Architecture: amd64
Go Version: go1.21.6
Operating System: linux
GOPATH=/home/user/go
GOROOT=/usr/local/go

The .travis.yml also gives us some hints:

      dist: bionic
      go: 1.21.x

Dockerfile attempt

Using a dockerfile like this:

from ubuntu:bionic

RUN apt-get update && apt-get install gcc-multilib git ca-certificates wget -yq --no-install-recommends
RUN git clone --branch master https://github.com/ethereum/go-ethereum.git

RUN wget https://go.dev/dl/go1.21.6.linux-amd64.tar.gz && \
	rm -rf /usr/local/go && \
	tar -C /usr/local -xzf go1.21.6.linux-amd64.tar.gz && \
	export PATH=$PATH:/usr/local/go/bin 

RUN cd go-ethereum && git checkout fe91d476ba3e29316b6dc99b6efd4a571481d888 && \
	CI=true TRAVIS=true TRAVIS_COMMIT="fe91d476ba3e29316b6dc99b6efd4a571481d888" go run ./build/ci.go install ./cmd/geth/
RUN md5sum ./build/bin/geth

In order to make the docker-version bundle the git data, we set the TRAVIS,CI env variables. See internal/build/env.go for reasons.

The two builds are not exactly alike in size:

root@208cb9fcfa68:/go-ethereum# ls -l ./build/bin/geth         
-rwxr-xr-x 1 root root 58129760 Feb 14 09:53 ./build/bin/geth

$ ls -la ./geth-linux-amd64-1.13.13-unstable-fe91d476/geth
-rwxr-xr-x 1 user user 58129968 Feb 13 14:55 ./geth-linux-amd64-1.13.13-unstable-fe91d476/geth

Content-wise:

root@208cb9fcfa68:/go-ethereum# strings ./build/bin/geth | head    
/lib64/ld-linux-x86-64.so.2
RAMLiBUAnrbn5zHLQ2v2/WlYmiboMK5ddsyu5qL-z/zajlwZgTLCfStG3HorG6/Utx6Jmui4qlzsokyGBwE
D %$
DD@ 
#@ $
@@	j
k(dB0
0	 b
ljI^
q6-p

VS

$ strings ./geth-linux-amd64-1.13.13-unstable-fe91d476/geth | head
/lib64/ld-linux-x86-64.so.2
JHKPXlVR27nUe4y9sY68/WlYmiboMK5ddsyu5qL-z/zajlwZgTLCfStG3HorG6/QrO6sKmnFVHR7U-WHF3U
x3vo
D %$
DD@
#@ $
@@	j
k(dB0
0	 b
ljI^

The text was updated successfully, but these errors were encountered:

holiman · 2024-02-14T11:26:51Z

Actually, ignore reproducing the same build as the travis builder, we don't even reproduce the same build on the same system:

root@208cb9fcfa68:/go-ethereum# rm ./build/bin/geth 

root@208cb9fcfa68:/go-ethereum# CI=true TRAVIS=true TRAVIS_COMMIT="fe91d476ba3e29316b6dc99b6efd4a571481d888" go run ./build/ci.go install -dlgo ./cmd/geth
gotool.go:96: -dlgo version matches active Go version 1.21.6, skipping download.
>>> /usr/local/go/bin/go build -ldflags "-X github.com/ethereum/go-ethereum/internal/version.gitCommit=fe91d476ba3e29316b6dc99b6efd4a571481d888 -X github.com/ethereum/go-ethereum/internal/version.gitDate=20240213 -extldflags '-Wl,-z,stack-size=0x800000'" -tags urfave_cli_no_docs,ckzg -trimpath -v -o /go-ethereum/build/bin/geth ./cmd/geth

root@208cb9fcfa68:/go-ethereum# md5sum ./build/bin/geth
1337ffaed216a31fa9a77caf138f642f  ./build/bin/geth

root@208cb9fcfa68:/go-ethereum# rm ./build/bin/geth 

root@208cb9fcfa68:/go-ethereum# CI=true TRAVIS=true TRAVIS_COMMIT="fe91d476ba3e29316b6dc99b6efd4a571481d888" go run ./build/ci.go install -dlgo ./cmd/geth
gotool.go:96: -dlgo version matches active Go version 1.21.6, skipping download.
>>> /usr/local/go/bin/go build -ldflags "-X github.com/ethereum/go-ethereum/internal/version.gitCommit=fe91d476ba3e29316b6dc99b6efd4a571481d888 -X github.com/ethereum/go-ethereum/internal/version.gitDate=20240213 -extldflags '-Wl,-z,stack-size=0x800000'" -tags urfave_cli_no_docs,ckzg -trimpath -v -o /go-ethereum/build/bin/geth ./cmd/geth

root@208cb9fcfa68:/go-ethereum# md5sum ./build/bin/geth
4e5180c9678db91d506e223c9a25838a  ./build/bin/geth

holiman · 2024-02-14T11:31:12Z

If we disable the C building, then we get reliable builds on a single machine

root@208cb9fcfa68:/go-ethereum# rm ./build/bin/geth; CGO_ENABLED=0 CI=true TRAVIS=true TRAVIS_COMMIT="fe91d476ba3e29316b6dc99b6efd4a571481d888" go run ./build/ci.go install -dlgo ./cmd/geth; md5sum ./build/bin/geth
gotool.go:96: -dlgo version matches active Go version 1.21.6, skipping download.
>>> /usr/local/go/bin/go build -ldflags "-X github.com/ethereum/go-ethereum/internal/version.gitCommit=fe91d476ba3e29316b6dc99b6efd4a571481d888 -X github.com/ethereum/go-ethereum/internal/version.gitDate=20240213 -extldflags '-Wl,-z,stack-size=0x800000'" -tags urfave_cli_no_docs,ckzg -trimpath -v -o /go-ethereum/build/bin/geth ./cmd/geth
9f99056d1537a6f00704e25cc77e8a3f  ./build/bin/geth

root@208cb9fcfa68:/go-ethereum# rm ./build/bin/geth; CGO_ENABLED=0 CI=true TRAVIS=true TRAVIS_COMMIT="fe91d476ba3e29316b6dc99b6efd4a571481d888" go run ./build/ci.go install -dlgo ./cmd/geth; md5sum ./build/bin/geth
gotool.go:96: -dlgo version matches active Go version 1.21.6, skipping download.
>>> /usr/local/go/bin/go build -ldflags "-X github.com/ethereum/go-ethereum/internal/version.gitCommit=fe91d476ba3e29316b6dc99b6efd4a571481d888 -X github.com/ethereum/go-ethereum/internal/version.gitDate=20240213 -extldflags '-Wl,-z,stack-size=0x800000'" -tags urfave_cli_no_docs,ckzg -trimpath -v -o /go-ethereum/build/bin/geth ./cmd/geth
9f99056d1537a6f00704e25cc77e8a3f  ./build/bin/geth

holiman · 2024-04-25T13:16:06Z

Got a report that these paths are present in the output:

│ -/home/travis/gopath/pkg/mod/github.com/karalabe/usb@v0.0.2/libusb/libusb/os/linux_usbfs.c

[user@work hid]$ go build ./demo.go && strings demo | grep home
/home/user/go/src/github.com/karalabe/hid/libusb/libusb/libusbi.h
/home/user/go/src/github.com/karalabe/hid/libusb/libusb/os/linux_usbfs.c
/home/user/go/src/github.com/karalabe/hid/libusb/libusb/os/events_posix.c
/home/user/go/src/github.com/karalabe/hid/libusb/libusb/os/linux_netlink.c
/home/user/go/src/github.com/karalabe/hid/libusb/libusb/core.c
/home/user/go/src/github.com/karalabe/hid/libusb/libusb/hotplug.c
/home/user/go/src/github.com/karalabe/hid/libusb/libusb/io.c
/home/user/go/src/github.com/karalabe/hid/wchar.go
/home/user/go/src/github.com/karalabe/hid/hid_enabled.go
/home/user/go/src/github.com/karalabe/hid/demo.go

[user@work hid]$ go build  -ldflags="-w -s" ./demo.go && strings demo | grep home
/home/user/go/src/github.com/karalabe/hid/libusb/libusb/libusbi.h
/home/user/go/src/github.com/karalabe/hid/libusb/libusb/os/linux_usbfs.c
/home/user/go/src/github.com/karalabe/hid/libusb/libusb/os/events_posix.c
/home/user/go/src/github.com/karalabe/hid/libusb/libusb/os/linux_netlink.c
/home/user/go/src/github.com/karalabe/hid/libusb/libusb/core.c
/home/user/go/src/github.com/karalabe/hid/libusb/libusb/hotplug.c
/home/user/go/src/github.com/karalabe/hid/libusb/libusb/io.c
/home/user/go/src/github.com/karalabe/hid/wchar.go
/home/user/go/src/github.com/karalabe/hid/hid_enabled.go
/home/user/go/src/github.com/karalabe/hid/demo.go


[user@work hid]$ go build -trimpath ./demo.go && strings demo | grep home
[user@work hid]$

This works when imported as a library too

[user@work go-ethereum]$ go build ./cmd/geth  &&  strings ./geth | grep "home/user" | head -n 5
/home/user/go/pkg/mod/github.com/karalabe/hid@v1.0.1-0.20240306101548-573246063e52/libusb/libusb/libusbi.h
/home/user/go/pkg/mod/github.com/karalabe/hid@v1.0.1-0.20240306101548-573246063e52/libusb/libusb/os/linux_usbfs.c
/home/user/go/pkg/mod/github.com/karalabe/hid@v1.0.1-0.20240306101548-573246063e52/libusb/libusb/os/events_posix.c
/home/user/go/pkg/mod/github.com/karalabe/hid@v1.0.1-0.20240306101548-573246063e52/libusb/libusb/os/linux_netlink.c
/home/user/go/pkg/mod/github.com/karalabe/hid@v1.0.1-0.20240306101548-573246063e52/libusb/libusb/core.c
[user@work go-ethereum]$ go build -trimpath ./cmd/geth  &&  strings ./geth | grep "home/user" | head -n 5
[user@work go-ethereum]$

I don't see these paths in the output binary

[user@work go-ethereum]$ CI=true TRAVIS=true TRAVIS_COMMIT="fe91d476ba3e29316b6dc99b6efd4a571481d888" go run ./build/ci.go install -dlgo ./cmd/geth &&  strings ./build/bin/geth | grep "home/user" | head -n 5
/home/user/.cache/go1.22.2.linux-amd64.tar.gz is up-to-date
>>> /home/user/.cache/geth-go-1.22.2-linux-amd64/go/bin/go build -ldflags "-X github.com/ethereum/go-ethereum/internal/version.gitCommit=fe91d476ba3e29316b6dc99b6efd4a571481d888 -X github.com/ethereum/go-ethereum/internal/version.gitDate=20240213 -extldflags '-Wl,-z,stack-size=0x800000'" -tags urfave_cli_no_docs,ckzg -trimpath -v -o /home/user/go/src/github.com/ethereum/go-ethereum/build/bin/geth ./cmd/geth

vivi365 · 2024-04-30T09:07:11Z

Hi,

Running this:

wget https://gethstore.blob.core.windows.net/builds/geth-linux-amd64-1.13.15-c5ba367e.tar.gz
tar -xvf geth-linux-amd64-1.13.15-c5ba367e.tar.gz
cd geth-linux-amd64-1.13.15-c5ba367e
grep -a 'home/travis' geth | strings

I get four occurrences of full Travis paths in the bundle:

/home/travis/gopath/pkg/mod/github.com/karalabe/usb@v0.0.2/libusb/libusb/os/linux_netlink.c
/home/travis/gopath/pkg/mod/github.com/karalabe/usb@v0.0.2/libusb/libusb/os/linux_usbfs.c
/home/travis/gopath/pkg/mod/github.com/karalabe/usb@v0.0.2/libusb/libusb/io.c
/home/travis/gopath/pkg/mod/github.com/ethereum/c-kzg-4844@v0.4.0/bindings/go/../../src/c_kzg_4844.c

Which are part of the read-only data.
readelf -p .rodata geth | grep 'travis'

-trimpath seems to work so perhaps there is something else going on -- looking into it.

Here are some files to reproduce more descriptive diffs using diffoscope.

holiman · 2024-04-30T09:28:23Z

Right. And here's how it looks against a newer binary (1.14.0)

/home/travis/gopath/pkg/mod/github.com/karalabe/hid@v1.0.1-0.20240306101548-573246063e52/libusb/libusb/libusbi.h
/home/travis/gopath/pkg/mod/github.com/karalabe/hid@v1.0.1-0.20240306101548-573246063e52/libusb/libusb/os/events_posix.c
/home/travis/gopath/pkg/mod/github.com/karalabe/hid@v1.0.1-0.20240306101548-573246063e52/libusb/libusb/os/linux_netlink.c
/home/travis/gopath/pkg/mod/github.com/karalabe/hid@v1.0.1-0.20240306101548-573246063e52/libusb/libusb/os/linux_usbfs.c
/home/travis/gopath/pkg/mod/github.com/karalabe/hid@v1.0.1-0.20240306101548-573246063e52/libusb/libusb/core.c
/home/travis/gopath/pkg/mod/github.com/karalabe/hid@v1.0.1-0.20240306101548-573246063e52/libusb/libusb/hotplug.c
/home/travis/gopath/pkg/mod/github.com/karalabe/hid@v1.0.1-0.20240306101548-573246063e52/libusb/libusb/io.c
/home/travis/gopath/pkg/mod/github.com/ethereum/c-kzg-4844@v1.0.0/bindings/go/../../src/c_kzg_4844.c

holiman · 2024-05-06T06:37:26Z

@vivi365 made a great finding here: golang/go#67011, trimpath is broken in ubuntu bionic.

Following that example, I did the same (but with hid, to reduce the build time)

First dockerfile, bionic, 18.04, which is an ESM

FROM ubuntu:bionic
RUN apt-get update && apt-get install gcc-multilib git ca-certificates wget -yq --no-install-recommends
RUN git clone --branch master --depth 1 https://github.com/karalabe/hid
RUN wget https://go.dev/dl/go1.21.6.linux-amd64.tar.gz && \
	rm -rf /usr/local/go && \
	tar -C /usr/local -xzf go1.21.6.linux-amd64.tar.gz && \
	export PATH=$PATH:/usr/local/go/bin

RUN cd hid && CGO_ENABLED=1 /usr/local/go/bin/go build -trimpath ./demo.go
RUN mv /hid/demo /demo && readelf -p .rodata demo | tee 1.txt | grep /hid/libusb | tee 2.txt

results in

#10 0.139   [ 3a745]  /hid/libusb/libusb/libusbi.h
#10 0.139   [ 3a890]  /hid/libusb/libusb/core.c
#10 0.139   [ 3aade]  /hid/libusb/libusb/hotplug.c
#10 0.139   [ 3ab5c]  /hid/libusb/libusb/io.c
#10 0.139   [ 3b9b8]  /hid/libusb/libusb/os/events_posix.c
#10 0.139   [ 3b9e0]  /hid/libusb/libusb/os/linux_netlink.c
#10 0.139   [ 3ba08]  /hid/libusb/libusb/os/linux_usbfs.c

For

ubuntu:focal 20.04 LTS, (not yet ESM)
ubuntu:jammy 22.04, LTS (not yet ESM)
ubuntu:noble 24.04, LTS (not yet ESM)

#9 0.470   [ 3b950]  /_/github.com/karalabe/hid/libusb/libusb/libusbi.h
#9 0.470   [ 3b988]  /_/github.com/karalabe/hid/libusb/libusb/os/events_posix.c
#9 0.470   [ 3b9c8]  /_/github.com/karalabe/hid/libusb/libusb/os/linux_netlink.c
#9 0.470   [ 3ba08]  /_/github.com/karalabe/hid/libusb/libusb/os/linux_usbfs.c
#9 0.470   [ 3ba48]  /_/github.com/karalabe/hid/libusb/libusb/core.c
#9 0.470   [ 3baa8]  /_/github.com/karalabe/hid/libusb/libusb/hotplug.c
#9 0.470   [ 3bae0]  /_/github.com/karalabe/hid/libusb/libusb/io.c

This is good, now it stripped the path /hid/ and all we see is the package-internal paths.

So seems that particular bug is only present in ubuntu. We should bump the CI-builders.

holiman added the type:bug label Feb 14, 2024

holiman mentioned this issue May 7, 2024

travis: use ubuntu noble (24.04) instead of bionic (18.04) #29723

Merged

fjl closed this as completed in #29723 May 7, 2024

holiman reopened this May 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reproducible builds #28987

Reproducible builds #28987

holiman commented Feb 14, 2024 •

edited

holiman commented Feb 14, 2024

holiman commented Feb 14, 2024

holiman commented Apr 25, 2024 •

edited

vivi365 commented Apr 30, 2024

holiman commented Apr 30, 2024

holiman commented May 6, 2024 •

edited

Reproducible builds #28987

Reproducible builds #28987

Comments

holiman commented Feb 14, 2024 • edited