Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

etcd: add recipe #616

Closed
wants to merge 3 commits into from
Closed

etcd: add recipe #616

wants to merge 3 commits into from

Conversation

geissonator
Copy link
Contributor

Investigating the potential use of etcd within an OpenBMC distribution to manage data sharing and leader election in a multi-BMC system. Start off with a basic recipe to build the package.

Signed-off-by: Andrew Geissler geissonator@yahoo.com

@geissonator
Copy link
Contributor Author

geissonator commented Sep 28, 2022

This works ok, although there are some warnings:

WARNING: etcd-3.5.5-r0 do_package_qa: QA Issue: File /usr/lib/go/src/go.etcd.io/etcd/v3/bin/etcdctl in package etcd-dev contains reference to TMPDIR
File /usr/lib/go/src/go.etcd.io/etcd/v3/bin/etcd in package etcd-dev contains reference to TMPDIR
File /usr/lib/go/src/go.etcd.io/etcd/v3/bin/etcdutl in package etcd-dev contains reference to TMPDIR [buildpaths]
WARNING: etcd-3.5.5-r0 do_package_qa: QA Issue: File /usr/bin/etcdctl in package etcd contains reference to TMPDIR
File /usr/bin/etcd in package etcd contains reference to TMPDIR
File /usr/bin/etcdutl in package etcd contains reference to TMPDIR [buildpaths]

I think someone with more expertise with the go bbclass and bb files could do a much better job with this then I did, but this is at least a working base.

@kraj
Copy link
Contributor

kraj commented Sep 30, 2022

it fails to build on yocto Autobuilders see https://autobuilder.yoctoproject.org/typhoon/#/builders/88/builds/2064/steps/14/logs/stdio

| go: github.com/bgentry/speakeasy@v0.1.0: Get "https://proxy.golang.org/github.com/bgentry/speakeasy/@v/v0.1.0.mod": dial tcp: lookup proxy.golang.org on 172.29.10.1:53: dial udp 172.29.10.1:53: connect: network is unreachable
| must be run from 'go.etcd.io/etcd/v3' module directory
| WARNING: exit code 255 from a shell command.
NOTE: recipe etcd-3.5.5-r0: task do_compile: Failed

@geissonator
Copy link
Contributor Author

| go: github.com/bgentry/speakeasy@v0.1.0: Get "https://proxy.golang.org/github.com/bgentry/speakeasy/@v/v0.1.0.mod": dial tcp: lookup proxy.golang.org on 172.29.10.1:53: dial udp 172.29.10.1:53: connect: network is unreachable

Darn, I see this in our CI as well, but not when I just build normally. I'm thinking it may be related to the container env that our CI is running in. Still digging, I fired up the docker container that our build uses but it worked in there as well.

I see some bugs that indicate something like "export GOPROXY=direct" may be needed but that's tough to inject into the etcd build env.

Copy link
Contributor

@kraj kraj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please look at network fetching issues.

@geissonator
Copy link
Contributor Author

Please look at network fetching issues.

I found a pretty lengthy discussion on this issue up at https://lists.openembedded.org/g/openembedded-core/topic/89464905#162517. It's not clear what the resolution was to this so for now I'm just going to add the original workaround in that discussion to the new etcd recipe.

@geissonator
Copy link
Contributor Author

@kraj I sort of forgot about this PR. I did try and address your request on the network issue. Was my solution ok or was there a better way to do it?

@kraj
Copy link
Contributor

kraj commented Mar 23, 2023

@kraj I sort of forgot about this PR. I did try and address your request on the network issue. Was my solution ok or was there a better way to do it?

I dont know why I did not pick it up. Perhaps it failed elsewhere but I did not update the comments here. I have cherry-picked it for master-next and lets see what happens

@kraj
Copy link
Contributor

kraj commented Mar 23, 2023

@geissonator
Copy link
Contributor Author

geissonator commented Mar 24, 2023

Wow, that's a lot of output. I think this is the error:

| HASH[build go.etcd.io/etcd/pkg/v3/wait]: "go1.20.1"
| HASH[build go.etcd.io/etcd/pkg/v3/wait]: "compile\n"
| HASH[build go.etcd.io/etcd/pkg/v3/wait]: "module go.etcd.io/etcd/pkg/v3@v3.5.5\n"
| HASH[build go.etcd.io/etcd/pkg/v3/wait]: "go 1.16\n"
| HASH[build go.etcd.io/etcd/pkg/v3/wait]: "goos linux goarch amd64\n"
| HASH[build go.etcd.io/etcd/pkg/v3/wait]: "import \"go.etcd.io/etcd/pkg/v3/wait\"\n"
| HASH[build go.etcd.io/etcd/pkg/v3/wait]: "omitdebug false standard false local false prefix \"\"\n"
| HASH[build go.etcd.io/etcd/pkg/v3/wait]: "trimpath\n"
| HASH[build go.etcd.io/etcd/pkg/v3/wait]: "compile compile version go1.20.1 [\"-dynlink\" \"-linkshared\"] []\n"
| HASH[build go.etcd.io/etcd/pkg/v3/wait]: "GOAMD64=v1\n"
| HASH /home/pokybuild/yocto-worker/meta-oe/build/build/tmp/work/core2-64-poky-linux/etcd/3.5.5-r0/build/src/go.etcd.io/etcd/v3/pkg/wait/wait.go: 8a5ebdedd75393212140092a98bf21336f5df3b588a45c1d286aff77a69d9f85
| HASH[build go.etcd.io/etcd/pkg/v3/wait]: "file wait.go il697ddTkyEhQAkqmL8h\n"
| HASH /home/pokybuild/yocto-worker/meta-oe/build/build/tmp/work/core2-64-poky-linux/etcd/3.5.5-r0/build/src/go.etcd.io/etcd/v3/pkg/wait/wait_time.go: 6ad3a7a9aef180ab564f9080fa7b4f7e99ba394e87ca61cd7142ea95ca2af75f
| HASH[build go.etcd.io/etcd/pkg/v3/wait]: "file wait_time.go atOnqa7xgKtWT5CA-ntP\n"
| HASH[build go.etcd.io/etcd/pkg/v3/wait]: "import log Qi7_3pxEobdOUXQ-54QD\n"
| HASH[build go.etcd.io/etcd/pkg/v3/wait]: "import sync WZayHmMqLALjkIRwnjZU\n"
| HASH[build go.etcd.io/etcd/pkg/v3/wait]: 7630d3da4db48fb04140e3ebc540d2141317ac61ee440c1d98e74d61645eb985
| go.etcd.io/etcd/pkg/v3/wait
| HASH subkey 7630d3da4db48fb04140e3ebc540d2141317ac61ee440c1d98e74d61645eb985 "srcfiles" = 319801406a3893ee7adf441910c3d839df4fa4aaaf5c321bb4bccd19b6da61a2
| # github.com/cespare/xxhash/v2
| asm: xxhash_amd64.s:120: when dynamic linking, R15 is clobbered by a global variable access and is used here: 00092 (/home/pokybuild/yocto-worker/meta-oe/build/build/tmp/work/core2-64-poky-linux/etcd/3.5.5-r0/build/pkg/mod/github.com/cespare/xxhash/v2@v2.1.1/xxhash_amd64.s:120)	ADDQ	R15, AX
| asm: assembly failed

Which doesn't mean a whole lot to me. Certainly looks a lot like cespare/xxhash#54. So I'll see if I can figure out how to get that fix in.

@geissonator
Copy link
Contributor Author

Yeah, fix went into v2.1.2 of xxhash and the compile error shows us using 2.1.1. Let me see if moving to the latest released etcd fixes this up.

@geissonator
Copy link
Contributor Author

ok @kraj I noticed you fixed a similar issue via c68a306 so I did something similar within a commit in this PR:

ee22866

@kraj
Copy link
Contributor

kraj commented Mar 24, 2023

its better but still seeing something like below on arm64 ( rpi4-64bit ) builds

ERROR: etcd-3.5.7-r0 do_package: Fatal errors occurred in subprocesses:
Command '['aarch64-yoe-linux-strip', '--remove-section=.comment', '--remove-section=.note', '/mnt/b/yoe/master/build/tmp/work/cortexa72-yoe-linux/etcd/3.5.7-r0/package/usr/lib/go/src/go.etcd.io/etcd/v3/bin/etcdutl']' returned non-zero exit status 1.
Subprocess output:aarch64-yoe-linux-strip: /mnt/b/yoe/master/build/tmp/work/cortexa72-yoe-linux/etcd/3.5.7-r0/package/usr/lib/go/src/go.etcd.io/etcd/v3/bin/etcdutl: file format not recognized
Command '['aarch64-yoe-linux-strip', '--remove-section=.comment', '--remove-section=.note', '/mnt/b/yoe/master/build/tmp/work/cortexa72-yoe-linux/etcd/3.5.7-r0/package/usr/lib/go/src/go.etcd.io/etcd/v3/bin/etcdctl']' returned non-zero exit status 1.
Subprocess output:aarch64-yoe-linux-strip: /mnt/b/yoe/master/build/tmp/work/cortexa72-yoe-linux/etcd/3.5.7-r0/package/usr/lib/go/src/go.etcd.io/etcd/v3/bin/etcdctl: file format not recognized
Command '['aarch64-yoe-linux-strip', '--remove-section=.comment', '--remove-section=.note', '/mnt/b/yoe/master/build/tmp/work/cortexa72-yoe-linux/etcd/3.5.7-r0/package/usr/lib/go/src/go.etcd.io/etcd/v3/bin/etcd']' returned non-zero exit status 1.
Subprocess output:aarch64-yoe-linux-strip: /mnt/b/yoe/master/build/tmp/work/cortexa72-yoe-linux/etcd/3.5.7-r0/package/usr/lib/go/src/go.etcd.io/etcd/v3/bin/etcd: file format not recognized
Command '['aarch64-yoe-linux-strip', '--remove-section=.comment', '--remove-section=.note', '/mnt/b/yoe/master/build/tmp/work/cortexa72-yoe-linux/etcd/3.5.7-r0/package/usr/bin/etcdutl']' returned non-zero exit status 1.
Subprocess output:aarch64-yoe-linux-strip: /mnt/b/yoe/master/build/tmp/work/cortexa72-yoe-linux/etcd/3.5.7-r0/package/usr/bin/etcdutl: file format not recognized
Command '['aarch64-yoe-linux-strip', '--remove-section=.comment', '--remove-section=.note', '/mnt/b/yoe/master/build/tmp/work/cortexa72-yoe-linux/etcd/3.5.7-r0/package/usr/bin/etcdctl']' returned non-zero exit status 1.
Subprocess output:aarch64-yoe-linux-strip: /mnt/b/yoe/master/build/tmp/work/cortexa72-yoe-linux/etcd/3.5.7-r0/package/usr/bin/etcdctl: file format not recognized
Command '['aarch64-yoe-linux-strip', '--remove-section=.comment', '--remove-section=.note', '/mnt/b/yoe/master/build/tmp/work/cortexa72-yoe-linux/etcd/3.5.7-r0/package/usr/bin/etcd']' returned non-zero exit status 1.
Subprocess output:aarch64-yoe-linux-strip: /mnt/b/yoe/master/build/tmp/work/cortexa72-yoe-linux/etcd/3.5.7-r0/package/usr/bin/etcd: file format not recognized

ERROR: Logfile of failure stored in: /mnt/b/yoe/master/build/tmp/work/cortexa72-yoe-linux/etcd/3.5.7-r0/temp/log.do_package.3543547
ERROR: Task (/mnt/b/yoe/master/sources/meta-openembedded/meta-oe/recipes-extended/etcd/etcd_3.5.7.bb:do_package) failed with exit code '1'
NOTE: Tasks Summary: Attempted 1125 tasks of which 1122 didn't need to be rerun and 1 failed.
NO

@geissonator
Copy link
Contributor Author

That latest failure is a bit confounding. Is there a way to recreate that issue without a "bitbake world"? The error seems to indicate that with rpi4-64bit, the command to strip out debug symbols from the generated binaries isn't working. I could hack a INHIBIT_PACKAGE_STRIP="1" into the recipe but that doesn't seem appropriate.

@kraj
Copy link
Contributor

kraj commented Mar 28, 2023

yeah this could be a generic issues with go binaries, I am not sure or perhaps something in compiler options that etcd is using to build itself.

@geissonator
Copy link
Contributor Author

yeah this could be a generic issues with go binaries, I am not sure or perhaps something in compiler options that etcd is using to build itself.

Yeah, some internet searching shows multiple issues over the years with go and stripping, although nothing concrete other then it seems to work in some situations and not others. Not stripping adds 9MB in size to the etcd app which is pretty negligible on the multi-GB emmc embedded systems we plan to put this on. So, for now lets just prevent the stripping.

@kraj
Copy link
Contributor

kraj commented Apr 19, 2023

yeah this could be a generic issues with go binaries, I am not sure or perhaps something in compiler options that etcd is using to build itself.

Yeah, some internet searching shows multiple issues over the years with go and stripping, although nothing concrete other then it seems to work in some situations and not others. Not stripping adds 9MB in size to the etcd app which is pretty negligible on the multi-GB emmc embedded systems we plan to put this on. So, for now lets just prevent the stripping.

still fails

https://autobuilder.yoctoproject.org/typhoon/#/builders/88/builds/2670/steps/15/logs/stdio

@geissonator
Copy link
Contributor Author

Sigh...something different this time. Gobin? I'll dig in. Also, if there's an easy way to recreate what CI does, that would be great. I can't recreate any of the failures being seen by it when just building for my AST2600 OpenBMC image.

| % 'env' 'GO111MODULE=off' 'go' 'get' 'github.com/myitcv/gobin'
| stderr: # cd /home/pokybuild/yocto-worker/meta-oe/build/build/tmp/work/core2-64-poky-linux/etcd/3.5.7-r0/build/src/github.com/myitcv; git clone -- https://github.com/myitcv/gobin /home/pokybuild/yocto-worker/meta-oe/build/build/tmp/work/core2-64-poky-linux/etcd/3.5.7-r0/build/src/github.com/myitcv/gobin
| stderr: Cloning into '/home/pokybuild/yocto-worker/meta-oe/build/build/tmp/work/core2-64-poky-linux/etcd/3.5.7-r0/build/src/github.com/myitcv/gobin'...
| stderr: fatal: unable to access 'https://github.com/myitcv/gobin/': error setting certificate file: /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/etc/ssl/certs/ca-certificates.crt
| stderr: package github.com/myitcv/gobin: exit status 128
| FAIL: (code:1):
|   % 'env' 'GO111MODULE=off' 'go' 'get' 'github.com/myitcv/gobin'

@geissonator
Copy link
Contributor Author

@kraj could you try submitting this one again? I've spent the last day looking at the error, and trying to recreate it but it really kind of looks like a CI machine issue with it's certificate:

unable to access 'https://github.com/myitcv/gobin/': error setting certificate file: /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/etc/ssl/certs/ca-certificates.crt

That github repo is fine and a clone of it works fine manually.

@geissonator
Copy link
Contributor Author

@kraj could you try submitting again to CI? I'm wondering if I should try putting this on the mailing list to see if anyone with more go expertise could look things over for me (I know next to nothing about go)? We are looking to use this etcd recipe (along with the work in #679) to start prototyping a yocto based distributed BMC design.

Investigating the potential use of etcd within an OpenBMC distribution
to manage data sharing and leader election in a multi-BMC system. Start
off with a basic recipe to build the package.

Signed-off-by: Andrew Geissler <geissonator@yahoo.com>
Fixes:
| # github.com/cespare/xxhash/v2
| asm: xxhash_amd64.s:120: when dynamic linking, R15 is clobbered by a global
| variable access and is used here:
| 00092 (/home/pokybuild/yocto-worker/meta-oe/cespare/xxhash/v2@v2.1.1/xxhash_amd64.s:120)	ADDQ	R15, AX
| asm: assembly failed

Upstream-Status: Backport [etcd-io/etcd@f0f77fc]

Limited PR with just this patch submitted via this PR:
  etcd-io/etcd#15556

Signed-off-by: Andrew Geissler <geissonator@yahoo.com>
@kraj
Copy link
Contributor

kraj commented May 2, 2023

@kraj could you try submitting again to CI? I'm wondering if I should try putting this on the mailing list to see if anyone with more go expertise could look things over for me (I know next to nothing about go)? We are looking to use this etcd recipe (along with the work in #679) to start prototyping a yocto based distributed BMC design.

queued for tonight's build.

@kraj
Copy link
Contributor

kraj commented May 2, 2023

@kraj could you try submitting again to CI? I'm wondering if I should try putting this on the mailing list to see if anyone with more go expertise could look things over for me (I know next to nothing about go)? We are looking to use this etcd recipe (along with the work in #679) to start prototyping a yocto based distributed BMC design.

queued for tonight's build.

still fails on Yocto Autobuilders - https://autobuilder.yoctoproject.org/typhoon/#/builders/88/builds/2708/steps/15/logs/stdio

@geissonator
Copy link
Contributor Author

Thanks @kraj , sorry this such a pain. I don't understand why CI sees this certificate issue but I can't recreate it. I'll probably have to try the mailing list to see if anyone has any thoughts.

| % 'env' 'GO111MODULE=off' 'go' 'get' 'github.com/myitcv/gobin'
| stderr: # cd /home/pokybuild/yocto-worker/meta-oe/build/build/tmp/work/core2-64-poky-linux/etcd/3.5.7-r0/build/src/github.com/myitcv; git clone -- https://github.com/myitcv/gobin /home/pokybuild/yocto-worker/meta-oe/build/build/tmp/work/core2-64-poky-linux/etcd/3.5.7-r0/build/src/github.com/myitcv/gobin
| stderr: Cloning into '/home/pokybuild/yocto-worker/meta-oe/build/build/tmp/work/core2-64-poky-linux/etcd/3.5.7-r0/build/src/github.com/myitcv/gobin'...
| stderr: fatal: unable to access 'https://github.com/myitcv/gobin/': error setting certificate file: /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/etc/ssl/certs/ca-certificates.crt
| stderr: package github.com/myitcv/gobin: exit status 128

The only reference to gobin I see is in

# go install: cannot install cross-compiled binaries when GOBIN is set
which doesn't seem to be the issue here.

@kraj
Copy link
Contributor

kraj commented May 2, 2023

Thanks @kraj , sorry this such a pain. I don't understand why CI sees this certificate issue but I can't recreate it. I'll probably have to try the mailing list to see if anyone has any thoughts.

| % 'env' 'GO111MODULE=off' 'go' 'get' 'github.com/myitcv/gobin'
| stderr: # cd /home/pokybuild/yocto-worker/meta-oe/build/build/tmp/work/core2-64-poky-linux/etcd/3.5.7-r0/build/src/github.com/myitcv; git clone -- https://github.com/myitcv/gobin /home/pokybuild/yocto-worker/meta-oe/build/build/tmp/work/core2-64-poky-linux/etcd/3.5.7-r0/build/src/github.com/myitcv/gobin
| stderr: Cloning into '/home/pokybuild/yocto-worker/meta-oe/build/build/tmp/work/core2-64-poky-linux/etcd/3.5.7-r0/build/src/github.com/myitcv/gobin'...
| stderr: fatal: unable to access 'https://github.com/myitcv/gobin/': error setting certificate file: /usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/etc/ssl/certs/ca-certificates.crt
| stderr: package github.com/myitcv/gobin: exit status 128

The only reference to gobin I see is in

# go install: cannot install cross-compiled binaries when GOBIN is set

which doesn't seem to be the issue here.

I also tried a build with

--- a/meta-oe/recipes-extended/etcd/etcd_3.5.7.bb
+++ b/meta-oe/recipes-extended/etcd/etcd_3.5.7.bb
@@ -19,7 +19,7 @@ RDEPENDS:${PN}-dev = " \
     bash \
 "

-export GO111MODULE="on"
+export GO111MODULE="off"

 inherit go

but that does not work either. Downloads should not happen after do_fetch since network is turned off for rest of tasks.
in this case it still seems to access network during do_compile which maybe working on systems where ca-certs are used from build host, but here ca-certs from host are not valid since the tool is from buildtools tarball. So it maybe finding a problem which is latent in other builds. e.g. when I build this recipe on archlinux it seems to compile ok.

@geissonator
Copy link
Contributor Author

Ahh, that's a good point on the network access during build. I see out in the etcd repo, they go look for this gobin repo as a part of the build process at https://github.com/etcd-io/etcd/blob/release-3.5/scripts/test_lib.sh#L308. I'll see if I can figure something out. Maybe it's as easy as just installing gobin in our do_fetch step.

This tool is installed as a part of the build process (build.sh sources
test_lib.sh)

This tool has been removed in the latest etcd main branch. Installing it
as a part of the build process breaks bitbake (it doesn't allow
downloading of packages once in the build steps).

This tool is not needed to build etcd (it appears to be used for some
optional test cases).

The following upstream commit removes the use of gobin in the main
branch, but it's not clear to me if the alternative of just running "go
install" will fix this issue. Fundamentally etcd allows package install
during build and bitbake does not:
etcd-io/etcd@900fc8d

Upstream-Status: Inappropriate

Signed-off-by: Andrew Geissler <geissonator@yahoo.com>
@geissonator
Copy link
Contributor Author

Turns out the gobin install during the build isn't even needed. I verified etcd has removed the use of gobin (an archived repo) in the main branch upstream. So I just added a patch that removes it in my latest push here.

@kraj
Copy link
Contributor

kraj commented May 2, 2023

Turns out the gobin install during the build isn't even needed. I verified etcd has removed the use of gobin (an archived repo) in the main branch upstream. So I just added a patch that removes it in my latest push here.

makes sense, I have picked it up for next round again.

@kraj
Copy link
Contributor

kraj commented May 4, 2023

@geissonator thanks for sticking with me. Finally this PR is merged at 4cd2006

@kraj kraj closed this May 4, 2023
@geissonator
Copy link
Contributor Author

Ahh, great, thanks for all the help @kraj!

@shr-project
Copy link
Contributor

@geissonator It fails in do_install when multilib is enabled and libdir is e.g. /usr/lib64:

DEBUG: Executing shell function do_install
ls: cannot access 'etcd/3.5.7-r0/build/bin/linux_arm64/': No such file or directory
install: cannot stat 'etcd/3.5.7-r0/image/usr/lib/go/src/go.etcd.io/etcd/v3/bin/etcd': No such file or directory
WARNING: exit code 1 from a shell command.

Why do we need these binaries in both libdir and bindir? Using libdir instead of hardcoding /usr/lib/ in do_install:append should fix this.

The error from ls is unrelated, just side-effect of "bad" check for ${B}/${GO_BUILD_BINDIR}/ content in https://git.openembedded.org/openembedded-core/tree/meta/classes-recipe/go.bbclass#n135

@shr-project
Copy link
Contributor

Fix sent to ML
https://lists.openembedded.org/g/openembedded-devel/message/102607

kraj pushed a commit to YoeDistro/meta-openembedded that referenced this pull request May 15, 2023
* I don't know anything about etcd including why we need these binaries duplicated
  in libdir and bindir, but use ${libdir} instead of hardcoded /usr/lib to fix
  do_install with multilib enabled when libdir is /usr/lib64 and it was failing
  with:
  DEBUG: Executing shell function do_install
  ls: cannot access 'etcd/3.5.7-r0/build/bin/linux_arm64/': No such file or directory
  install: cannot stat 'etcd/3.5.7-r0/image/usr/lib/go/src/go.etcd.io/etcd/v3/bin/etcd': No such file or directory
  WARNING: exit code 1 from a shell command.

* the error from ls is unrelated, fixed in:
  https://lists.openembedded.org/g/openembedded-core/message/181269

* also reported on github:
  openembedded#616

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Khem Raj <raj.khem@gmail.com>
@kraj
Copy link
Contributor

kraj commented May 15, 2023

Thanks @shr-project this patch looks ok

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants