[#137] FreeBSD support #361

akhramov · 2021-09-15T10:28:54Z

Now that containerd supports FreeBSD, we can port nerdctl, too!
🚀. For full-fledged functionality, we will need to port

buildkit. Status: POC is ready, need to upstream fixes to
dependencies.
runtime + containerd shim. runj/knast should work just fine.
CNI bridge plugin for networking.

This change fixes FreeBSD compilation errors by renaming linux files
to unix where necessary, or otherwise introducing FreeBSD versions for
the necessary files.

Signed-off-by: Artem Khramov akhramov@pm.me

AkihiroSuda · 2021-09-16T04:40:52Z

Thanks for working on this.

Could you update make artifacts to cross-compile GOOS=freebsd GOARCH=amd64 binary on GitHub Actions?

nerdctl/Makefile

Lines 83 to 84 in 95dd996

    
           GOOS=windows GOARCH=amd64     make -C $(CURDIR) binaries 
        
           tar $(TAR_FLAGS) -czvf $(CURDIR)/_output/nerdctl-$(VERSION_TRIMMED)-windows-amd64.tar.gz _output/nerdctl.exe

Also please add docs/freebsd.md to clarify the installation steps, usage, and the known limitations.

Eventually we should have CI, but it can be another PR if it is difficult.
We can use Cirrus CI to run FreeBSD virtual machines. https://cirrus-ci.org/guide/FreeBSD/

AkihiroSuda · 2021-09-16T04:41:30Z

cmd/nerdctl/main_freebsd.go

+	return
+}
+
+func bashCompleteNamespaceNames(clicontext *cli.Context) {


This code shouldn't be OS-dependent

AkihiroSuda · 2021-09-16T04:41:35Z

cmd/nerdctl/main_freebsd.go

+	return
+}
+
+func bashCompleteSnapshotterNames(clicontext *cli.Context) {


This code shouldn't be OS-dependent

AkihiroSuda · 2021-09-16T04:41:52Z

cmd/nerdctl/run.go

 		WithoutRunMount(), // unmount default tmpfs on "/run": https://github.com/containerd/nerdctl/issues/157
 	)

+	if runtime.GOOS != "freebsd" {


nit: runtime.GOOS == "linux" would be more accurate check

AkihiroSuda · 2021-09-16T04:42:21Z

cmd/nerdctl/run_freebsd.go

+}
+
+func WithoutRunMount() func(ctx context.Context, client oci.Client, c *containers.Container, s *oci.Spec) error {
+	// not valid on windows


not windows

AkihiroSuda · 2021-09-16T04:42:46Z

cmd/nerdctl/run_freebsd.go

+)
+
+func runBashComplete(clicontext *cli.Context) {
+	// noop


This shouldn't be noop, but can be deferred to another PR

AkihiroSuda · 2021-09-16T04:43:30Z

pkg/defaults/defaults_freebsd.go

+	gocni "github.com/containerd/go-cni"
+)
+
+const AppArmorProfileName = "nerdctl-default"


This should be empty.

Eventually this variable should be only referred from *_linux.go, though.

AkihiroSuda · 2021-09-16T04:44:03Z

pkg/mountutil/mountutil_freebsd.go

+
+// DefaultPropagationMode is the default propagation of mounts
+// where user doesn't specify mount propagation explicitly.
+// See also: https://github.com/moby/moby/blob/v20.10.7/volume/mounts/windows_parser.go#L440-L442


not windows

AkihiroSuda · 2021-09-16T04:45:32Z

Probably we need to update

nerdctl/cmd/nerdctl/run_runtime.go

Line 44 in 95dd996

if strings.HasPrefix(runtimeStr, "io.containerd.") {

too, if we want to support wtf.sbk.runj.v1

AkihiroSuda · 2021-09-16T04:46:02Z

cc @samuelkarp

akhramov · 2021-09-17T16:59:31Z

cmd/nerdctl/run.go

@@ -747,7 +752,7 @@ func withCustomHosts(src string) func(context.Context, oci.Client, *containers.C
 }

 func generateLogURI(dataStore string) (*url.URL, error) {
-	selfExe, err := os.Readlink("/proc/self/exe")


@AkihiroSuda This seems to be os-specific. is -d broken now on windows?

nerdctl run on Windows is completely WIP (help wanted)
#197

akhramov · 2021-09-17T17:02:25Z

.github/workflows/test.yml

@@ -161,3 +161,21 @@ jobs:
        command: ssh default -- "CONTAINERD_SNAPSHOTTER=fuse-overlayfs /vagrant/nerdctl.test -test.v -test.kill-daemon"
    - name: "Uninstall rootless containerd"
      run: ssh default -- containerd-rootless-setuptool.sh uninstall
+
+  test-unit-freebsd-amd64:


@AkihiroSuda Instead of using Cirrus, we can run FreeBSD jobs in GH actions this way. What do you think?

cmd/nerdctl/run_freebsd.go

akhramov · 2021-09-17T17:45:11Z

@AkihiroSuda thanks for the thorough review.

I'm going to focus on resolving the crosscompilation issue.

Also I would like to hear your input on doc -- I wasn't sure what to put there.

AkihiroSuda · 2021-09-21T05:56:00Z

.github/workflows/test.yml

+          ln -sf hack/Vagrantfile Vagrantfile
+          vagrant up ${{ matrix.box }}
+      - name: "Run unit tests"
+        run: vagrant ssh ${{ matrix.box }} -- "cd /vagrant; go test -v ./pkg/..."


Would is be possible to run nerdctl run SOME-IMAGE echo hello here?

AkihiroSuda · 2021-09-21T05:57:55Z

cmd/nerdctl/run_freebsd.go

+
+// executablePath returns the absolute path to the current binary
+func executablePath() (string, error) {
+	return filepath.Abs(os.Args[0])


https://pkg.go.dev/os#Executable

AkihiroSuda · 2021-09-21T10:13:44Z

9s
Run vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Box 'freebsd/FreeBSD-13.0-STABLE' could not be found. Attempting to find and install...
    default: Box Provider: virtualbox
    default: Box Version: >= 0
==> default: Loading metadata for box 'freebsd/FreeBSD-13.0-STABLE'
    default: URL: https://vagrantcloud.com/freebsd/FreeBSD-13.0-STABLE
==> default: Adding box 'freebsd/FreeBSD-13.0-STABLE' (v2021.09.16) for provider: virtualbox
    default: Downloading: https://vagrantcloud.com/freebsd/boxes/FreeBSD-13.0-STABLE/versions/2021.09.16/providers/virtualbox.box

Progress: 0% (Rate: 0*/s, Estimated time remaining: --:--:--)
Progress: 100% (Rate: 635/s, Estimated time remaining: --:--:--)
An error occurred while downloading the remote file. The error
message, if any, is reproduced below. Please fix this error and try
again.

The requested URL returned error: 429

vagrant up seems extremely flaky.
Could you mirror the image to somewhere more stable, or could you use Cirrus instead of Vagrant?

akhramov · 2021-09-21T11:13:03Z

@AkihiroSuda indeed it's flaky. I'm sorry for the notifications mess I caused. I mitigated the problem using the cache action.

Let's test that, otherwise I'll set up a Cirrus pipeline.

AkihiroSuda · 2021-09-21T11:39:15Z

.github/workflows/test.yml

+        with:
+          path: ~/.vagrant.d/boxes
+          key: ${{ runner.os }}-vagrant-${{ hashFiles('Vagrantfile') }}
+          restore-keys: |


Why is this different from key?

AkihiroSuda · 2021-09-21T11:42:31Z

hack/Vagrantfile.freebsd13

+  config.vm.provision "shell", inline: <<-SHELL
+    pkg bootstrap
+    pkg install -y go containerd runj
+    cp /usr/local/bin/containerd-shim-runj-v1 /usr/local/bin/containerd-shim-runc-v2


This looks slightly invalid.
I’d suggest just setting the default runtime to runj

Now that containerd supports FreeBSD, we can port nerdctl, too! :rocket:. For full-fledged functionality, we will need to port * buildkit. Status: POC is ready, need to upstream fixes to dependencies. * runtime + containerd shim. runj/knast should work just fine. * CNI bridge plugin for networking. This change fixes FreeBSD compilation errors by renaming linux files to unix where necessary, or otherwise introducing FreeBSD versions for the necessary files. Signed-off-by: Artem Khramov <akhramov@pm.me>

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>

AkihiroSuda

Pushed follow-up commits.

LGTM if green.

Thanks!

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>

AkihiroSuda · 2021-09-21T14:24:11Z

CI seems flaky.

Can be fixed later, though

time="2021-09-21T14:19:05Z" level=fatal msg="failed to create shim: failed to mount rootfs component &{nullfs /var/lib/containerd/io.containerd.snapshotter.v1.native/snapshots/3 [rw]}: mount [[-o rw -t nullfs /var/lib/containerd/io.containerd.snapshotter.v1.native/snapshots/3 /run/containerd/io.containerd.runtime.v2.task/default/70886b2a11483b463360ab4f419294075698a64754c02bc1dda57fc642e60ba9/rootfs]] failed: "mount_nullfs: /run/containerd/io.containerd.runtime.v2.task/default/70886b2a11483b463360ab4f419294075698a64754c02bc1dda57fc642e60ba9/rootfs: Resource deadlock avoided\n": exit status 1: unknown"

leandroscardua · 2022-05-06T22:50:18Z

Hi Team,

Any update on the FreeBSD support on ContainerD/nerdctl?

gmshake · 2022-05-07T01:58:33Z

@leandroscardua
There is a known issue #868 and WIP PR to resolve it: moby/sys#114 .
Hopefully nerdctl on FreeBSD can work smoothly when the issue is resolved.

leandroscardua · 2022-05-07T16:33:09Z

Thank you for the information @gmshake

akhramov force-pushed the feature/FreeBSD-support branch 4 times, most recently from 1ca61e2 to 14b0025 Compare September 15, 2021 11:11

AkihiroSuda added the platform/FreeBSD FreeBSD label Sep 15, 2021

AkihiroSuda reviewed Sep 16, 2021

View reviewed changes

AkihiroSuda added this to the v0.12.0 milestone Sep 16, 2021

akhramov force-pushed the feature/FreeBSD-support branch 6 times, most recently from 4e87563 to 856eaf8 Compare September 17, 2021 17:33

akhramov commented Sep 17, 2021

View reviewed changes

akhramov force-pushed the feature/FreeBSD-support branch 2 times, most recently from 1aa2823 to 55768c3 Compare September 19, 2021 09:50

akhramov requested a review from AkihiroSuda September 19, 2021 10:06

akhramov force-pushed the feature/FreeBSD-support branch from 55768c3 to 327b973 Compare September 20, 2021 09:43

AkihiroSuda reviewed Sep 21, 2021

View reviewed changes

akhramov force-pushed the feature/FreeBSD-support branch 7 times, most recently from 42886b4 to e69cc74 Compare September 21, 2021 10:06

akhramov force-pushed the feature/FreeBSD-support branch from e69cc74 to 99274ad Compare September 21, 2021 10:17

akhramov force-pushed the feature/FreeBSD-support branch from 99274ad to 2af44af Compare September 21, 2021 11:13

AkihiroSuda reviewed Sep 21, 2021

View reviewed changes

akhramov force-pushed the feature/FreeBSD-support branch 2 times, most recently from 7521254 to b0ad974 Compare September 21, 2021 12:41

akhramov force-pushed the feature/FreeBSD-support branch from b0ad974 to d93be27 Compare September 21, 2021 13:25

akhramov requested a review from AkihiroSuda September 21, 2021 13:28

freebsd: remove bogus build annotations

9d02413

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>

AkihiroSuda approved these changes Sep 21, 2021

View reviewed changes

docs: update for FreeBSD

fef5ff2

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>

AkihiroSuda force-pushed the feature/FreeBSD-support branch from 01a3010 to fef5ff2 Compare September 21, 2021 14:03

AkihiroSuda merged commit be79a84 into containerd:master Sep 21, 2021

This was referenced Sep 21, 2021

FreeBSD CI is flaky: mount_nullfs: ...: Resource deadlock avoided #379

Closed

Cant' build on FreeBSD #137

Closed

AkihiroSuda added the impact/major label Sep 21, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[#137] FreeBSD support #361

[#137] FreeBSD support #361

akhramov commented Sep 15, 2021

AkihiroSuda commented Sep 16, 2021

AkihiroSuda Sep 16, 2021

AkihiroSuda Sep 16, 2021

AkihiroSuda Sep 16, 2021

AkihiroSuda Sep 16, 2021

AkihiroSuda Sep 16, 2021

AkihiroSuda Sep 16, 2021

AkihiroSuda Sep 16, 2021

AkihiroSuda commented Sep 16, 2021

AkihiroSuda commented Sep 16, 2021

akhramov Sep 17, 2021

AkihiroSuda Sep 21, 2021

akhramov Sep 17, 2021

akhramov commented Sep 17, 2021

AkihiroSuda Sep 21, 2021

AkihiroSuda Sep 21, 2021

AkihiroSuda commented Sep 21, 2021 •

edited

akhramov commented Sep 21, 2021

AkihiroSuda Sep 21, 2021

AkihiroSuda Sep 21, 2021

AkihiroSuda left a comment

AkihiroSuda commented Sep 21, 2021 •

edited

leandroscardua commented May 6, 2022

gmshake commented May 7, 2022

leandroscardua commented May 7, 2022

[#137] FreeBSD support #361

[#137] FreeBSD support #361

Conversation

akhramov commented Sep 15, 2021

AkihiroSuda commented Sep 16, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AkihiroSuda commented Sep 16, 2021

AkihiroSuda commented Sep 16, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akhramov commented Sep 17, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AkihiroSuda commented Sep 21, 2021 • edited

akhramov commented Sep 21, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AkihiroSuda left a comment

Choose a reason for hiding this comment

AkihiroSuda commented Sep 21, 2021 • edited

leandroscardua commented May 6, 2022

gmshake commented May 7, 2022

leandroscardua commented May 7, 2022

AkihiroSuda commented Sep 21, 2021 •

edited

AkihiroSuda commented Sep 21, 2021 •

edited