[WIP] Add fuse-manager #324

ilyee · 2021-05-19T06:47:12Z

What is fusemanager

fuse-manager is a fuse daemon that handles all fuse processes, so fuse goroutines won't get killed when we restart stargz-snapshotter, and runnings containers don't need to be restarted.

By separating out fuse-manager, filesystem operations will all be handed by fuse-manager, and stargz-snapshotter is only responsable of snapshot operations.

How to run it

Run fuse-manager with option fusestore (address for the fusemanager's store) and socket (address for the fusemanager's gRPC socket)
Run stargz-snapshotter with option fusemanager-address (same with socket), and enable-fusemanager with 'true' to enable it

How it works

fuse-manager interacts with stargz-snapshotter through gRPC, its client implements Filesystem interface, so can be passed to snbase.NewSnapshotter to create a new snapshotter instance.

After running grpc server, fuse-manager waits for Init method to be called. When stargz-snapshotter starts(or restarts), Init will be called to pass fs related configs to fuse-manager, which will create a new fs instance for handling new incoming Mount/Check/Unmount requests.

ilyee · 2021-05-19T06:48:10Z

This pr refers to issue #318

ktock

Thank you very much for PR!

snapshot/snapshot.go

fs/fs.go

fusemanager/fusemanager.go

fusemanager/api/api.proto

fusemanager/fusemanager.go

fusemanager/fusestore.go

cmd/containerd-stargz-grpc/main.go

cmd/fuse-manager/main.go

cmd/containerd-stargz-grpc/main.go

fusemanager/fusemanager.go

AkihiroSuda

What's the purpose of this? Why not just reimplement reloading config?
s/fuse-manager/stargz-fuse-manager/
How is the fuse-manager daemon started?

ilyee · 2021-05-20T06:40:50Z

@AkihiroSuda

What's the purpose of this? Why not just reimplement reloading config?

As mentioned in #318 , reloading config needs a restart of snapshotter, so we need to restart all running containers and restore stargz layers. And a separate fuse daemon will focuse on fuse goroutines so snapshotter don't need to handle their lifecycles, a config update will be done quickly.

How is the fuse-manager daemon started?

fuse-manager should be started before stargz-snapshotter.

Its startup command is like fuse-manager -fusestore=/var/lib/containerd-stargz-grpc/fusestore.db -socket=/run/containerd-stargz-grpc/fuse-manager.sock with this options:

socket: gRPC socket path
fusestore: path to backend storage of fs related configs with mountpoint as key to restore fuse mounts after restarting fuse-manager
log-level: logging level

And with option fusemanager-address we can enable stargz-snapshotter to use fuse-manager, otherwise it will run without fuse-manager as well.

AkihiroSuda · 2021-05-20T06:51:25Z

fuse-manager should be started before stargz-snapshotter.

Needs systemd unit
https://github.com/containerd/stargz-snapshotter/tree/02bf457fbb2044ead1d4958e364010f42b43e6ea/script/config/etc/systemd/system

AkihiroSuda · 2021-05-20T06:54:13Z

Or can we let stargz-snapshotter start stargz-fuse-manager in the background and detach it, just like containerd starts containerd-shim-runc-v2 processes in the background and detach them.

ilyee · 2021-05-20T06:56:31Z

Or can we let stargz-snapshotter start stargz-fuse-manager in the background and detach it, just like containerd starts containerd-shim-runc-v2 processes in the background and detach them.

It makes sense, I'll try both and make an update

ktock · 2021-05-20T06:58:57Z

SGTM about the shim-like approach.

ilyee · 2021-06-03T12:18:35Z

Now stargz-fuse-manager uses a shim-like approach to start

Please take a look if free @AkihiroSuda @ktock

ktock

Overall looks good but some minor nits

cmd/containerd-stargz-grpc/main.go

fusemanager/api/api.proto

cmd/containerd-stargz-grpc/main.go

ktock · 2021-06-23T14:08:03Z

@AkihiroSuda PTAL when you have time

Before we merge this, we need to

merge ~~[WIP] Add fuse-manager #324~~ Makefile: add make generate for proto code generation #341,
rebase this PR on that patch and
make sure everything work well (including make validate-generated in CI)

let me turn this PR into draft until we complete them for avoiding to accidentally merge this PR.

cmd/containerd-stargz-grpc/main.go

fusemanager/fusemanager.go

AkihiroSuda

Can we have a design doc to explain why we need this?

ilyee · 2021-07-20T03:12:38Z

let me turn this PR into draft until we complete them for avoiding to accidentally merge this PR.

@ktock seems like these have been completed, can draft be removed?

ktock · 2021-07-20T12:11:10Z

@ilyee Could you add changes for #324 (review) ?

ilyee · 2021-07-20T12:22:03Z

@ilyee Could you add changes for #324 (review) ?

sure, I'll handle it

ilyee · 2021-08-13T15:53:04Z

@ilyee Could you add changes for #324 (review) ?

@ktock related doc is added in doc/overview.md, PTAL if free, thx

AkihiroSuda · 2021-08-26T01:40:51Z

Needs rebase

Signed-off-by: Zuti He <ilyeeelihe@gmail.com>

ilyee · 2021-09-06T12:23:57Z

Needs rebase

@AkihiroSuda done

ktock

Please see comments.

We need to think about having tests for this.

ktock · 2021-09-13T11:53:56Z

cmd/containerd-stargz-grpc/main.go

@@ -58,14 +61,19 @@ const (
 	defaultLogLevel            = logrus.InfoLevel
 	defaultRootDir             = "/var/lib/containerd-stargz-grpc"
 	defaultImageServiceAddress = "/run/containerd/containerd.sock"
+	defaultFuseManagerAddress  = "/run/containerd-stargz-grpc/fuse-namanger.sock"


This typo doesn't seem to be fixed (/run/containerd-stargz-grpc/fuse-manager.sock)

ktock · 2021-09-13T11:59:45Z

fusemanager/service.go

+			}
+			return runtime.NewImageServiceClient(conn), nil
+		}
+		f, _ := cri.NewCRIKeychain(ctx, connectCRI)


Why CRI server is ignored here? fuse-manager should provide CRI proxy as well.

ktock · 2021-09-13T12:02:26Z

snapshot/snapshot.go

+	if config.restoreSnapshots {
+		if err := o.restoreRemoteSnapshot(ctx); err != nil {
+			return nil, errors.Wrap(err, "failed to restore remote snapshot")
+		}


We should not change the default behaviour here. Restoring should be enabled by default and disabled by an option.

ktock · 2021-09-13T12:02:47Z

snapshot/snapshot.go

 	ctx := context.Background()
-	if err := o.cleanup(ctx, cleanupCommitted); err != nil {
+	if err := o.cleanup(ctx, o.cleanupCommitted); err != nil {


We should not change the default behaviour here. cleanupCommitted should be enabled by default and disabled by an option.

ktock reviewed May 19, 2021

View reviewed changes

snapshot/snapshot.go Outdated Show resolved Hide resolved

fs/fs.go Outdated Show resolved Hide resolved

ktock reviewed May 19, 2021

View reviewed changes

fusemanager/fusemanager.go Outdated Show resolved Hide resolved

ilyee force-pushed the fuse-manager branch 2 times, most recently from 7e7a0f0 to ccd8af2 Compare May 19, 2021 12:02

ilyee changed the title ~~[WIP] Add fuse-manager~~ Add fuse-manager May 19, 2021

ktock reviewed May 20, 2021

View reviewed changes

ilyee changed the title ~~Add fuse-manager~~ [WIP] Add fuse-manager May 20, 2021

ilyee force-pushed the fuse-manager branch 2 times, most recently from ca6c291 to 1745155 Compare May 20, 2021 05:13

AkihiroSuda requested changes May 20, 2021

View reviewed changes

AkihiroSuda added enhancement New feature or request status/needs-rebase impact/changelog labels May 20, 2021

ilyee force-pushed the fuse-manager branch 10 times, most recently from 6233527 to 4f1e4c6 Compare June 3, 2021 11:25

ktock reviewed Jun 23, 2021

View reviewed changes

cmd/containerd-stargz-grpc/main.go Outdated Show resolved Hide resolved

fusemanager/api/api.proto Outdated Show resolved Hide resolved

cmd/containerd-stargz-grpc/main.go Outdated Show resolved Hide resolved

ktock marked this pull request as draft June 23, 2021 14:08

ktock self-assigned this Jun 23, 2021

ilyee force-pushed the fuse-manager branch from 07bf8c7 to dc6b185 Compare June 23, 2021 14:09

AkihiroSuda reviewed Jun 23, 2021

View reviewed changes

cmd/containerd-stargz-grpc/main.go Outdated Show resolved Hide resolved

AkihiroSuda reviewed Jun 23, 2021

View reviewed changes

cmd/containerd-stargz-grpc/main.go Show resolved Hide resolved

AkihiroSuda reviewed Jun 23, 2021

View reviewed changes

cmd/containerd-stargz-grpc/main.go Outdated Show resolved Hide resolved

AkihiroSuda reviewed Jun 23, 2021

View reviewed changes

fusemanager/fusemanager.go Show resolved Hide resolved

AkihiroSuda reviewed Jun 23, 2021

View reviewed changes

ilyee force-pushed the fuse-manager branch 2 times, most recently from dca6dc9 to 33eaed8 Compare June 27, 2021 05:18

ilyee force-pushed the fuse-manager branch 2 times, most recently from ddfb938 to d52ebb4 Compare July 20, 2021 12:08

ilyee force-pushed the fuse-manager branch 2 times, most recently from 0cdde0e to 87101f6 Compare August 13, 2021 10:10

ilyee marked this pull request as ready for review August 13, 2021 15:34

AkihiroSuda marked this pull request as draft August 26, 2021 01:40

Add fuse-manager

c2625b3

Signed-off-by: Zuti He <ilyeeelihe@gmail.com>

ilyee force-pushed the fuse-manager branch from 87101f6 to c2625b3 Compare September 6, 2021 11:23

AkihiroSuda removed the status/needs-rebase label Sep 6, 2021

ilyee marked this pull request as ready for review September 7, 2021 03:56

ktock reviewed Sep 13, 2021

View reviewed changes

Kern-- mentioned this pull request Mar 22, 2023

[FEATURE] Split soci snapshotter into fuse and snapshotter processes awslabs/soci-snapshotter#518

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Add fuse-manager #324

[WIP] Add fuse-manager #324

ilyee commented May 19, 2021

ilyee commented May 19, 2021

ktock left a comment

AkihiroSuda left a comment

ilyee commented May 20, 2021 •

edited

AkihiroSuda commented May 20, 2021

AkihiroSuda commented May 20, 2021

ilyee commented May 20, 2021

ktock commented May 20, 2021

ilyee commented Jun 3, 2021

ktock left a comment

ktock commented Jun 23, 2021 •

edited

AkihiroSuda left a comment

ilyee commented Jul 20, 2021

ktock commented Jul 20, 2021

ilyee commented Jul 20, 2021

ilyee commented Aug 13, 2021 •

edited

AkihiroSuda commented Aug 26, 2021

ilyee commented Sep 6, 2021

ktock left a comment

ktock Sep 13, 2021

ktock Sep 13, 2021

ktock Sep 13, 2021

ktock Sep 13, 2021

[WIP] Add fuse-manager #324

Are you sure you want to change the base?

[WIP] Add fuse-manager #324

Conversation

ilyee commented May 19, 2021

What is fusemanager

How to run it

How it works

ilyee commented May 19, 2021

ktock left a comment

Choose a reason for hiding this comment

AkihiroSuda left a comment

Choose a reason for hiding this comment

ilyee commented May 20, 2021 • edited

AkihiroSuda commented May 20, 2021

AkihiroSuda commented May 20, 2021

ilyee commented May 20, 2021

ktock commented May 20, 2021

ilyee commented Jun 3, 2021

ktock left a comment

Choose a reason for hiding this comment

ktock commented Jun 23, 2021 • edited

AkihiroSuda left a comment

Choose a reason for hiding this comment

ilyee commented Jul 20, 2021

ktock commented Jul 20, 2021

ilyee commented Jul 20, 2021

ilyee commented Aug 13, 2021 • edited

AkihiroSuda commented Aug 26, 2021

ilyee commented Sep 6, 2021

ktock left a comment

Choose a reason for hiding this comment

ktock Sep 13, 2021

Choose a reason for hiding this comment

ktock Sep 13, 2021

Choose a reason for hiding this comment

ktock Sep 13, 2021

Choose a reason for hiding this comment

ktock Sep 13, 2021

Choose a reason for hiding this comment

ilyee commented May 20, 2021 •

edited

ktock commented Jun 23, 2021 •

edited

ilyee commented Aug 13, 2021 •

edited