Store auth in keyring and transfer it to nydusd via environment variable #512

sctb512 · 2023-07-21T11:40:04Z

Store auth for nydusd in the keyring instead of the configuration file.
Set auth via environment variable to avoid displaying it in command parameter.

Related to dragonflyoss/nydus#1381 and dragonflyoss/nydus#1382

codecov · 2023-07-21T12:22:07Z

Codecov Report

Merging #512 (c317c16) into main (0e3857e) will increase coverage by 0.53%.
Report is 2 commits behind head on main.
The diff coverage is 50.76%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #512      +/-   ##
==========================================
+ Coverage   37.83%   38.37%   +0.53%     
==========================================
  Files          60       62       +2     
  Lines        7092     7339     +247     
==========================================
+ Hits         2683     2816     +133     
- Misses       4097     4188      +91     
- Partials      312      335      +23

Files Changed	Coverage Δ
config/config.go	`31.38% <ø> (ø)`
config/daemonconfig/daemonconfig.go	`0.00% <0.00%> (ø)`
config/global.go	`30.86% <0.00%> (-0.79%)`	⬇️
pkg/daemon/daemon.go	`0.00% <0.00%> (ø)`
pkg/manager/daemon_adaptor.go	`0.00% <0.00%> (ø)`
pkg/manager/manager.go	`14.84% <0.00%> (-0.05%)`	⬇️
pkg/auth/keyring.go	`60.11% <60.11%> (ø)`
pkg/auth/cache.go	`64.58% <64.58%> (ø)`

... and 2 files with indirect coverage changes

changweige · 2023-07-31T02:30:59Z

pkg/manager/daemon_adaptor.go

@@ -192,6 +195,19 @@ func (m *Manager) BuildDaemonCommand(d *daemon.Daemon, bin string, upgrade bool)

 	cmd := exec.Command(nydusdPath, args...)

+	if config.IsKeyringEnabled() && !config.IsFusedevSharedModeEnabled() {


Daemon has its own working mode which means nydusd daemons can work in the different modes in a single node.
Better to judge by daemon, we already have some helper functions doing this

changweige · 2023-07-31T02:34:02Z

pkg/filesystem/fs.go

-		err = daemonconfig.SupplementDaemonConfig(cfg, imageID, snapshotID, false, labels, params)
+		var updateErr error
+		err = daemonconfig.SupplementDaemonConfig(cfg, imageID, snapshotID, false, labels, params, func(imageHost string, keyChain *auth.PassKeyChain) {
+			logrus.Infof("add key for %s", imageHost)


It's better to lower the log level to Debug

changweige · 2023-07-31T02:45:21Z

pkg/manager/daemon_adaptor.go

@@ -192,6 +195,19 @@ func (m *Manager) BuildDaemonCommand(d *daemon.Daemon, bin string, upgrade bool)

 	cmd := exec.Command(nydusdPath, args...)

+	if config.IsKeyringEnabled() && !config.IsFusedevSharedModeEnabled() {
+		c, err := daemonconfig.NewDaemonConfig(d.States.FsDriver, configPath)


I am afraid the current way to identify a container image reference's host is not agile. For the dedicated mode nydusd daemon, it only has a rafs instance in its Instances where we can calculate what host is

changweige · 2023-07-31T02:46:11Z

pkg/keyring/keyring.go

@@ -0,0 +1,42 @@
+package keyring


Please add license claim

changweige · 2023-07-31T02:48:54Z

pkg/daemon/daemon.go

@@ -66,6 +71,7 @@ type Daemon struct {
 	// fusedev shared mode: zero, one or more RAFS instances
 	// fscache shared mode: zero, one or more RAFS instances
 	Instances rafsSet
+	authCache *lru.Cache


Auth should be shared among different nydusd daemons. So why to bind the cache to a single nydusd daemon?

Fixed. Move it to the manager.

changweige · 2023-07-31T02:50:15Z

pkg/daemon/daemon.go

@@ -151,6 +158,31 @@ func (d *Daemon) RemoveInstance(snapshotID string) {
 	d.DecRef()
 }

+func (d *Daemon) UpdateAuth(imageHost, auth string) error {
+	key, err := keyring.Add(imageHost, auth)


Is it necessary to consider the atomicity when updating?

changweige · 2023-07-31T02:50:59Z

pkg/daemon/daemon.go

+	}
+
+	data, err := keyring.Search(imageHost)
+	if err == nil {


Checking if err is nil looks strange.

imeoer · 2023-07-31T02:54:31Z

This approach seems to only solve the nydusd registry credential persistence problem, but when the credential is changed (e.g. registry user/pass dynamically generated by a credential helper), nydusd may not be able to timely detect the update. I think #504 is a better implementation.

sctb512 · 2023-08-01T06:14:59Z

This approach seems to only solve the nydusd registry credential persistence problem, but when the credential is changed (e.g. registry user/pass dynamically generated by a credential helper), nydusd may not be able to timely detect the update. I think #504 is a better implementation.

I believe this PR does not conflict with #504. It is better to get authentication from the credential server if snapshotter does not send it through env.

imeoer · 2023-08-02T02:08:33Z

pkg/auth/cache.go

+
+import (
+	"github.com/pkg/errors"
+	"k8s.io/utils/lru"


It seems we have used github.com/golang/groupcache/lru.

imeoer · 2023-08-02T02:11:48Z

pkg/auth/keyring.go

+	lock *sync.RWMutex
+}
+
+func NewKeyRing() *KeyRing {


Maybe give some unit test cases for this?

Agree, please try to add some unit test, especially for keying garbage collection and if the boundary is correctly handled.

imeoer · 2023-08-02T02:23:01Z

pkg/manager/daemon_adaptor.go

@@ -192,6 +199,25 @@ func (m *Manager) BuildDaemonCommand(d *daemon.Daemon, bin string, upgrade bool)

 	cmd := exec.Command(nydusdPath, args...)

+	if config.IsKeyringEnabled() && !d.IsSharedDaemon() {


Can we support shared daemon?

Yes, it calls c.FillAuth(keyChain) to fill auth before sending mount request to nydus daemon.

changweige

Could we also have an e2e test case defined in integration/entrypoint.sh

changweige · 2023-08-04T01:50:35Z

config/daemonconfig/daemonconfig.go

-		c.FillAuth(keyChain)
+		if config.IsKeyringEnabled() && fn != nil {
+			if err := fn(registryHost, keyChain); err != nil {
+				if errors.Is(err, unix.EINVAL) {


I think the error type should be transparent for the caller

Signed-off-by: Bin Tang <tangbin.bin@bytedance.com>

sctb512 changed the title ~~Store auth in keyring and set to nydusd via environment variable~~ Store auth in keyring and transfer it to nydusd via environment variable Jul 21, 2023

sctb512 force-pushed the get-auth-from-keyring branch 6 times, most recently from af671f5 to d03befe Compare July 27, 2023 08:32

changweige reviewed Jul 31, 2023

View reviewed changes

sctb512 force-pushed the get-auth-from-keyring branch 4 times, most recently from 30ee1e2 to 25b9d09 Compare August 1, 2023 06:08

sctb512 force-pushed the get-auth-from-keyring branch 6 times, most recently from 92d4b96 to e79e344 Compare August 1, 2023 08:11

sctb512 requested a review from changweige August 1, 2023 11:15

imeoer reviewed Aug 2, 2023

View reviewed changes

sctb512 force-pushed the get-auth-from-keyring branch from e79e344 to 44b5fee Compare August 2, 2023 07:14

sctb512 force-pushed the get-auth-from-keyring branch 27 times, most recently from ee9c16d to a9c1dd0 Compare August 4, 2023 01:57

changweige reviewed Aug 4, 2023

View reviewed changes

keyring: for test

c317c16

Signed-off-by: Bin Tang <tangbin.bin@bytedance.com>

sctb512 force-pushed the get-auth-from-keyring branch from a9c1dd0 to c317c16 Compare August 4, 2023 02:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Store auth in keyring and transfer it to nydusd via environment variable #512

Store auth in keyring and transfer it to nydusd via environment variable #512

sctb512 commented Jul 21, 2023 •

edited

codecov bot commented Jul 21, 2023 •

edited

changweige Jul 31, 2023

sctb512 Aug 1, 2023

changweige Jul 31, 2023

sctb512 Aug 1, 2023

changweige Jul 31, 2023 •

edited

sctb512 Aug 1, 2023

changweige Jul 31, 2023

sctb512 Aug 1, 2023

changweige Jul 31, 2023

sctb512 Aug 1, 2023

changweige Jul 31, 2023

sctb512 Aug 1, 2023

changweige Jul 31, 2023

sctb512 Aug 1, 2023

imeoer commented Jul 31, 2023

sctb512 commented Aug 1, 2023 •

edited

imeoer Aug 2, 2023

sctb512 Aug 2, 2023

imeoer Aug 2, 2023

changweige Aug 2, 2023

sctb512 Aug 2, 2023

imeoer Aug 2, 2023

sctb512 Aug 2, 2023 •

edited

changweige left a comment

changweige Aug 4, 2023

		@@ -192,6 +195,19 @@ func (m Manager) BuildDaemonCommand(d daemon.Daemon, bin string, upgrade bool)

		cmd := exec.Command(nydusdPath, args...)

		if config.IsKeyringEnabled() && !config.IsFusedevSharedModeEnabled() {

		@@ -192,6 +199,25 @@ func (m Manager) BuildDaemonCommand(d daemon.Daemon, bin string, upgrade bool)

		cmd := exec.Command(nydusdPath, args...)

		if config.IsKeyringEnabled() && !d.IsSharedDaemon() {

Store auth in keyring and transfer it to nydusd via environment variable #512

Are you sure you want to change the base?

Store auth in keyring and transfer it to nydusd via environment variable #512

Conversation

sctb512 commented Jul 21, 2023 • edited

codecov bot commented Jul 21, 2023 • edited

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

changweige Jul 31, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

imeoer commented Jul 31, 2023

sctb512 commented Aug 1, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sctb512 Aug 2, 2023 • edited

Choose a reason for hiding this comment

changweige left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sctb512 commented Jul 21, 2023 •

edited

codecov bot commented Jul 21, 2023 •

edited

changweige Jul 31, 2023 •

edited

sctb512 commented Aug 1, 2023 •

edited

sctb512 Aug 2, 2023 •

edited