Store BGP passwords in secrets #1264

pperiyasamy · 2022-03-15T12:58:39Z

provides a feature to store bgp peer passwords in secrets instead of plain text.

fedepaol · 2022-03-17T08:08:16Z

@sabinaaledort mind reviewing this?

sabinaaledort · 2022-03-17T12:34:02Z

internal/config/config.go

+			}
+			password = string(dstPass)
+		} else {
+			return nil, fmt.Errorf("invalid secret specified in the secret %q", p.Spec.SecretRef)


can we give more info in the err msg? maybe mentioning the type mismatch.

sure, done.

sabinaaledort · 2022-03-17T12:40:55Z

api/v1beta1/bgppeer_types.go

@@ -81,6 +82,10 @@ type BGPPeerSpec struct {
 	// +optional
 	Password string `json:"password,omitempty" yaml:"password,omitempty"`

+	// secretRef is name of the authentication secret for BGP Peer


secretRef is the name or a reference? i'm still not sure how it's going to look like

changed it to use corev1.SecretReference and added few unit tests. hope that explains it clearly.

sabinaaledort · 2022-03-17T12:43:07Z

looks good. the only thing i'm not sure about is how this field is going to look like. maybe worth adding an example or a unit test? (or maybe it's just me that don't get it)

fedepaol · 2022-03-15T14:48:31Z

internal/config/config.go

+		if err != nil {
+			return nil, err
+		}
+		if secret.Type == corev1.SecretTypeOpaque || secret.Type == corev1.SecretTypeBasicAuth {


If basicauth works, I'd stick with just it. It's easier to document instead of needing to explain how the secret must look like.

fedepaol · 2022-03-15T14:50:53Z

manifests/metallb-frr.yaml

@@ -577,6 +577,7 @@ rules:
  - services
  - endpoints
  - nodes
+  - secrets


This means the speaker will be able to read the secrets of all the namespaces, which is too much.
I'd move it to the Role section, so we limit ourselves to our namespace.

I tried with having secrets in Role, but below error occurred from reconciler while listing secrets from metallb-system namespace, so still holding it in cluster role.
E0317 23:21:27.475103 100 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.2/tools/cache/reflector.go:167: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:metallb-system:speaker" cannot list resource "secrets" in API group "" at the cluster scope
Any idea what can be done to avoid this ?

fedepaol · 2022-03-17T13:52:13Z

api/v1beta1/bgppeer_types.go

@@ -81,6 +82,10 @@ type BGPPeerSpec struct {
 	// +optional
 	Password string `json:"password,omitempty" yaml:"password,omitempty"`

+	// secretRef is name of the authentication secret for BGP Peer
+	// +optional
+	SecretRef *v1.LocalObjectReference `json:"secretRef,omitempty" yaml:"secretRef,omitempty"`


I just found out that there's a corev1.SecretReference which is maybe better https://pkg.go.dev/k8s.io/api/core/v1#SecretReference

Also, I'd add this in the v1beta2 version (which #1245 is adding)

good find! done.

fedepaol · 2022-03-17T13:52:44Z

api/v1beta1/bgppeer_types.go

@@ -81,6 +82,10 @@ type BGPPeerSpec struct {
 	// +optional
 	Password string `json:"password,omitempty" yaml:"password,omitempty"`

+	// secretRef is name of the authentication secret for BGP Peer
+	// +optional
+	SecretRef *v1.LocalObjectReference `json:"secretRef,omitempty" yaml:"secretRef,omitempty"`


I'd rename this as PasswordSecret, meaning the secret that holds the password.

fedepaol · 2022-03-17T13:54:14Z

internal/config/config.go

@@ -169,7 +175,7 @@ func validateHoldTime(ht time.Duration) error {
 }

 // Parse loads and validates a Config from bs.
-func For(resources ClusterResources, validate Validate) (*Config, error) {
+func For(ctx context.Context, cli client.Client, resources ClusterResources, validate Validate) (*Config, error) {


This function is pure, meaning it has no side effects and the result does not depend on the system state.
It's important that stays this way, because otherwise we'd need to inject a mock of the client to test it.

So, we need to collect the secrets upfront and pass them here, in a way or another.

sure, done.

fedepaol · 2022-03-17T13:55:03Z

internal/config/config.go

@@ -319,9 +325,35 @@ func peerFromCR(p metallbv1beta1.BGPPeer) (*Peer, error) {
 		}
 	}

+	if p.Spec.Password != "" && p.Spec.SecretRef != nil {
+		return nil, fmt.Errorf("can not have both password and secret ref set in the peer config")


Return the peer name, it's gonna be easier for the user to understand which bgppeer is wrong

fedepaol · 2022-03-17T13:56:07Z

internal/config/config.go

+				srcPass []byte
+				ok      bool
+			)
+			if srcPass, ok = secret.Data["password"]; !ok {


I'd wrap the parsing in a function.

fedepaol · 2022-03-17T16:32:04Z

e2etest/pkg/config/update.go

-	peer, err := o.peerToCR(p)
+	secretName := fmt.Sprintf("bgppeer-%d-%d-secret", p.ASN, p.Port)
+	existing := &corev1.Secret{}
+	err := o.Client.Get(context.TODO(), types.NamespacedName{Name: secretName, Namespace: o.namespace}, existing)


We need to ensure both work, so I'd add a specific test that just fills the secret and checks the session is established (as opposed to changing all the tests)

yes, added specific e2e tests to cover this.

fedepaol · 2022-03-17T16:32:36Z

internal/config/config.go

@@ -319,9 +325,35 @@ func peerFromCR(p metallbv1beta1.BGPPeer) (*Peer, error) {
 		}
 	}

+	if p.Spec.Password != "" && p.Spec.SecretRef != nil {


Can you add a unit test that verifies this fails?

done, added relevant unit tests to cover this change.

pperiyasamy

Thanks for great design inputs ! rebased it with new changes.

pperiyasamy · 2022-03-18T11:05:50Z

api/v1beta1/bgppeer_types.go

@@ -81,6 +82,10 @@ type BGPPeerSpec struct {
 	// +optional
 	Password string `json:"password,omitempty" yaml:"password,omitempty"`

+	// secretRef is name of the authentication secret for BGP Peer


changed it to use corev1.SecretReference and added few unit tests. hope that explains it clearly.

pperiyasamy · 2022-03-18T11:06:26Z

api/v1beta1/bgppeer_types.go

@@ -81,6 +82,10 @@ type BGPPeerSpec struct {
 	// +optional
 	Password string `json:"password,omitempty" yaml:"password,omitempty"`

+	// secretRef is name of the authentication secret for BGP Peer
+	// +optional
+	SecretRef *v1.LocalObjectReference `json:"secretRef,omitempty" yaml:"secretRef,omitempty"`


good find! done.

pperiyasamy · 2022-03-18T11:06:34Z

api/v1beta1/bgppeer_types.go

@@ -81,6 +82,10 @@ type BGPPeerSpec struct {
 	// +optional
 	Password string `json:"password,omitempty" yaml:"password,omitempty"`

+	// secretRef is name of the authentication secret for BGP Peer
+	// +optional
+	SecretRef *v1.LocalObjectReference `json:"secretRef,omitempty" yaml:"secretRef,omitempty"`


pperiyasamy · 2022-03-18T11:07:40Z

e2etest/pkg/config/update.go

-	peer, err := o.peerToCR(p)
+	secretName := fmt.Sprintf("bgppeer-%d-%d-secret", p.ASN, p.Port)
+	existing := &corev1.Secret{}
+	err := o.Client.Get(context.TODO(), types.NamespacedName{Name: secretName, Namespace: o.namespace}, existing)


yes, added specific e2e tests to cover this.

pperiyasamy · 2022-03-18T11:08:15Z

internal/config/config.go

@@ -169,7 +175,7 @@ func validateHoldTime(ht time.Duration) error {
 }

 // Parse loads and validates a Config from bs.
-func For(resources ClusterResources, validate Validate) (*Config, error) {
+func For(ctx context.Context, cli client.Client, resources ClusterResources, validate Validate) (*Config, error) {


sure, done.

pperiyasamy · 2022-03-18T11:08:59Z

internal/config/config.go

@@ -319,9 +325,35 @@ func peerFromCR(p metallbv1beta1.BGPPeer) (*Peer, error) {
 		}
 	}

+	if p.Spec.Password != "" && p.Spec.SecretRef != nil {
+		return nil, fmt.Errorf("can not have both password and secret ref set in the peer config")


pperiyasamy · 2022-03-18T11:09:10Z

internal/config/config.go

+		if err != nil {
+			return nil, err
+		}
+		if secret.Type == corev1.SecretTypeOpaque || secret.Type == corev1.SecretTypeBasicAuth {


pperiyasamy · 2022-03-18T11:09:20Z

internal/config/config.go

+				srcPass []byte
+				ok      bool
+			)
+			if srcPass, ok = secret.Data["password"]; !ok {


pperiyasamy · 2022-03-18T11:09:32Z

internal/config/config.go

+			}
+			password = string(dstPass)
+		} else {
+			return nil, fmt.Errorf("invalid secret specified in the secret %q", p.Spec.SecretRef)


sure, done.

pperiyasamy · 2022-03-18T11:13:56Z

manifests/metallb-frr.yaml

@@ -577,6 +577,7 @@ rules:
  - services
  - endpoints
  - nodes
+  - secrets


I tried with having secrets in Role, but below error occurred from reconciler while listing secrets from metallb-system namespace, so still holding it in cluster role.
E0317 23:21:27.475103 100 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.2/tools/cache/reflector.go:167: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:metallb-system:speaker" cannot list resource "secrets" in API group "" at the cluster scope
Any idea what can be done to avoid this ?

liornoy · 2022-03-23T16:47:50Z

e2etest/pkg/metallb/config.go

+}
+
+// PeersForContainers2 returns the metallb config peers related to the given containers.
+func PeersForContainers2(containers []*frrcontainer.FRR, ipFamily ipfamily.Family, useSecret bool) []config.Peer {


Just a suggestion - maybe a more meaningful function name here?

fedepaol · 2022-03-25T16:42:40Z

internal/config/config.go

+	} else if p.Spec.PasswordSecret.Name != "" {
+		var secret corev1.Secret
+		var ok bool
+		if secret, ok = passwordSecrets[p.Spec.PasswordSecret.Name]; !ok {


I'd rather have the := form and checking if !ok on the next line
secret, ok: = ...
if !ok {
}

fedepaol · 2022-03-25T16:44:11Z

internal/config/config.go

@@ -370,9 +372,23 @@ func peerFromCR(p metallbv1beta2.BGPPeer) (*Peer, error) {
 		nodeSels = []labels.Selector{labels.Everything()}
 	}

+	if p.Spec.Password != "" && p.Spec.PasswordSecret.Name != "" {


How about wrapping this all in a passwordForPeer function ?

fedepaol · 2022-03-25T16:44:59Z

internal/config/config.go

+		srcPass []byte
+		ok      bool
+	)
+	if srcPass, ok = secret.Data["password"]; !ok {


Same:
pass, ok := ...
if !ok {
}

fedepaol · 2022-03-25T16:49:05Z

internal/k8s/controllers/config_controller.go

+		return ctrl.Result{}, err
+	}
+
+	metallbCRsAndSecrets := config.ClusterResources{


I'd just call this "resources"

fedepaol · 2022-03-25T16:50:04Z

internal/k8s/controllers/config_controller.go

 		Pools:              ipPools.Items,
 		Peers:              bgpPeers.Items,
 		BFDProfiles:        bfdProfiles.Items,
 		L2Advs:             l2Advertisements.Items,
 		BGPAdvs:            bgpAdvertisements.Items,
 		LegacyAddressPools: addressPools.Items,
 	}
+	metallbCRsAndSecrets.PasswordSecrets = make(map[string]corev1.Secret)


I'd wrap this (and the List above) in a func to put in the bottom of the file, and then use the result when you do the literal initializer above.

fedepaol · 2022-03-25T16:51:10Z

e2etest/pkg/metallb/config.go

@@ -74,3 +85,22 @@ func WithRouterID(peers []metallbv1beta2.BGPPeer, routerID string) []metallbv1be
 	}
 	return peers
 }
+
+func BGPPeerSecretRefernces(containers []*frrcontainer.FRR) map[string]corev1.Secret {


nit: BGPPeerSercretReferences

fedepaol · 2022-03-25T16:53:26Z

e2etest/pkg/metallb/config.go

+}
+
+// PeersForContainersWithSecret returns the metallb config peers related to the given containers.
+func PeersForContainersWithSecret(containers []*frrcontainer.FRR, ipFamily ipfamily.Family, useSecret bool) []metallbv1beta2.BGPPeer {


I would use a vararg with a modifier function, pretty much like we do in

metallb/e2etest/pkg/service/create.go

Line 20 in 10a0249

func CreateWithBackend(cs clientset.Interface, namespace string, jigName string, tweak ...func(svc *corev1.Service)) (*corev1.Service, *e2eservice.TestJig) {

and expose a WithSecret func that does the modification.

fedepaol · 2022-03-25T16:57:08Z

Re: using namespaced secrets, I filed #1287 today, which seems to make the trick.
I'd wait that pr to get in (that in turn depends on #1275), the changes will happen mostly at manifest level and a few lines in k8s.go.

fedepaol · 2022-03-28T08:46:15Z

internal/config/config.go

+	}
+	var password string
+	if p.Spec.Password != "" {
+		password = p.Spec.Password


you can just return here

fedepaol · 2022-03-28T08:49:52Z

e2etest/pkg/metallb/config.go

@@ -74,3 +78,22 @@ func WithRouterID(peers []metallbv1beta2.BGPPeer, routerID string) []metallbv1be
 	}
 	return peers
 }
+
+func BGPPeerSecretReferences(containers []*frrcontainer.FRR) map[string]corev1.Secret {


This can return a slice, I think?

I think this can be still be a map to align with PasswordSecrets in ClusterResources, I chose map over slice as its mainly used for a secret lookup for secret ref in bgp peer.

You are right.

fedepaol · 2022-03-31T11:31:32Z

@pperiyasamy #1287 was merged, you can now make the secrets namescoped too

This makes BGPPeer refer to k8s secret object of type kubernetes.io/basic-auth contains only with password. It enables BGP peer password stored as a secret instead of plain text. Signed-off-by: Periyasamy Palanisamy <pepalani@redhat.com>

This consumes newly introduced PasswordSecret field in the speaker when provided and it also honors existing password field. Signed-off-by: Periyasamy Palanisamy <pepalani@redhat.com>

This validates BGP pairing with router container when BGPPeer CR uses secret ref for storing BGP peer password. Signed-off-by: Periyasamy Palanisamy <pepalani@redhat.com>

Since there is no need for secrets to be in cluster role, this commit moves that into namespaced scope. Signed-off-by: Periyasamy Palanisamy <pepalani@redhat.com>

pperiyasamy · 2022-04-01T11:34:10Z

@pperiyasamy #1287 was merged, you can now make the secrets namescoped too

Thanks @fedepaol for the PR #1287, now rebased it accordingly.

fedepaol · 2022-04-01T11:36:34Z

LGTM, thanks @pperiyasamy !

junkiebev · 2022-07-12T20:59:17Z

In the event that one uses a secret ref:

I'd presume that that secret is required to be in the same namespace as the MetalLB deployment, but the docs don't mention that
What's the Key it uses to grab a value? I'd guess password, based on my reading, but the docs don't specify

fedepaol · 2022-07-12T21:09:14Z

Hi @junkiebev , thanks for the feedback! You are right, the documentation is lacking details about this change, as it slipped under the other changes of the last release.
Would you mind filing an issue for this so we won't loose track of it? Thanks again.

pperiyasamy requested review from fedepaol, johananl, rata and russellb as code owners March 15, 2022 12:58

pperiyasamy changed the title ~~[WiP] Store BGP passwords in secrets~~ WIP: Store BGP passwords in secrets Mar 15, 2022

pperiyasamy force-pushed the bgppeer-password-as-secret branch 2 times, most recently from 65faa9b to e989afc Compare March 16, 2022 13:38

pperiyasamy changed the title ~~WIP: Store BGP passwords in secrets~~ Store BGP passwords in secrets Mar 16, 2022

sabinaaledort reviewed Mar 17, 2022

View reviewed changes

fedepaol reviewed Mar 17, 2022

View reviewed changes

fedepaol mentioned this pull request Mar 17, 2022

Read BGP password via Secret reference metallb/metallb-operator#176

Closed

pperiyasamy force-pushed the bgppeer-password-as-secret branch from e989afc to 10950ff Compare March 18, 2022 11:14

pperiyasamy commented Mar 18, 2022

View reviewed changes

pperiyasamy force-pushed the bgppeer-password-as-secret branch 3 times, most recently from 271e8e0 to bc3ebf9 Compare March 22, 2022 09:25

liornoy reviewed Mar 23, 2022

View reviewed changes

pperiyasamy force-pushed the bgppeer-password-as-secret branch 2 times, most recently from 46584e8 to 5643582 Compare March 25, 2022 09:00

fedepaol reviewed Mar 25, 2022

View reviewed changes

pperiyasamy force-pushed the bgppeer-password-as-secret branch from 5643582 to 1e364f5 Compare March 28, 2022 08:15

fedepaol reviewed Mar 28, 2022

View reviewed changes

pperiyasamy force-pushed the bgppeer-password-as-secret branch from 1e364f5 to 3e70452 Compare April 1, 2022 07:21

pperiyasamy added 4 commits April 1, 2022 10:19

Add PasswordSecret field into BGPPeer CRD

201e1c3

This makes BGPPeer refer to k8s secret object of type kubernetes.io/basic-auth contains only with password. It enables BGP peer password stored as a secret instead of plain text. Signed-off-by: Periyasamy Palanisamy <pepalani@redhat.com>

Adapt Speaker for new PasswordSecret field

ea56535

This consumes newly introduced PasswordSecret field in the speaker when provided and it also honors existing password field. Signed-off-by: Periyasamy Palanisamy <pepalani@redhat.com>

E2E: Add test for using BGPPeer secret

93375ff

This validates BGP pairing with router container when BGPPeer CR uses secret ref for storing BGP peer password. Signed-off-by: Periyasamy Palanisamy <pepalani@redhat.com>

Make secrets rbac into namespaced scope

782d134

Since there is no need for secrets to be in cluster role, this commit moves that into namespaced scope. Signed-off-by: Periyasamy Palanisamy <pepalani@redhat.com>

pperiyasamy force-pushed the bgppeer-password-as-secret branch from 3e70452 to 782d134 Compare April 1, 2022 08:22

fedepaol merged commit f14aefd into metallb:main Apr 1, 2022

pperiyasamy deleted the bgppeer-password-as-secret branch April 1, 2022 13:20

junkiebev mentioned this pull request Jul 12, 2022

Store BGP passwords in secrets #408

Closed

junkiebev mentioned this pull request Jul 12, 2022

[documentation] BGP Secret Reference Usage #1500

Closed

cprivitere mentioned this pull request Oct 12, 2022

Add MetalLB 0.13 support kubernetes-sigs/cloud-provider-equinix-metal#322

Closed

eyenx mentioned this pull request Oct 19, 2022

chore(misc/metallb): update to 0.13.x adfinis/helm-charts#834

Merged

6 tasks

Store BGP passwords in secrets #1264

Store BGP passwords in secrets #1264

Conversation

pperiyasamy commented Mar 15, 2022

fedepaol commented Mar 17, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sabinaaledort commented Mar 17, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pperiyasamy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fedepaol commented Mar 25, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fedepaol commented Mar 31, 2022

pperiyasamy commented Apr 1, 2022

fedepaol commented Apr 1, 2022

junkiebev commented Jul 12, 2022 • edited

fedepaol commented Jul 12, 2022

junkiebev commented Jul 12, 2022 •

edited