Create and populate OIDC blob store for the cluster #3564

rajdeepc2792 · 2024-05-07T16:44:19Z

Which issue this PR addresses:

Jira issue :- ARO-4373
Related Docs:-
https://msazure.visualstudio.com/AzureRedHatOpenShift/_wiki/wikis/AzureRedHatOpenShift.wiki/603739/OIDC-Traffic-Flow
Similar Implementation:-
https://gitlab.cee.redhat.com/service/uhc-clusters-service/-/tree/master/pkg/aws/cloudcredentialbuilder

What this PR does / why we need it:

The RP needs to generate a keypair
The private key needs to be stored in cosmos, It needs to be encrypted if it's stored in cosmos
The private key will be passed to the installer-aro-wrapper via a secret as part of the existing cluster document in CosmosDB(Not implemented here)
Set / populate OIDC issuerURL in the internal API
The RP should generate OIDC docs
Should populate them in regional storage accounts
The OIDC doc needs to be resolvable through the regional AFD instances and should follow a hashing pattern

Test plan for issue:

Is there any documentation that needs to be updated for this PR?

Doc for setting up the full RP. (Part of the PR)

How do you know this will function as expected in production?

Testing the implementation in all the environments.

rajdeepc2792 · 2024-05-07T17:07:47Z

/azp run ci,e2e

azure-pipelines · 2024-05-07T17:07:59Z

Azure Pipelines successfully started running 2 pipeline(s).

cadenmarchese

not a full review, just a few questions as you're working through the draft and testing. looks great!

pkg/util/storage/manager.go

github-actions · 2024-05-07T18:45:39Z

Please rebase pull request.

rajdeepc2792 · 2024-05-08T18:11:59Z

/azp run e2e

azure-pipelines · 2024-05-08T18:12:05Z

Pull request contains merge conflicts.

pkg/api/openshiftcluster.go

bennerv · 2024-05-08T19:41:32Z

pkg/api/openshiftcluster.go

+	ResourceGroupID               string               `json:"resourceGroupId,omitempty"`
+	FipsValidatedModules          FipsValidatedModules `json:"fipsValidatedModules,omitempty"`
+	OIDCIssuer                    OIDCIssuer           `json:"oidcIssuer,omitempty"`
+	BoundServiceAccountSigningKey SecureString         `json:"boundServiceAccountSigningKey,omitempty"`


we're not going to populate this on older clusters, correct? So this field should only be set on WI clusters?

Yes this value will only be generated and populated for MIWI clusters as part of OIDC blob creations for the clusters.

Do you want to make it a pointer then?

Changed it SecureBytes now.

pkg/cluster/deploybaseresources.go

pkg/cluster/install.go

pkg/env/env.go

pkg/frontend/asyncoperationresult_get.go

bennerv · 2024-05-08T19:47:48Z

pkg/util/oidcbuilder/jwks.go

+)
+
+func CreateKeyPair() (encPrivateKey []byte, encPublicKey []byte, err error) {
+	bitSize := 4096


is this the bit size & recommendation of OpenShift for SA private keys?

Creation of jwks is mostly copied from Cloud-credential-operator same as ROSA STS

4096 should be ok for FIPS compliance as well: https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-56Br2.pdf

Also - would it be possible to have the cloud credential operator folks publish functions for generating key pairs, so that way ROSA and ARO can consume those directly instead of repeating the same code?

@niontive they do actually: https://github.com/openshift/cloud-credential-operator/blob/master/docs/ccoctl.md#creating-rsa-keys-1 It's a CLI tool though, it's a bit different for us since we own the OIDC stuff.

github-actions · 2024-05-13T15:32:05Z

Please rebase pull request.

pkg/api/openshiftcluster.go

pkg/cluster/install.go

pkg/cluster/deploybaseresources.go

rajdeepc2792 · 2024-05-31T21:42:30Z

/azp run ci

azure-pipelines · 2024-05-31T21:42:40Z

Azure Pipelines successfully started running 1 pipeline(s).

pkg/cluster/delete.go

niontive · 2024-06-03T16:38:57Z

pkg/env/prod.go

@@ -261,6 +273,19 @@ func (p *prod) ACRDomain() string {
 	return p.acrDomain
 }

+func (p *prod) OIDCStorageAccountName() string {


My main concern is we'll have to maintain the storage account name in two separate places. Ex: https://msazure.visualstudio.com/AzureRedHatOpenShift/_git/ARO-Pipelines?path=/rp/oidc/Region/Templates/modules/storage.bicep

Would you be good if we:

Define the storage account names in rp-config

Have ARO-Pipelines and ARO-RP consume that name from rp-config

cc @gouthamMN

Thanks Nic for raising this, I agree and back your idea for the consistency. It was hard anyways to get the container naming using the assumption present in the function.
Although I would like to get the consensus on the this.
cc: @cadenmarchese

As per discussion, this is going to be handle as a separate issue after the PR is merged. I will post the Jira story for the same here when created.

niontive · 2024-06-03T17:05:30Z

pkg/util/oidcbuilder/jwks.go

+	"fmt"
+
+	"github.com/pkg/errors"
+	"gopkg.in/square/go-jose.v2"


Looks like Square deprecated this and it's maintained here now: https://github.com/go-jose/go-jose/blob/main/go.mod

Avoided the direct dependency on square but the github.com/coreos/go-oidc still uses it indirectly.

pkg/util/oidcbuilder/jwks.go

pkg/util/oidcbuilder/oidcbuilder.go

rajdeepc2792 · 2024-06-03T19:18:04Z

/azp run e2e

azure-pipelines · 2024-06-03T19:18:13Z

Azure Pipelines successfully started running 1 pipeline(s).

bennerv · 2024-06-04T17:11:58Z

pkg/cluster/cluster.go

@@ -169,6 +176,11 @@ func New(ctx context.Context, log *logrus.Entry, _env env.Interface, db database
 		return nil, err
 	}

+	rpBlob, err := azblob.NewManager(_env, _env.SubscriptionID(), msiCredential)


why pass sub & _env if they're both part of env?

Changed it to rpBlob, err := azblob.NewManager(_env.Environment(), _env.SubscriptionID(), msiCredential)
Haven't passed _env directly as there can be a use case where different SubscriptionID is needed.

pkg/cluster/deploybaseresources.go

github-actions · 2024-06-06T12:57:18Z

Please rebase pull request.

…ytes

rajdeepc2792 · 2024-06-06T22:47:34Z

/azp run ci,e2e

azure-pipelines · 2024-06-06T22:47:48Z

Azure Pipelines successfully started running 2 pipeline(s).

kimorris27

Nice work! I just made some small suggestions here and there.

kimorris27 · 2024-06-07T20:36:17Z

pkg/cluster/deploybaseresources.go

+	blobContainerName := env.OIDCBlobContainerPrefix + m.doc.ID
+
+	publicAccess := azstorage.PublicAccessNone
+	// Public access on OIDC Container needed for development environments because of no AFD avaialbility


Super nitpicky: avaialbility -> availability

kimorris27 · 2024-06-07T20:44:29Z

pkg/util/azblob/manager.go

+		if !bloberror.HasCode(err, bloberror.ContainerNotFound) {
+			return err
+		}
+		needToCreateBlobContainer = true


I think you can simplify this by removing needToCreateBlobContainer and returning nil on this line.

kimorris27 · 2024-06-07T21:09:31Z

pkg/util/azureclient/mgmt/storage/blobcontainers.go

+	"github.com/Azure/ARO-RP/pkg/util/azureclient"
+)
+
+type BlobContainersClient interface {


Is this being used? I couldn't find any references to it. Could it be removed?

kimorris27 · 2024-06-07T21:17:37Z

pkg/cluster/deploybaseresources_test.go

@@ -1394,3 +1399,214 @@ func TestNewPublicLoadBalancer(t *testing.T) {
 		})
 	}
 }
+
+func TestCreateOIDC(t *testing.T) {


You could consider adding two more test cases:

Fail to create OIDCBuilder

OIDCBuilder fails to populate OIDC blob

kimorris27 · 2024-06-07T21:37:02Z

pkg/util/oidcbuilder/jwks.go

+		Bytes:   x509.MarshalPKCS1PrivateKey(privateKey),
+	})
+
+	// Generate public key from private keypair


I would reword this comment for clarity's sake. IIUC what's happening here is that the keypair has already been generated and you are simply putting the public key data into a byte array.

Suggested change

// Generate public key from private keypair

// Serialize public key into a byte array to prepare to store it in the OIDC storage blob

kimorris27 · 2024-06-07T21:37:56Z

pkg/util/oidcbuilder/jwks.go

+	// Generate public key from private keypair
+	pubKeyBytes, err := x509.MarshalPKIXPublicKey(&privateKey.PublicKey)
+	if err != nil {
+		return nil, nil, errors.Wrapf(err, "failed to generate public key from private")


Suggested change

return nil, nil, errors.Wrapf(err, "failed to generate public key from private")

return nil, nil, errors.Wrapf(err, "failed to serialize public key")

See my comment a few lines above this for the reasoning behind this suggestion.

kimorris27 · 2024-06-07T21:45:33Z

pkg/util/oidcbuilder/oidcbuilder_test.go

+// Copyright (c) Microsoft Corporation.
+// Licensed under the Apache License 2.0.
+
+func TestEnsureOIDCDocs(t *testing.T) {


You could consider adding a few more test cases here to cover some other ways in which the code could fail:

Public key is not of type RSA (the default case in the switch)

Failed to fetch key ID from public key

JSON encoding of web key set failed (if you can - I'm not sure if you can easily test this)

rajdeepc2792 added work-in-progress chainsaw Pull requests or issues owned by Team Chainsaw labels May 7, 2024

rajdeepc2792 force-pushed the rajdeepc2792/ARO-4373 branch 2 times, most recently from 58d9129 to 1b8076a Compare May 7, 2024 17:01

cadenmarchese reviewed May 7, 2024

View reviewed changes

pkg/util/storage/manager.go Outdated Show resolved Hide resolved

pkg/util/storage/manager.go Outdated Show resolved Hide resolved

github-actions bot added the needs-rebase branch needs a rebase label May 7, 2024

rajdeepc2792 force-pushed the rajdeepc2792/ARO-4373 branch from 1b8076a to 52b785b Compare May 8, 2024 18:34

github-actions bot removed the needs-rebase branch needs a rebase label May 8, 2024

rajdeepc2792 added the hold Hold label May 8, 2024

bennerv reviewed May 8, 2024

View reviewed changes

github-actions bot added the needs-rebase branch needs a rebase label May 13, 2024

rajdeepc2792 force-pushed the rajdeepc2792/ARO-4373 branch from 52b785b to feb76f9 Compare May 28, 2024 18:21

github-actions bot removed the needs-rebase branch needs a rebase label May 28, 2024

rajdeepc2792 force-pushed the rajdeepc2792/ARO-4373 branch from feb76f9 to c145d49 Compare May 28, 2024 18:35

cadenmarchese reviewed May 30, 2024

View reviewed changes

pkg/api/openshiftcluster.go Show resolved Hide resolved

pkg/cluster/install.go Show resolved Hide resolved

pkg/cluster/deploybaseresources.go Outdated Show resolved Hide resolved

rajdeepc2792 force-pushed the rajdeepc2792/ARO-4373 branch 4 times, most recently from d684aeb to a067322 Compare May 31, 2024 21:20

rajdeepc2792 marked this pull request as ready for review May 31, 2024 21:20

rajdeepc2792 requested review from jewzaam, hawkowl, rogbas and petrkotas as code owners May 31, 2024 21:20

rajdeepc2792 added ready-for-review and removed work-in-progress labels May 31, 2024

cadenmarchese mentioned this pull request Jun 3, 2024

Making Gateway and RP Managed Boot Diagnostics #3603

Open

niontive reviewed Jun 3, 2024

View reviewed changes

pkg/cluster/delete.go Outdated Show resolved Hide resolved

niontive reviewed Jun 3, 2024

View reviewed changes

pkg/util/oidcbuilder/jwks.go Show resolved Hide resolved

niontive reviewed Jun 3, 2024

View reviewed changes

pkg/util/oidcbuilder/oidcbuilder.go Outdated Show resolved Hide resolved

rajdeepc2792 self-assigned this Jun 4, 2024

bennerv reviewed Jun 4, 2024

View reviewed changes

github-actions bot added needs-rebase branch needs a rebase and removed ready-for-review labels Jun 6, 2024

rajdeepc2792 added 6 commits June 6, 2024 16:45

ARO-4373 set envs for storage account name and AFD endpoints

6f7ef42

ARO-4373 add the BoundServiceAccountSigningKey to ClusterProfile

f11c0a3

ARO-4373 generate keypair and oidc docs for miwi clusters

dcf54d1

vendor changes for ARO-4373

e4f37ec

ARO-4373 add unit test cases

b0e9907

ARO-4373 change BoundServiceAccountSigningKey SecureString to SecureB…

8da8feb

…ytes

rajdeepc2792 force-pushed the rajdeepc2792/ARO-4373 branch from a067322 to 5bffb1a Compare June 6, 2024 22:18

github-actions bot removed the needs-rebase branch needs a rebase label Jun 6, 2024

ARO-4373 resolve PR comments

9149938

rajdeepc2792 force-pushed the rajdeepc2792/ARO-4373 branch from 5bffb1a to 9149938 Compare June 6, 2024 22:36

kimorris27 reviewed Jun 7, 2024

View reviewed changes

	// Generate public key from private keypair
	// Serialize public key into a byte array to prepare to store it in the OIDC storage blob

	return nil, nil, errors.Wrapf(err, "failed to generate public key from private")
	return nil, nil, errors.Wrapf(err, "failed to serialize public key")

Create and populate OIDC blob store for the cluster #3564

Are you sure you want to change the base?

Create and populate OIDC blob store for the cluster #3564

Conversation

rajdeepc2792 commented May 7, 2024 • edited

Which issue this PR addresses:

What this PR does / why we need it:

Test plan for issue:

Is there any documentation that needs to be updated for this PR?

How do you know this will function as expected in production?

rajdeepc2792 commented May 7, 2024

azure-pipelines bot commented May 7, 2024

cadenmarchese left a comment

Choose a reason for hiding this comment

github-actions bot commented May 7, 2024

rajdeepc2792 commented May 8, 2024

azure-pipelines bot commented May 8, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented May 13, 2024

rajdeepc2792 commented May 31, 2024

azure-pipelines bot commented May 31, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rajdeepc2792 commented Jun 3, 2024

azure-pipelines bot commented Jun 3, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Jun 6, 2024

rajdeepc2792 commented Jun 6, 2024

azure-pipelines bot commented Jun 6, 2024

kimorris27 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rajdeepc2792 commented May 7, 2024 •

edited