Managed Infrastructure Maintenance Operator - Milestone 1 #3571

hawkowl · 2024-05-10T08:09:47Z

Which issue this PR addresses:

Part of https://issues.redhat.com/browse/ARO-4895.

What this PR does / why we need it:

This PR is the initial feature branch for the MIMO M1 milestone.

Q&A

What should I use this PR for?

This PR is for people to understand the general state of MIMO, how it works, and how they can implement tasks in it. It is not meant to merge as-is.

Where should I start looking?

pkg/mimo/tasks/example/task.go if you want to see how the developer experience is. pkg/mimo/actuator/manager.go for the service logic.

Looks great, when can I have it?

See https://issues.redhat.com/browse/ARO-4895 for all the things we need before we can roll this out. Namely, we need to do some porting of the infrastructure away from being solely on the VMSSes before we can roll this out.

What is final, and what is subject to change?

The runtime logic in service.go is very subject to change. Right now, it doesn't use changefeeds for a complicated reason (because I want to see if we could get away with using the official CosmosDB SDK as a test...) and I tried to make it so that it would handle a lot of clusters if it had to, but not get ahead of myself with the scaling of the actuator. (There's several approaches we could take -- this is more like the Monitor, but we could use the backend of OpenShiftClusters style + dequeueing. However, since we don't want two tasks for the same cluster worked on at the same time, the same dequeue style doesn't seem very efficient. In the interim, I'm approaching it from the angle of the monitor where there's the 255 or so buckets, but foregoing the auto bucket balancing and instead just statically distributing them based on something like a StatefulSet instance ID, relying on the buckets to be relatively statistically spread out. I'm not sure if that's even a good idea, but I can worry about it after vacation, and people can chime in with ideas.)

The CLI uses urfave/cli/v2 instead of Cobra because of the better support for environment vars as selectively-required arguments than cobra+viper. This might be worth standardising on, or moving to cobra and dealing with a bit more manual boilerplate.

The idea of the TaskFunc and TaskContext is relatively final, although the arguments/params are not. The idea of the separation is seen in pkg/mimo/tasks/example/task_test.go where one can ignore the rest of MIMO and just focus on the task implementation and test environment. This, hopefully, makes the business logic fairly portable in case of some other better solution in the future.

This PR is very big!

Yes, since it consists of not only the database, CLI, task, and other machinery for MIMO, but likely other improvements/code cleanups/etc. Anything that is not part of the new functionality (e.g. refactors) are planned to be split out as and where they make sense to do so.

Wow your commit history sucks

git rebase will end up putting those skeletons in the closet, don't worry :)

Test plan for issue:

Some unit tests are included. Proper QA will need to occur.

Is there any documentation that needs to be updated for this PR?

Yes, see https://issues.redhat.com/browse/ARO-4895 .

How do you know this will function as expected in production?

Telemetry, monitoring, and documentation will need to be fleshed out. See https://issues.redhat.com/browse/ARO-4895 for details.

github-actions · 2024-05-15T16:33:33Z

Please rebase pull request.

jaitaiwan

I've started a review, and reached my ingestion limit. I'll keep reviewing later.

jaitaiwan · 2024-05-23T00:25:48Z

go.mod

@@ -228,6 +231,7 @@ require (
 	github.com/robfig/cron v1.2.0 // indirect
 	github.com/rogpeppe/go-internal v1.12.0 // indirect
 	github.com/russross/blackfriday v1.6.0 // indirect
+	github.com/russross/blackfriday/v2 v2.1.0 // indirect


Not sure why we need a markdown library in our go.mod?

jaitaiwan · 2024-05-23T00:31:18Z

pkg/api/mimodocument.go

+// Licensed under the Apache License 2.0.
+
+type MaintenanceManifestDocuments struct {
+	Count                        int                            `json:"_count,omitempty"`


Why do we need a Count? Is that not the same as len(MaintenanceManifestDocuments)?

it's returned by the cosmosdb API

jaitaiwan · 2024-05-23T00:37:55Z

pkg/database/mimo.go

+)
+
+const (
+	MaintenanceManifestQueryForCluster  = `SELECT * FROM MaintenanceManifests doc WHERE doc.maintenanceManifest.state IN ("Pending") AND doc.clusterID = @clusterID`


Is select * any less performant than select <fields>?

since it's json, probably not

jaitaiwan · 2024-05-23T00:40:42Z

pkg/database/mimo.go

+
+type MaintenanceManifests interface {
+	Create(context.Context, *api.MaintenanceManifestDocument) (*api.MaintenanceManifestDocument, error)
+	GetByClusterID(context.Context, string, string) (cosmosdb.MaintenanceManifestDocumentIterator, error)


Based on the fact that the manifests also include a ResourceId I think we should also have a get based on that.

jaitaiwan · 2024-05-23T00:48:56Z

pkg/database/mimo.go

+	PatchWithLease(context.Context, string, string, MaintenanceManifestDocumentMutator) (*api.MaintenanceManifestDocument, error)
+	Lease(context.Context, string, string) (*api.MaintenanceManifestDocument, error)
+	EndLease(context.Context, string, string, api.MaintenanceManifestState, *string) (*api.MaintenanceManifestDocument, error)
+	Dequeue(ctx context.Context, clusterID string, id string) (*api.MaintenanceManifestDocument, error)


Is this the equivalent of cancelling a MaintenanceSet?

dequeue is part of the logic of claiming it as being worked on -- see the openshiftcluster one for where this is from

jaitaiwan · 2024-05-23T01:05:20Z

pkg/database/mimo.go

+	if err, ok := err.(*cosmosdb.Error); ok && err.StatusCode == http.StatusConflict {
+		err.StatusCode = http.StatusPreconditionFailed
+	}


Why are we overwriting the http status condition?

It looks like to me it's because of line 143. We're saying that in case of a conflict we want to change it to a status that will have the cosmosdb Retry function retry the request. If this is the case I think commenting this would be helpful in-case the functionality of functions that use the cosmosdb Retry function change in the future.

jaitaiwan · 2024-05-23T01:09:03Z

pkg/database/mimo.go

+	return c.c.Get(ctx, clusterID, id, nil)
+}
+
+// QueueLength returns maintenanceManifests un-queued document count.


Comment is a little confusing. I think we need to work a little more on definitional language here.
e.g.:

Scheduled

Queued

Pending

etc
And what these statuses might all mean in practice.

It seems to me like QueueLength is returning the list of MaintenanceSets that are pending delivery to the actuator?

jaitaiwan · 2024-05-23T01:23:24Z

pkg/database/mimo.go

+func (c *maintenanceManifests) Patch(ctx context.Context, clusterID string, id string, f MaintenanceManifestDocumentMutator) (*api.MaintenanceManifestDocument, error) {
+	var doc *api.MaintenanceManifestDocument
+
+	err := cosmosdb.RetryOnPreconditionFailed(func() (err error) {
+		doc, err = c.Get(ctx, clusterID, id)
+		if err != nil {
+			return
+		}
+
+		err = f(doc)
+		if err != nil {
+			return
+		}
+
+		doc, err = c.c.Replace(ctx, doc.ClusterID, doc, nil)
+		return
+	})
+
+	return doc, err
+}
+
+func (c *maintenanceManifests) patch(ctx context.Context, clusterID string, id string, f MaintenanceManifestDocumentMutator, options *cosmosdb.Options) (*api.MaintenanceManifestDocument, error) {
+	var doc *api.MaintenanceManifestDocument
+
+	err := cosmosdb.RetryOnPreconditionFailed(func() (err error) {
+		doc, err = c.Get(ctx, clusterID, id)
+		if err != nil {
+			return
+		}
+
+		err = f(doc)
+		if err != nil {
+			return
+		}
+
+		doc, err = c.update(ctx, doc, options)
+		return
+	})
+
+	return doc, err
+}


Functionally these two functions are almost identical. What's the purpose of c.update vs c.c.Replace?

It looks like c.update calls c.c.Replace appears that this is pretty much duplicate functions

Okay so only difference is the ability to pass-in cosmosdb functions. I think these two should be mostly combined.

I think I've managed to deduplicate this now

jaitaiwan · 2024-05-23T01:26:57Z

pkg/database/mimo.go

+	return c.c.ChangeFeed(nil)
+}
+
+func (c *maintenanceManifests) GetByClusterID(ctx context.Context, clusterID string, continuation string) (cosmosdb.MaintenanceManifestDocumentIterator, error) {


Can the type of continuation be defined as not a string but some other type for the purposes of understanding what values are acceptable?

I think it's an opaque string, since it's part of the cosmosdb API

jaitaiwan · 2024-05-23T01:30:19Z

pkg/database/mimo.go

+	PatchWithLease(context.Context, string, string, MaintenanceManifestDocumentMutator) (*api.MaintenanceManifestDocument, error)
+	Lease(context.Context, string, string) (*api.MaintenanceManifestDocument, error)


It's not clear from the implementation what the purpose of Lease and PatchWithLease are supposed to be

I've cleaned these up a bit and added comments

hawkowl · 2024-06-06T02:47:12Z

/azp run ci, e2e

azure-pipelines · 2024-06-06T02:47:26Z

Azure Pipelines successfully started running 2 pipeline(s).

github-actions · 2024-06-06T14:18:04Z

Please rebase pull request.

This was referenced May 10, 2024

[turbo WIP] MIMO PoC #3210

Closed

MIMO M1 (Database) #3560

Closed

github-actions bot added the needs-rebase branch needs a rebase label May 15, 2024

jaitaiwan reviewed May 23, 2024

View reviewed changes

github-actions bot removed the needs-rebase branch needs a rebase label May 27, 2024

hawkowl added 22 commits June 6, 2024 11:53

add base MIMO documents

94b76e9

generated

debbc8f

add MIMO to dev

c49344a

go generate

e964a64

clean

38ab5cb

mimo CLI skeleton

deaa5d3

mimo changes

5453c68

temp

fb2d371

some db sorting out

12cf0a7

database creation logic

5da3887

move some boilerplate out

f27293b

fix

6d9f428

vendor

d0a8990

fix

c642f64

revert

2181589

fix

a68a006

update

74ad463

tests

2b4f7ce

cleanups to actuator

9b23702

add an example test

700f64d

go mod update

0343e67

some database cleanups

335c6fd

hawkowl force-pushed the hawkowl/mimo-m1 branch from 6d97470 to 335c6fd Compare June 6, 2024 02:15

github-actions bot added the needs-rebase branch needs a rebase label Jun 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Managed Infrastructure Maintenance Operator - Milestone 1 #3571

Managed Infrastructure Maintenance Operator - Milestone 1 #3571

hawkowl commented May 10, 2024

github-actions bot commented May 15, 2024

jaitaiwan left a comment

jaitaiwan May 23, 2024

jaitaiwan May 23, 2024

hawkowl Jun 6, 2024

jaitaiwan May 23, 2024

hawkowl Jun 6, 2024

jaitaiwan May 23, 2024

jaitaiwan May 23, 2024

hawkowl Jun 6, 2024

jaitaiwan May 23, 2024

jaitaiwan May 23, 2024

jaitaiwan May 23, 2024

jaitaiwan May 23, 2024

jaitaiwan May 23, 2024

jaitaiwan May 23, 2024

hawkowl Jun 6, 2024

jaitaiwan May 23, 2024

hawkowl Jun 6, 2024

jaitaiwan May 23, 2024

hawkowl Jun 6, 2024

hawkowl commented Jun 6, 2024

azure-pipelines bot commented Jun 6, 2024

github-actions bot commented Jun 6, 2024

		PatchWithLease(context.Context, string, string, MaintenanceManifestDocumentMutator) (*api.MaintenanceManifestDocument, error)
		Lease(context.Context, string, string) (*api.MaintenanceManifestDocument, error)

Managed Infrastructure Maintenance Operator - Milestone 1 #3571

Are you sure you want to change the base?

Managed Infrastructure Maintenance Operator - Milestone 1 #3571

Conversation

hawkowl commented May 10, 2024

Which issue this PR addresses:

What this PR does / why we need it:

Q&A

What should I use this PR for?

Where should I start looking?

Looks great, when can I have it?

What is final, and what is subject to change?

This PR is very big!

Wow your commit history sucks

Test plan for issue:

Is there any documentation that needs to be updated for this PR?

How do you know this will function as expected in production?

github-actions bot commented May 15, 2024

jaitaiwan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hawkowl commented Jun 6, 2024

azure-pipelines bot commented Jun 6, 2024

github-actions bot commented Jun 6, 2024