envtest flake: VolSync_Handler Ensure PVC from ReplicationDestination When ReplicationDestination exists with snapshot latestImage When the latest image volume snapshot exists When pvc to be restored has already been created [It] ensure PVC should not fail

[ShyamsundarR]

The test fails in 2 ways, a GET failure or an UPDATE failure.

The issue seems to be a stale cache, or a stale object that is being updated with annotations.

Core reconciler would try again hence not a code issue per-se, whereas in the tests this is only tried once. Potential fix can leverage error string patterns (like this) to try again (here)to ensure this flake is not seen.

VolSync_Handler Ensure PVC from ReplicationDestination When ReplicationDestination exists with snapshot latestImage When the latest image volume snapshot exists When pvc to be restored has already been created [It] ensure PVC should not fail
.../ramen/controllers/volsync/vshandler_test.go:1329

  Timeline >>
  2024-05-03T11:57:16.382-0400	INFO	Latest Image for ReplicationDestination	{"latestImage": "testingsnap001"}
  2024-05-03T11:57:16.385-0400	DEBUG	VolumeSnapshot validated and protected	{"volumesnapshot name": "testingsnap001"}
  2024-05-03T11:57:16.385-0400	DEBUG	Updated	{"pvcName": "testpvc1", "snapshotRef": {"apiGroup":"snapshot.storage.k8s.io","kind":"VolumeSnapshot","name":"testingsnap001"}, "snapRestoreSize": "<nil>", "pvc": {"namespace": "vh-2jx9b", "name": "testpvc1"}}
  2024-05-03T11:57:16.388-0400	DEBUG	PVC createOrUpdate Complete	{"pvcName": "testpvc1", "snapshotRef": {"apiGroup":"snapshot.storage.k8s.io","kind":"VolumeSnapshot","name":"testingsnap001"}, "snapRestoreSize": "<nil>", "op": "created"}
  2024-05-03T11:57:16.391-0400	INFO	Latest Image for ReplicationDestination	{"latestImage": "testingsnap001"}
  2024-05-03T11:57:16.391-0400	DEBUG	VolumeSnapshot validated and protected	{"volumesnapshot name": "testingsnap001"}
  2024-05-03T11:57:16.391-0400	DEBUG	Updated	{"pvcName": "testpvc1", "snapshotRef": {"apiGroup":"snapshot.storage.k8s.io","kind":"VolumeSnapshot","name":"testingsnap001"}, "snapRestoreSize": "<nil>", "pvc": {"apiVersion": "v1", "kind": "PersistentVolumeClaim", "namespace": "vh-2jx9b", "name": "testpvc1"}}
  2024-05-03T11:57:16.391-0400	DEBUG	PVC createOrUpdate Complete	{"pvcName": "testpvc1", "snapshotRef": {"apiGroup":"snapshot.storage.k8s.io","kind":"VolumeSnapshot","name":"testingsnap001"}, "snapRestoreSize": "<nil>", "op": "unchanged"}
  2024-05-03T11:57:16.394-0400	DEBUG	Failed to Update PVC	{"error": "Operation cannot be fulfilled on persistentvolumeclaims \"testpvc1\": the object has been modified; please apply your changes to the latest version and try again"}
  [FAILED] in [It] - .../ramen/controllers/volsync/vshandler_test.go:1332 @ 05/03/24 11:57:16.394
  << Timeline

  [FAILED] Expected success, but got an error:
      <*errors.StatusError | 0xc00139e820>: 
      Operation cannot be fulfilled on persistentvolumeclaims "testpvc1": the object has been modified; please apply your changes to the latest version and try again
      {
          ErrStatus: {
              TypeMeta: {Kind: "", APIVersion: ""},
              ListMeta: {
                  SelfLink: "",
                  ResourceVersion: "",
                  Continue: "",
                  RemainingItemCount: nil,
              },
              Status: "Failure",
              Message: "Operation cannot be fulfilled on persistentvolumeclaims \"testpvc1\": the object has been modified; please apply your changes to the latest version and try again",
              Reason: "Conflict",
              Details: {
                  Name: "testpvc1",
                  Group: "",
                  Kind: "persistentvolumeclaims",
                  UID: "",
                  Causes: nil,
                  RetryAfterSeconds: 0,
              },
              Code: 409,
          },
      }
  In [It] at: .../ramen/controllers/volsync/vshandler_test.go:1332 @ 05/03/24 11:57:16.394
------------------------------
•••••••••••••

Summarizing 1 Failure:
  [FAIL] VolSync_Handler Ensure PVC from ReplicationDestination When ReplicationDestination exists with snapshot latestImage When the latest image volume snapshot exists When pvc to be restored has already been created [It] ensure PVC should not fail
  .../ramen/controllers/volsync/vshandler_test.go:1332

VolSync_Handler Ensure PVC from ReplicationDestination When ReplicationDestination exists with snapshot latestImage When the latest image volume snapshot exists [JustBeforeEach] When pvc to be restored has already been created but has incorrect datasource ensure PVC should delete the pvc with incorrect datasource and return err
  [JustBeforeEach] .../ramen/controllers/volsync/vshandler_test.go:1207
  [It] .../ramen/controllers/volsync/vshandler_test.go:1365

  Timeline >>
  2024-05-03T11:58:57.583-0400	INFO	Latest Image for ReplicationDestination	{"latestImage": "testingsnap001"}
  2024-05-03T11:58:57.585-0400	DEBUG	VolumeSnapshot validated and protected	{"volumesnapshot name": "testingsnap001"}
  2024-05-03T11:58:57.585-0400	DEBUG	Updated	{"pvcName": "testpvc1", "snapshotRef": {"apiGroup":"snapshot.storage.k8s.io","kind":"VolumeSnapshot","name":"testingsnap001"}, "snapRestoreSize": "<nil>", "pvc": {"namespace": "vh-247jt", "name": "testpvc1"}}
  2024-05-03T11:58:57.587-0400	DEBUG	PVC createOrUpdate Complete	{"pvcName": "testpvc1", "snapshotRef": {"apiGroup":"snapshot.storage.k8s.io","kind":"VolumeSnapshot","name":"testingsnap001"}, "snapRestoreSize": "<nil>", "op": "created"}
  2024-05-03T11:58:57.587-0400	DEBUG	Failed to GET PVC	{"error": "PersistentVolumeClaim \"testpvc1\" not found"}
  [FAILED] in [JustBeforeEach] - .../ramen/controllers/volsync/vshandler_test.go:1209 @ 05/03/24 11:58:57.587
  << Timeline

  [FAILED] Unexpected error:
      <*fmt.wrapError | 0xc001045960>: 
      PersistentVolumeClaim "testpvc1" not found
      {
          msg: "PersistentVolumeClaim \"testpvc1\" not found",
          err: <*errors.StatusError | 0xc0002037c0>{
              ErrStatus: {
                  TypeMeta: {Kind: "", APIVersion: ""},
                  ListMeta: {
                      SelfLink: "",
                      ResourceVersion: "",
                      Continue: "",
                      RemainingItemCount: nil,
                  },
                  Status: "Failure",
                  Message: "PersistentVolumeClaim \"testpvc1\" not found",
                  Reason: "NotFound",
                  Details: {
                      Name: "testpvc1",
                      Group: "",
                      Kind: "PersistentVolumeClaim",
                      UID: "",
                      Causes: nil,
                      RetryAfterSeconds: 0,
                  },
                  Code: 404,
              },
          },
      }
  occurred
  In [JustBeforeEach] at: .../ramen/controllers/volsync/vshandler_test.go:1209 @ 05/03/24 11:58:57.587
------------------------------
•••••••••••••••••••

Summarizing 1 Failure:
  [FAIL] VolSync_Handler Ensure PVC from ReplicationDestination When ReplicationDestination exists with snapshot latestImage When the latest image volume snapshot exists [JustBeforeEach] When pvc to be restored has already been created but has incorrect datasource ensure PVC should delete the pvc with incorrect datasource and return err
  .../ramen/controllers/volsync/vshandler_test.go:1209