Mixed VR and Volsync workloads fail on Relocate to a cluster that the workload was relocated or failed over from

[ShyamsundarR]

Test is to have a workload that uses one of each RBD and CephFS PVCs and to failover and then relocate such a workload back to the preferredCluster.

On initial Failover as VRG is not deleted for Volsync cases, the VR and PVC remain on the preferredCluster, with the PVC in Terminating state. Thus, on a future relocate to this cluster (or a failover for that matter), the ClusterDataReady is never reported as True, as the PVC is in Terminating state and the restore of the PVC from the s3 store fails.

This causes the action to be stuck and not make forward progress.

Thoughts on fixes:

• Handle VR deletion and PVC finalizer removal as part of VRG moving to Secondary, thus these stale resources are garbage collected as needed
• Delete a Secondary VRG and then once deleted recreate it for Volsync needs

The former is preferable as that allows VRG to shift between Primary and Seconday as the case maybe, rather than enforcing a VRG movement for VR as Primary->Secondary->Delete and then recreate.