fix: if artifact GC Pod fails make sure error is propagated as a Condition #10019
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ition Signed-off-by: Julie Vogelman <julie_vogelman@intuit.com>
Signed-off-by: Julie Vogelman <julie_vogelman@intuit.com>
Signed-off-by: Julie Vogelman <julie_vogelman@intuit.com>
…wArtifactGCTasks Signed-off-by: Julie Vogelman <julie_vogelman@intuit.com>
juliev0
commented
Nov 13, 2022
| if !found { | ||
| return fmt.Errorf("Artifact GC Pod '%s' doesn't have annotation '%s'?", pod.Name, common.AnnotationKeyArtifactGCStrategy) | ||
| } | ||
| strategy := wfv1.ArtifactGCStrategy(strategyStr) |
There was a problem hiding this comment.
note this code is just moved up from below
juliev0
commented
Nov 13, 2022
| err := woc.controller.wfclientset.ArgoprojV1alpha1().WorkflowArtifactGCTasks(woc.wf.Namespace).Delete(ctx, task.Name, metav1.DeleteOptions{}) | ||
| if err != nil { | ||
| woc.log.Errorf("error deleting WorkflowArtifactGCTask: %s: %v", task.Name, err) | ||
| } |
There was a problem hiding this comment.
this deletion code (which deletes both the WorkflowArtifactGCTask, as well as the Pod which it owns), used to just be in processCompletedWorkflowArtifactGCTask() but really this logic is dependent on both the Pod succeeding and the artifacts succeeding
workflow/controller/artifact_gc.go
Outdated
| woc.log.Debugf("processing WorkflowArtifactGCTask %s", artifactGCTask.Name) | ||
|
|
||
| foundGCFailure := false | ||
| for nodeName, nodeResult := range artifactGCTask.Status.ArtifactResultsByNode { | ||
| // find this node result in the Workflow Status | ||
| wfNode, found := woc.wf.Status.Nodes[nodeName] | ||
| if !found { | ||
| return fmt.Errorf("node named '%s' returned by WorkflowArtifactGCTask '%s' wasn't found in Workflow '%s' Status", nodeName, artifactGCTask.Name, woc.wf.Name) | ||
| return false, fmt.Errorf("node named '%s' returned by WorkflowArtifactGCTask '%s' wasn't found in Workflow '%s' Status", nodeName, artifactGCTask.Name, woc.wf.Name) |
There was a problem hiding this comment.
had to look that up. I guess it gives you a quoted string - good idea
There was a problem hiding this comment.
okay, just did a find and replace for all instances of '%s'
Signed-off-by: Julie Vogelman <julie_vogelman@intuit.com>
Signed-off-by: Julie Vogelman <julie_vogelman@intuit.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Partial fix to #9953
Issue was that we were only adding the Condition based on failure of individual artifacts. In this case, the GC Pod never got to the point of attempting deletion of an artifact (it couldn't read what the artifacts were in fact), so there was no reporting of individual artifacts' status. Now a Condition will be written if the Pod fails as well.
Signed-off-by: Julie Vogelman julie_vogelman@intuit.com
Fixes #TODO
Please do not open a pull request until you have checked ALL of these:
make pre-commit -Bto fix codegen and lint problems.If changes were requested, and you've made them, dismiss the review to get it reviewed again.