self reporting progress does not work because WorkflowTaskResult doesn't trigger workflow reconciliation

[mweibel]

Pre-requisites

• I have double-checked my configuration
• I can confirm the issues exists when I tested with :latest
• I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

when a workflow is using self reporting progress it writes in intervals to ARGO_PROGRESS_FILE. Since #8000 this creates a WorkflowTaskResult instead of doing a pod patch. #8135 changed to use an own reconciliation loop for WorkflowTaskResult. This does not trigger a reconcile of the workflow anymore.

After #8135, the only time a workflow progress gets updated is when the workflow switches into a completed state, or every 20m (since workflows are automatically reconciled every 20m anyway).

Version

latest

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: progress-
spec:
  entrypoint: main
  templates:
    - name: main
      dag:
        tasks:
          - name: progress
            template: progress
    - name: progress
      container:
        image: alpine:3.14
        command: [ "/bin/sh", "-c" ]
        args:
          - |
            for i in `seq 1 100`; do sleep 10; echo "$(($i*10))"'/1000' > $ARGO_PROGRESS_FILE; done

Logs from the workflow controller

time="2022-11-22T14:26:04.973Z" level=info msg="Processing workflow" namespace=render-test workflow=progress-wxg8s
time="2022-11-22T14:26:04.973Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2022-11-22T14:26:04.973Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2022-11-22T14:26:04.973Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2022-11-22T14:26:04.973Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.DAGTask (progress)"
time="2022-11-22T14:26:04.973Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.DAGTask (progress)"
time="2022-11-22T14:26:04.973Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.DAGTask (progress)"
time="2022-11-22T14:26:04.973Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.DAGTask (progress)"
time="2022-11-22T14:26:04.973Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.DAGTask (progress)"
time="2022-11-22T14:26:04.973Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.DAGTask (progress)"
time="2022-11-22T14:26:04.973Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.DAGTask (progress)"
time="2022-11-22T14:26:04.973Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2022-11-22T14:26:04.973Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2022-11-22T14:26:04.973Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2022-11-22T14:26:04.973Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2022-11-22T14:26:04.973Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2022-11-22T14:26:04.973Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.WorkflowStep (progress)"
time="2022-11-22T14:26:04.973Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.WorkflowStep (progress)"
time="2022-11-22T14:26:04.973Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.WorkflowStep (progress)"
time="2022-11-22T14:26:04.976Z" level=info msg="Updated phase  -> Running" namespace=render-test workflow=progress-wxg8s
time="2022-11-22T14:26:04.976Z" level=debug msg="Evaluating node progress-wxg8s: template: *v1alpha1.WorkflowStep (main), boundaryID: " namespace=render-test workflow=progress-wxg8s
time="2022-11-22T14:26:04.976Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2022-11-22T14:26:04.976Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2022-11-22T14:26:04.976Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2022-11-22T14:26:04.976Z" level=debug msg="Initializing node progress-wxg8s: template: *v1alpha1.WorkflowStep (main), boundaryID: " namespace=render-test workflow=progress-wxg8s
time="2022-11-22T14:26:04.976Z" level=info msg="DAG node progress-wxg8s initialized Running" namespace=render-test workflow=progress-wxg8s
time="2022-11-22T14:26:04.976Z" level=info msg="All of node progress-wxg8s.progress dependencies [] completed" namespace=render-test workflow=progress-wxg8s
time="2022-11-22T14:26:04.977Z" level=debug msg="Evaluating node progress-wxg8s.progress: template: *v1alpha1.DAGTask (progress), boundaryID: progress-wxg8s" namespace=render-test workflow=progress-wxg8s
time="2022-11-22T14:26:04.977Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.DAGTask (progress)"
time="2022-11-22T14:26:04.977Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.DAGTask (progress)"
time="2022-11-22T14:26:04.977Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.DAGTask (progress)"
time="2022-11-22T14:26:04.977Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"render-test\", Name:\"progress-wxg8s\", UID:\"cbbc7593-b2be-482e-b752-30a7bfa91a1f\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"15552\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowRunning' Workflow Running"
time="2022-11-22T14:26:04.977Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.NodeStatus (main)"
time="2022-11-22T14:26:04.977Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.NodeStatus (main)"
time="2022-11-22T14:26:04.977Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.NodeStatus (main)"
time="2022-11-22T14:26:04.977Z" level=debug msg="Initializing node progress-wxg8s.progress: template: *v1alpha1.DAGTask (progress), boundaryID: progress-wxg8s" namespace=render-test workflow=progress-wxg8s
time="2022-11-22T14:26:04.977Z" level=info msg="Pod node progress-wxg8s-730772787 initialized Pending" namespace=render-test workflow=progress-wxg8s
time="2022-11-22T14:26:04.977Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.NodeStatus (main)"
time="2022-11-22T14:26:04.977Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.NodeStatus (main)"
time="2022-11-22T14:26:04.977Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.NodeStatus (main)"
time="2022-11-22T14:26:04.977Z" level=debug msg="Executing node progress-wxg8s.progress with container template: progress\n" namespace=render-test workflow=progress-wxg8s
time="2022-11-22T14:26:04.977Z" level=debug namespace=render-test needLocation=false workflow=progress-wxg8s
time="2022-11-22T14:26:04.978Z" level=debug msg="Creating Pod: progress-wxg8s.progress (progress-wxg8s-progress-730772787)" namespace=render-test workflow=progress-wxg8s
time="2022-11-22T14:26:04.985Z" level=info msg="Created pod: progress-wxg8s.progress (progress-wxg8s-progress-730772787)" namespace=render-test workflow=progress-wxg8s
time="2022-11-22T14:26:04.985Z" level=info msg="TaskSet Reconciliation" namespace=render-test workflow=progress-wxg8s
time="2022-11-22T14:26:04.985Z" level=info msg=reconcileAgentPod namespace=render-test workflow=progress-wxg8s
time="2022-11-22T14:26:04.985Z" level=debug msg="Log changes patch: {\"metadata\":{\"annotations\":{\"workflows.argoproj.io/pod-name-format\":\"v2\"},\"labels\":{\"workflows.argoproj.io/phase\":\"Running\"}},\"status\":{\"artifactGCStatus\":{\"notSpecified\":true},\"artifactRepositoryRef\":{\"artifactRepository\":{},\"default\":true},\"nodes\":{\"progress-wxg8s\":{\"children\":[\"progress-wxg8s-730772787\"],\"displayName\":\"progress-wxg8s\",\"finishedAt\":null,\"id\":\"progress-wxg8s\",\"name\":\"progress-wxg8s\",\"phase\":\"Running\",\"startedAt\":\"2022-11-22T14:26:04Z\",\"templateName\":\"main\",\"templateScope\":\"local/progress-wxg8s\",\"type\":\"DAG\"},\"progress-wxg8s-730772787\":{\"boundaryID\":\"progress-wxg8s\",\"displayName\":\"progress\",\"finishedAt\":null,\"id\":\"progress-wxg8s-730772787\",\"name\":\"progress-wxg8s.progress\",\"phase\":\"Pending\",\"startedAt\":\"2022-11-22T14:26:04Z\",\"templateName\":\"progress\",\"templateScope\":\"local/progress-wxg8s\",\"type\":\"Pod\"}},\"phase\":\"Running\",\"startedAt\":\"2022-11-22T14:26:04Z\"}}"
time="2022-11-22T14:26:04.989Z" level=info msg="Workflow update successful" namespace=render-test phase=Running resourceVersion=15555 workflow=progress-wxg8s
time="2022-11-22T14:26:04.989Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"render-test\", Name:\"progress-wxg8s\", UID:\"cbbc7593-b2be-482e-b752-30a7bfa91a1f\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"15555\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowNodeRunning' Running node progress-wxg8s"
time="2022-11-22T14:26:14.987Z" level=info msg="Processing workflow" namespace=render-test workflow=progress-wxg8s
time="2022-11-22T14:26:14.987Z" level=info msg="Task-result reconciliation" namespace=render-test numObjs=0 workflow=progress-wxg8s
time="2022-11-22T14:26:14.987Z" level=info msg="node changed" namespace=render-test new.message= new.phase=Running new.progress=0/1 nodeID=progress-wxg8s-730772787 old.message= old.phase=Pending old.progress=0/1 workflow=progress-wxg8s
time="2022-11-22T14:26:14.988Z" level=debug msg="Evaluating node progress-wxg8s: template: *v1alpha1.WorkflowStep (main), boundaryID: " namespace=render-test workflow=progress-wxg8s
time="2022-11-22T14:26:14.988Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2022-11-22T14:26:14.988Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2022-11-22T14:26:14.988Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2022-11-22T14:26:14.988Z" level=debug msg="Executing node progress-wxg8s of DAG is Running" namespace=render-test workflow=progress-wxg8s
time="2022-11-22T14:26:14.989Z" level=debug msg="Evaluating node progress-wxg8s.progress: template: *v1alpha1.DAGTask (progress), boundaryID: progress-wxg8s" namespace=render-test workflow=progress-wxg8s
time="2022-11-22T14:26:14.989Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.DAGTask (progress)"
time="2022-11-22T14:26:14.989Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.DAGTask (progress)"
time="2022-11-22T14:26:14.989Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.DAGTask (progress)"
time="2022-11-22T14:26:14.989Z" level=debug msg="Executing node progress-wxg8s.progress of Pod is Running" namespace=render-test workflow=progress-wxg8s
time="2022-11-22T14:26:14.990Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.NodeStatus (main)"
time="2022-11-22T14:26:14.990Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.NodeStatus (main)"
time="2022-11-22T14:26:14.990Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.NodeStatus (main)"
time="2022-11-22T14:26:14.990Z" level=debug msg="Executing node progress-wxg8s.progress with container template: progress\n" namespace=render-test workflow=progress-wxg8s
time="2022-11-22T14:26:14.990Z" level=debug msg="Skipped pod progress-wxg8s.progress (progress-wxg8s-730772787) creation: already exists" namespace=render-test podPhase=Running workflow=progress-wxg8s
time="2022-11-22T14:26:14.990Z" level=info msg="TaskSet Reconciliation" namespace=render-test workflow=progress-wxg8s
time="2022-11-22T14:26:14.990Z" level=info msg=reconcileAgentPod namespace=render-test workflow=progress-wxg8s
time="2022-11-22T14:26:14.992Z" level=debug msg="Log changes patch: {\"status\":{\"conditions\":[{\"status\":\"True\",\"type\":\"PodRunning\"}],\"nodes\":{\"progress-wxg8s-730772787\":{\"hostNodeName\":\"core-worker\",\"phase\":\"Running\"}}}}"
time="2022-11-22T14:26:15.017Z" level=info msg="Workflow update successful" namespace=render-test phase=Running resourceVersion=15596 workflow=progress-wxg8s
time="2022-11-22T14:26:15.017Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"render-test\", Name:\"progress-wxg8s\", UID:\"cbbc7593-b2be-482e-b752-30a7bfa91a1f\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"15596\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowNodeRunning' Running node progress-wxg8s.progress"
time="2022-11-22T14:42:57.575Z" level=info msg="Processing workflow" namespace=render-test workflow=progress-wxg8s
time="2022-11-22T14:42:57.576Z" level=info msg="Task-result reconciliation" namespace=render-test numObjs=1 workflow=progress-wxg8s
time="2022-11-22T14:42:57.576Z" level=info msg="task-result changed" namespace=render-test nodeID=progress-wxg8s-730772787 workflow=progress-wxg8s
time="2022-11-22T14:42:57.576Z" level=info msg="node changed" namespace=render-test new.message= new.phase=Succeeded new.progress=950/1000 nodeID=progress-wxg8s-730772787 old.message= old.phase=Running old.progress=950/1000 workflow=progress-wxg8s
time="2022-11-22T14:42:57.576Z" level=debug msg="Evaluating node progress-wxg8s: template: *v1alpha1.WorkflowStep (main), boundaryID: " namespace=render-test workflow=progress-wxg8s
time="2022-11-22T14:42:57.576Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2022-11-22T14:42:57.576Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2022-11-22T14:42:57.576Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.WorkflowStep (main)"
time="2022-11-22T14:42:57.577Z" level=debug msg="Executing node progress-wxg8s of DAG is Running" namespace=render-test workflow=progress-wxg8s
time="2022-11-22T14:42:57.577Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.DAGTask (progress)"
time="2022-11-22T14:42:57.577Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.DAGTask (progress)"
time="2022-11-22T14:42:57.577Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.DAGTask (progress)"
time="2022-11-22T14:42:57.578Z" level=debug msg="Resolving the template" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.NodeStatus (progress)"
time="2022-11-22T14:42:57.578Z" level=debug msg="Getting the template" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.NodeStatus (progress)"
time="2022-11-22T14:42:57.578Z" level=debug msg="Getting the template by name" base="*v1alpha1.Workflow (namespace=render-test,name=progress-wxg8s)" depth=0 tmpl="*v1alpha1.NodeStatus (progress)"
time="2022-11-22T14:42:57.578Z" level=info msg="Outbound nodes of progress-wxg8s set to [progress-wxg8s-730772787]" namespace=render-test workflow=progress-wxg8s
time="2022-11-22T14:42:57.578Z" level=info msg="node progress-wxg8s phase Running -> Succeeded" namespace=render-test workflow=progress-wxg8s
time="2022-11-22T14:42:57.578Z" level=info msg="node progress-wxg8s finished: 2022-11-22 14:42:57.578412295 +0000 UTC" namespace=render-test workflow=progress-wxg8s
time="2022-11-22T14:42:57.578Z" level=info msg="Checking daemoned children of progress-wxg8s" namespace=render-test workflow=progress-wxg8s
time="2022-11-22T14:42:57.578Z" level=info msg="TaskSet Reconciliation" namespace=render-test workflow=progress-wxg8s
time="2022-11-22T14:42:57.578Z" level=info msg=reconcileAgentPod namespace=render-test workflow=progress-wxg8s
time="2022-11-22T14:42:57.578Z" level=info msg="Updated phase Running -> Succeeded" namespace=render-test workflow=progress-wxg8s
time="2022-11-22T14:42:57.578Z" level=info msg="Marking workflow completed" namespace=render-test workflow=progress-wxg8s
time="2022-11-22T14:42:57.579Z" level=info msg="Checking daemoned children of " namespace=render-test workflow=progress-wxg8s
time="2022-11-22T14:42:57.579Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"render-test\", Name:\"progress-wxg8s\", UID:\"cbbc7593-b2be-482e-b752-30a7bfa91a1f\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"15596\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowSucceeded' Workflow completed"
time="2022-11-22T14:42:57.581Z" level=debug msg="Log changes patch: {\"metadata\":{\"labels\":{\"workflows.argoproj.io/completed\":\"true\",\"workflows.argoproj.io/phase\":\"Succeeded\"}},\"status\":{\"conditions\":[{\"status\":\"False\",\"type\":\"PodRunning\"},{\"status\":\"True\",\"type\":\"Completed\"}],\"finishedAt\":\"2022-11-22T14:42:57Z\",\"nodes\":{\"progress-wxg8s\":{\"finishedAt\":\"2022-11-22T14:42:57Z\",\"outboundNodes\":[\"progress-wxg8s-730772787\"],\"phase\":\"Succeeded\"},\"progress-wxg8s-730772787\":{\"finishedAt\":\"2022-11-22T14:42:48Z\",\"outputs\":{\"exitCode\":\"0\"},\"phase\":\"Succeeded\",\"progress\":\"950/1000\",\"resourcesDuration\":{\"cpu\":2003,\"memory\":2003}}},\"phase\":\"Succeeded\"}}"
time="2022-11-22T14:42:57.584Z" level=info msg="cleaning up pod" action=deletePod key=render-test/progress-wxg8s-1340600742-agent/deletePod
time="2022-11-22T14:42:57.596Z" level=info msg="Workflow update successful" namespace=render-test phase=Succeeded resourceVersion=18133 workflow=progress-wxg8s
time="2022-11-22T14:42:57.596Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"render-test\", Name:\"progress-wxg8s\", UID:\"cbbc7593-b2be-482e-b752-30a7bfa91a1f\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"18133\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowNodeSucceeded' Succeeded node progress-wxg8s"
time="2022-11-22T14:42:57.596Z" level=debug msg="Event(v1.ObjectReference{Kind:\"Workflow\", Namespace:\"render-test\", Name:\"progress-wxg8s\", UID:\"cbbc7593-b2be-482e-b752-30a7bfa91a1f\", APIVersion:\"argoproj.io/v1alpha1\", ResourceVersion:\"18133\", FieldPath:\"\"}): type: 'Normal' reason: 'WorkflowNodeSucceeded' Succeeded node progress-wxg8s.progress"
time="2022-11-22T14:42:57.612Z" level=info msg="cleaning up pod" action=labelPodCompleted key=render-test/progress-wxg8s-progress-730772787/labelPodCompleted
time="2022-11-22T14:43:07.623Z" level=debug msg="Won't process Workflow since it's completed" key=render-test/progress-wxg8s

Logs from in your workflow's wait container

progress-wxg8s-progress-730772787 wait time="2022-11-22T14:26:06.596Z" level=info msg="Starting Workflow Executor" version=v3.4.3
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:26:06.598Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:26:06.598Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=render-test podName=progress-wxg8s-progress-730772787 template="{\"name\":\"progress\",\"inputs\":{},\"outputs\":{},\"metadata\":{},\"container\":{\"name\":\"\",\"image\":\"alpine:3.14\",\"command\":[\"/bin/sh\",\"-c\"],\"args\":[\"for i in `seq 1 100`; do sleep 10; echo \\\"$(($i*10))\\\"'/1000' \\u003e $ARGO_PROGRESS_FILE; done\\n\"],\"resources\":{}}}" version="&Version{Version:v3.4.3,BuildDate:2022-10-31T05:40:15Z,GitCommit:eddb1b78407adc72c08b4ed6be8f52f2a1f1316a,GitTag:v3.4.3,GitTreeState:clean,GoVersion:go1.18.7,Compiler:gc,Platform:linux/amd64,}"
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:26:06.599Z" level=info msg="Starting deadline monitor"
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:26:18.599Z" level=info progress=10/1000
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:26:27.599Z" level=info progress=20/1000
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:26:39.599Z" level=info progress=30/1000
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:26:48.599Z" level=info progress=40/1000
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:26:57.600Z" level=info progress=50/1000
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:27:06.635Z" level=info msg="Create workflowtaskresults 201"
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:27:09.599Z" level=info progress=60/1000
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:27:18.599Z" level=info progress=70/1000
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:27:27.599Z" level=info progress=80/1000
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:27:39.599Z" level=info progress=90/1000
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:27:48.600Z" level=info progress=100/1000
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:27:57.599Z" level=info progress=110/1000
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:28:06.626Z" level=info msg="Create workflowtaskresults 409"
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:28:06.637Z" level=info msg="Patch workflowtaskresults 200"
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:28:09.599Z" level=info progress=120/1000
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:28:18.599Z" level=info progress=130/1000
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:28:27.599Z" level=info progress=140/1000
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:28:39.599Z" level=info progress=150/1000
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:28:48.599Z" level=info progress=160/1000
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:28:57.600Z" level=info progress=170/1000
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:29:06.627Z" level=info msg="Create workflowtaskresults 409"
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:29:06.633Z" level=info msg="Patch workflowtaskresults 200"
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:29:09.599Z" level=info progress=180/1000
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:29:18.600Z" level=info progress=190/1000
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:29:27.599Z" level=info progress=200/1000
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:29:39.599Z" level=info progress=210/1000
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:29:48.600Z" level=info progress=220/1000
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:29:57.599Z" level=info progress=230/1000
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:30:06.606Z" level=info msg="Create workflowtaskresults 409"
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:30:06.611Z" level=info msg="Patch workflowtaskresults 200"
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:30:09.599Z" level=info progress=240/1000
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:30:18.599Z" level=info progress=250/1000
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:30:27.599Z" level=info progress=260/1000
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:30:39.599Z" level=info progress=270/1000
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:30:48.600Z" level=info progress=280/1000
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:30:57.599Z" level=info progress=290/1000
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:31:06.599Z" level=info msg="Alloc=6295 TotalAlloc=12943 Sys=19666 NumGC=6 Goroutines=8"
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:31:06.610Z" level=info msg="Create workflowtaskresults 409"
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:31:06.614Z" level=info msg="Patch workflowtaskresults 200"
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:31:09.599Z" level=info progress=300/1000
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:31:18.599Z" level=info progress=310/1000
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:31:27.599Z" level=info progress=320/1000
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:31:39.599Z" level=info progress=330/1000
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:31:48.599Z" level=info progress=340/1000
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:31:57.599Z" level=info progress=350/1000
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:32:06.608Z" level=info msg="Create workflowtaskresults 409"
progress-wxg8s-progress-730772787 wait time="2022-11-22T14:32:06.611Z" level=info msg="Patch workflowtaskresults 200"