Workflow "invocation scheduling failed" though workflow runs properly on EU

[abueg]

Describe the bug
Hello! 👋🏼

I ran a workflow on EU and received the "invocation scheduling failed" error, but the workflow runs successfully and produces output.

Galaxy Version and/or server at which you observed the bug
Galaxy Version: Galaxy EU

version_major	"22.05"
version_minor	"1"

To Reproduce
Steps to reproduce the behavior:

1. Use this workflow: https://gist.github.com/abueg/211a95e6bfab68cf34bbcd2c25c9fa1e
2. Use items from this history: https://usegalaxy.eu/u/labueg/h/filesforissue13sept2022
3. Run workflow on those items (everything should autofill except change HiC Forward Reads to bColStr4_s4_L001_R1_001.fastq.gz)
4. See error

Expected behavior
Since the workflow runs successfully, I would expect there to not be an invocation error.
Otherwise, if there was a problem with invocation, I would not expect the workflow to run successfully.

Screenshots

Additional context
Here is the trace from @bgruening :

Sep 11 14:42:28 sn06.galaxyproject.eu python[3117704]: galaxy.workflow.run ERROR 2022-09-11 14:42:28,362 [pN:workflow_scheduler_key_1,p:3117704,tN:WorkflowRequestMonitor.monitor_thread] Failed to execute scheduled workflow.
Sep 11 14:42:28 sn06.galaxyproject.eu python[3117704]: Traceback (most recent call last):
Sep 11 14:42:28 sn06.galaxyproject.eu python[3117704]:   File "/opt/galaxy/server/lib/galaxy/workflow/run.py", line 42, in __invoke
Sep 11 14:42:28 sn06.galaxyproject.eu python[3117704]:     outputs = invoker.invoke()
Sep 11 14:42:28 sn06.galaxyproject.eu python[3117704]:   File "/opt/galaxy/server/lib/galaxy/workflow/run.py", line 142, in invoke
Sep 11 14:42:28 sn06.galaxyproject.eu python[3117704]:     remaining_steps = self.progress.remaining_steps()
Sep 11 14:42:28 sn06.galaxyproject.eu python[3117704]:   File "/opt/galaxy/server/lib/galaxy/workflow/run.py", line 285, in remaining_steps
Sep 11 14:42:28 sn06.galaxyproject.eu python[3117704]:     self._recover_mapping(invocation_step)
Sep 11 14:42:28 sn06.galaxyproject.eu python[3117704]:   File "/opt/galaxy/server/lib/galaxy/workflow/run.py", line 490, in _recover_mapping
Sep 11 14:42:28 sn06.galaxyproject.eu python[3117704]:     step_invocation.workflow_step.module.recover_mapping(step_invocation, self)
Sep 11 14:42:28 sn06.galaxyproject.eu python[3117704]:   File "/opt/galaxy/server/lib/galaxy/workflow/modules.py", line 716, in recover_mapping
Sep 11 14:42:28 sn06.galaxyproject.eu python[3117704]:     progress.set_outputs_for_input(invocation_step, already_persisted=True)
Sep 11 14:42:28 sn06.galaxyproject.eu python[3117704]:   File "/opt/galaxy/server/lib/galaxy/workflow/run.py", line 396, in set_outputs_for_input
Sep 11 14:42:28 sn06.galaxyproject.eu python[3117704]:     raise ValueError(message)
Sep 11 14:42:28 sn06.galaxyproject.eu python[3117704]: ValueError: Step with id WorkflowStep[index=1,type=data_input] not found in inputs_step_id ({2486945: <galaxy.model.HistoryDatasetAssociation(115138356) at 0x7f764e669e20>, 2486950: <galaxy.model.HistoryDatasetAssociation(115138354) at 0x7f7635cc8640>, 2486947: <galaxy.model.HistoryDatasetAssociation(115011747) at 0x7f76366fcaf0>, 2486949: <galaxy.model.HistoryDatasetAssociation(115138355) at 0x7f76377c14f0>, 2486946: <galaxy.model.HistoryDatasetAssociation(115138364) at 0x7f764caf6190>})

[nsoranzo]

@abueg Thanks for the report. Is the error reproducible if you run the workflow again on the same inputs?

[abueg]

Yes, I ran the workflow again on the inputs from that history and received the error again:

I will keep an eye on the history with the results to see if it runs properly again.

[abueg]

The workflow has run properly, and the results are in this history here: https://usegalaxy.eu/u/labueg/h/copy-of-vgp-hic-yahs-edited-07-sept-no-re-start-with-bed-imported-from-url

[mvdbeek]

I'm having some trouble finding the Sequence graph input in your input history. This should be an AGP file, where can I find this ?

[mvdbeek]

Nevermind, I see what's happening here, the input is marked as a workflow output, but it is optional, so that's why it fails scheduling. That's an interesting case.

[abueg]

Ah yes sorry -- so the AGP file is optional, and we do usually run the workflow without it. Should we remove that optional input in this case, to avoid the error?

[mvdbeek]

No, that wasn't the issue in the end. I'm still not 100% sure why that is, but the issue can be avoided by making the optional SAK file runtime input a formal optional input, see https://usegalaxy.eu/u/m.vandenbeek/w/copy-of-vgp-hic-yahs-edited-07-sept-no-re-start-with-bed-imported-from-url ... doing that prevents the use of the old workflow run form, which I believe does the workflow submission a little different.

[abueg]

I see, I'll do that from now on -- thank you for looking into this! ☺️

[mvdbeek]

Thanks for the great and detailed report @abueg, this is fixed in #14641, the original workflow should now schedule correctly without any workaround (pending an update on the Galaxy servers, should happen during the weekend)