Dynamic service change output file, file gets uploaded, but never gets downloaded by target service.

[wvangeit]

What happened rarely on my local machine deployment, became more problematic on osparc-master.

Sometimes when a service (notebook, python runner, etc) writes a file to the output folder (port), the log shows that the file got uploaded.
However, the file on the receiving end doesn't get downloaded (it's not present in the input folder, and there is no log message by the sidecar saying that the file got downloaded)
This is a flaky issue. The same scripts work many times, but get this particular issue from time to time, at different stages of file interactions.

[sanderegg]

@wvangeit there is another option to transfer files/number/strings/(maybe objects) via the ZeroMQ facilities that @GitHK added between these 2 services that you have that are actually. They are in a feedback loop right?
Please talk with @GitHK to learn how to use that facility. Otherwise we can discuss as well but I will need to search on how this exactly works probably @GitHK will be much faster.
The ZeroMQ system completely bypasses the usual oSparc mechanism to transfer data and goes in a point to point connection between the services. At the cost of not being saved inside the oSparc project metadata, which is ok in your case right?

[wvangeit]

Yes, I'm very aware of the zeromq facilities, I have several prototypes using these. But this system has its own issues.
One of the main being that these ports are not exposed atm in the GUI.
Tbh this is also one of the reasons why I didn't worry 'too' much about flukes from time to time on my local system, because i knew the longer future would have the zeromq ports. Unfortunately these file upload errors became way more pronounced on our master deployments.

But oth, I also think that we have to make file communications more stable. This is basically affecting all the dynamic services (at least), so it's not just me seeing it. I remember Taylor already being happy I made this to make things slightly more deterministic: https://github.com/wvangeit/osparc-filecomms

[wvangeit]

This issue has a partial workaround, in the sense that if the code that sends a file keeps writing the file in regular intervals, it eventually gets picked up by the system. However, this obviously doesn't work if one can't control the code writing the file (file pickers, other services, etc)

[wvangeit]

I'm moving this to 'Error without workaround', because this is still happening too often.
As mentioned in the previous comments, it still generates problems when one can not control the incoming service.