Artifact upload fails consistently

[rikur]

Expected Behavior

Artifacts should be uploaded successfully

Real Behavior

Artifact fails most of the time.

https://cirrus-ci.com/task/6607929523568640

Uploading 1 artifacts for /var/folders/v7/fs2b0v3s0lz1n57gj9y4xb5m0000gn/T/cirrus-ci-build/detox.tar
Error from upload stream: failed to upload artifact file /var/folders/v7/fs2b0v3s0lz1n57gj9y4xb5m0000gn/T/cirrus-ci-build/detox.tar: EOF
Failed to upload artifacts: failed to upload artifact file /var/folders/v7/fs2b0v3s0lz1n57gj9y4xb5m0000gn/T/cirrus-ci-build/detox.tar: EOF
Re-trying to upload artifacts...Uploading 1 artifacts for /var/folders/v7/fs2b0v3s0lz1n57gj9y4xb5m0000gn/T/cirrus-ci-build/detox.tar
Error from upload stream: failed to upload artifact file /var/folders/v7/fs2b0v3s0lz1n57gj9y4xb5m0000gn/T/cirrus-ci-build/detox.tar: EOF
Failed to upload artifacts: failed to upload artifact file /var/folders/v7/fs2b0v3s0lz1n57gj9y4xb5m0000gn/T/cirrus-ci-build/detox.tar: EOF
Re-trying to upload artifacts...
Failed to upload artifacts after multiple tries: failed to upload artifact file /var/folders/v7/fs2b0v3s0lz1n57gj9y4xb5m0000gn/T/cirrus-ci-build/detox.tar: EOF

Related Info

This is a (tick one of the following):

• Website issue
  • Link to page:
• Task issue
  • OS: (Docker, macOS, Windows, FreeBSD)
  • Task name:
  • Script/cache name (if applies):

[fkorotkov]

@rikur, do you know how big is your archive?

[rikur]

I believe the .tar archive is ~50MB in size

[rikur]

@fkorotkov In this build, only the junit.xml artifact upload succeeded, all others fail:
https://cirrus-ci.com/task/5219494574948352

[fkorotkov]

That's weird. I see Cancellation received from client error on the backed side for the upload. Maybe there is some network issue for large files on your persistent worker. Right now the upload is happening via a gRPC stream sequentially for every artifact file. We've been planning to re-architecture the mechanism to use pre-signed URLs for S3/GCS/etc so the agent can upload files in parallel.

It's hard to debug the issue here but we'll prioritize the re-architecture and hopefully by the end of the next week we can test with the improved architecture.

[rikur]

Thank you @fkorotkov. I've been cognizant of the network being a likely culprit and checked that there's no firewalls interfering with the outbound traffic. But there could still be issues upstream that I'm unable to debug at this time. The persistent worker is connected via ethernet to a 1gbps uplink, but that doesn't guarantee much :)