Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Storage Emulator Hangs with Parallel Uploads (4+ Concurrent Requests) #6865

Open
luckyape opened this issue Mar 10, 2024 · 6 comments
Open

Storage Emulator Hangs with Parallel Uploads (4+ Concurrent Requests) #6865

luckyape opened this issue Mar 10, 2024 · 6 comments
Assignees
@tonyjhuang
Labels
emulators: storage Needs: Attention reproducible type: bug

Comments

@luckyape
Copy link

luckyape commented Mar 10, 2024 •
edited

[REQUIRED] Environment info

  • firebase-tools: 13.4.1
  • Platform: macOS Sonoma 14.3.1

[REQUIRED] Test case

      const media = [jpg1, jpg2, jpg3, jpg4]; // files range in size from 1mb to 1.5mb
      const uploadPromises = media.map((file) => {

        const storageRef = ref(storage, `temp/${file.path}`);
        return await uploadBytes(storageRef, file);
      });
      await Promise.all(uploadPromises);
      console.log('Upload promises complete');

[REQUIRED] Steps to reproduce

Run code with firebase storage and emulators configured. Observe that only the first two images upload successfully.

[REQUIRED] Expected behavior

4 images should be uploaded to storage.
Promises should resolve and 'Upload promises complete' should be logged to the console.
Synchronous execution of the uploads does work.

[REQUIRED] Actual behavior

Only 2 images are uploaded to storage.
Promises hang and do not resolve.

[debug] [2024-03-10T20:18:06.822Z] Mar. 10, 2024 5:18:06 P.M. io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
INFO: Detected non-HTTP/2 connection.
{"metadata":{"emulator":{"name":"firestore"},"message":"Mar. 10, 2024 5:18:06 P.M. io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead\nINFO: Detected non-HTTP/2 connection.\n"}}
[debug] [2024-03-10T20:18:06.854Z] Mar. 10, 2024 5:18:06 P.M. io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
INFO: Detected non-HTTP/2 connection.
{"metadata":{"emulator":{"name":"firestore"},"message":"Mar. 10, 2024 5:18:06 P.M. io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead\nINFO: Detected non-HTTP/2 connection.\n"}}
[debug] [2024-03-10T20:18:06.856Z] Mar. 10, 2024 5:18:06 P.M. io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
INFO: Detected non-HTTP/2 connection.
{"metadata":{"emulator":{"name":"firestore"},"message":"Mar. 10, 2024 5:18:06 P.M. io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead\nINFO: Detected non-HTTP/2 connection.\n"}}
[debug] [2024-03-10T20:18:06.918Z] Mar. 10, 2024 5:18:06 P.M. io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
INFO: Detected non-HTTP/2 connection.
{"metadata":{"emulator":{"name":"firestore"},"message":"Mar. 10, 2024 5:18:06 P.M. io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead\nINFO: Detected non-HTTP/2 connection.\n"}}
[debug] [2024-03-10T20:18:07.159Z] Mar. 10, 2024 5:18:07 P.M. io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
INFO: Detected non-HTTP/2 connection.
{"metadata":{"emulator":{"name":"firestore"},"message":"Mar. 10, 2024 5:18:07 P.M. io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead\nINFO: Detected non-HTTP/2 connection.\n"}}
[debug] [2024-03-10T20:18:07.216Z] Mar. 10, 2024 5:18:07 P.M. io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
INFO: Detected non-HTTP/2 connection.
{"metadata":{"emulator":{"name":"firestore"},"message":"Mar. 10, 2024 5:18:07 P.M. io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead\nINFO: Detected non-HTTP/2 connection.\n"}}
[info] {"result":{"permit":true},"id":15,"status":"ok"}
{"result":{"permit":true},"id":16,"status":"ok"} {"metadata":{"emulator":{"name":"storage"},"message":"{"result":{"permit":true},"id":15,"status":"ok"}\n{"result":{"permit":true},"id":16,"status":"ok"}"}}
[debug] [2024-03-10T20:18:07.947Z] >>> [apiv2][query] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast [none]
[debug] [2024-03-10T20:18:07.947Z] >>> [apiv2][body] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast {"eventId":"1710101887947","timestamp":"2024-03-10T20:18:07.947Z","eventType":"google.storage.object.finalize","resource":{"service":"storage.googleapis.com","name":"projects//buckets/hellograham.appspot.com/objects/temp/9a3c60d3-d789-4997-96d5-422ce35994be/3be4fc5f-d678-467e-bce7-655566f2d091.jpg","type":"storage#object"},"data":{"kind":"storage#object","name":"temp/9a3c60d3-d789-4997-96d5-422ce35994be/3be4fc5f-d678-467e-bce7-655566f2d091.jpg","bucket":"hellograham.appspot.com","generation":"1710101887908","metageneration":"1","contentType":"image/jpeg","timeCreated":"2024-03-10T20:18:07.908Z","updated":"2024-03-10T20:18:07.908Z","storageClass":"STANDARD","size":"1133688","md5Hash":"6XXm9KGgFk9ge8Pl/YYnpw==","etag":"6TEW5Thl0dy3G0IbWd9RgR8D3iA","metadata":{"firebaseStorageDownloadTokens":"906fda07-0dd7-4c49-85b5-0edb4e36d9b4"},"crc32c":"igflCw==","timeStorageClassUpdated":"2024-03-10T20:18:07.908Z","id":"hellograham.appspot.com/temp/9a3c60d3-d789-4997-96d5-422ce35994be/3be4fc5f-d678-467e-bce7-655566f2d091.jpg/1710101887908","selfLink":"http://127.0.0.1:9199/storage/v1/b/hellograham.appspot.com/o/temp%2F9a3c60d3-d789-4997-96d5-422ce35994be%2F3be4fc5f-d678-467e-bce7-655566f2d091.jpg","mediaLink":"http://127.0.0.1:9199/download/storage/v1/b/hellograham.appspot.com/o/temp%2F9a3c60d3-d789-4997-96d5-422ce35994be%2F3be4fc5f-d678-467e-bce7-655566f2d091.jpg?generation=1710101887908&alt=media"}}
[debug] [2024-03-10T20:18:07.951Z] <<< [apiv2][status] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast 200
[debug] [2024-03-10T20:18:07.951Z] <<< [apiv2][body] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast {"status":"multicast_acknowledged"}
[debug] [2024-03-10T20:18:07.952Z] >>> [apiv2][query] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast [none]
[debug] [2024-03-10T20:18:07.952Z] >>> [apiv2][body] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast {"specversion":"1.0","id":"338f9c11-a6ab-4b3e-9895-b073b28fcbb2","type":"google.cloud.storage.object.v1.finalized","source":"//storage.googleapis.com/projects//buckets/hellograham.appspot.com/objects/temp/9a3c60d3-d789-4997-96d5-422ce35994be/3be4fc5f-d678-467e-bce7-655566f2d091.jpg","time":"2024-03-10T20:18:07.908Z","data":{"kind":"storage#object","name":"temp/9a3c60d3-d789-4997-96d5-422ce35994be/3be4fc5f-d678-467e-bce7-655566f2d091.jpg","bucket":"hellograham.appspot.com","generation":"1710101887908","metageneration":"1","contentType":"image/jpeg","timeCreated":"2024-03-10T20:18:07.908Z","updated":"2024-03-10T20:18:07.908Z","storageClass":"STANDARD","size":"1133688","md5Hash":"6XXm9KGgFk9ge8Pl/YYnpw==","etag":"6TEW5Thl0dy3G0IbWd9RgR8D3iA","metadata":{"firebaseStorageDownloadTokens":"906fda07-0dd7-4c49-85b5-0edb4e36d9b4"},"crc32c":"igflCw==","timeStorageClassUpdated":"2024-03-10T20:18:07.908Z","id":"hellograham.appspot.com/temp/9a3c60d3-d789-4997-96d5-422ce35994be/3be4fc5f-d678-467e-bce7-655566f2d091.jpg/1710101887908","selfLink":"http://127.0.0.1:9199/storage/v1/b/hellograham.appspot.com/o/temp%2F9a3c60d3-d789-4997-96d5-422ce35994be%2F3be4fc5f-d678-467e-bce7-655566f2d091.jpg","mediaLink":"http://127.0.0.1:9199/download/storage/v1/b/hellograham.appspot.com/o/temp%2F9a3c60d3-d789-4997-96d5-422ce35994be%2F3be4fc5f-d678-467e-bce7-655566f2d091.jpg?generation=1710101887908&alt=media"}}
[debug] [2024-03-10T20:18:07.955Z] <<< [apiv2][status] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast 200
[debug] [2024-03-10T20:18:07.955Z] <<< [apiv2][body] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast {"status":"multicast_acknowledged"}
[debug] [2024-03-10T20:18:07.959Z] >>> [apiv2][query] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast [none]
[debug] [2024-03-10T20:18:07.959Z] >>> [apiv2][body] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast {"eventId":"1710101887959","timestamp":"2024-03-10T20:18:07.959Z","eventType":"google.storage.object.finalize","resource":{"service":"storage.googleapis.com","name":"projects//buckets/hellograham.appspot.com/objects/temp/9a3c60d3-d789-4997-96d5-422ce35994be/1e1f19ea-a9c8-4e37-8be3-57cd8a4c51da.jpg","type":"storage#object"},"data":{"kind":"storage#object","name":"temp/9a3c60d3-d789-4997-96d5-422ce35994be/1e1f19ea-a9c8-4e37-8be3-57cd8a4c51da.jpg","bucket":"hellograham.appspot.com","generation":"1710101887923","metageneration":"1","contentType":"image/jpeg","timeCreated":"2024-03-10T20:18:07.923Z","updated":"2024-03-10T20:18:07.923Z","storageClass":"STANDARD","size":"1515111","md5Hash":"KyqvqiFDwGtMrSnrGyXyYg==","etag":"y2qK0V857ty6e0G93oORBd4dtlc","metadata":{"firebaseStorageDownloadTokens":"a5c9469a-acd9-4e37-8115-bcca4c0028ed"},"crc32c":"w2JVsA==","timeStorageClassUpdated":"2024-03-10T20:18:07.923Z","id":"hellograham.appspot.com/temp/9a3c60d3-d789-4997-96d5-422ce35994be/1e1f19ea-a9c8-4e37-8be3-57cd8a4c51da.jpg/1710101887923","selfLink":"http://127.0.0.1:9199/storage/v1/b/hellograham.appspot.com/o/temp%2F9a3c60d3-d789-4997-96d5-422ce35994be%2F1e1f19ea-a9c8-4e37-8be3-57cd8a4c51da.jpg","mediaLink":"http://127.0.0.1:9199/download/storage/v1/b/hellograham.appspot.com/o/temp%2F9a3c60d3-d789-4997-96d5-422ce35994be%2F1e1f19ea-a9c8-4e37-8be3-57cd8a4c51da.jpg?generation=1710101887923&alt=media"}}
[debug] [2024-03-10T20:18:07.963Z] <<< [apiv2][status] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast 200
[debug] [2024-03-10T20:18:07.963Z] <<< [apiv2][body] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast {"status":"multicast_acknowledged"}
[debug] [2024-03-10T20:18:07.963Z] >>> [apiv2][query] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast [none]
[debug] [2024-03-10T20:18:07.963Z] >>> [apiv2][body] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast {"specversion":"1.0","id":"9692e1ed-b450-4b40-a74f-4e1db1dbeae7","type":"google.cloud.storage.object.v1.finalized","source":"//storage.googleapis.com/projects//buckets/hellograham.appspot.com/objects/temp/9a3c60d3-d789-4997-96d5-422ce35994be/1e1f19ea-a9c8-4e37-8be3-57cd8a4c51da.jpg","time":"2024-03-10T20:18:07.923Z","data":{"kind":"storage#object","name":"temp/9a3c60d3-d789-4997-96d5-422ce35994be/1e1f19ea-a9c8-4e37-8be3-57cd8a4c51da.jpg","bucket":"hellograham.appspot.com","generation":"1710101887923","metageneration":"1","contentType":"image/jpeg","timeCreated":"2024-03-10T20:18:07.923Z","updated":"2024-03-10T20:18:07.923Z","storageClass":"STANDARD","size":"1515111","md5Hash":"KyqvqiFDwGtMrSnrGyXyYg==","etag":"y2qK0V857ty6e0G93oORBd4dtlc","metadata":{"firebaseStorageDownloadTokens":"a5c9469a-acd9-4e37-8115-bcca4c0028ed"},"crc32c":"w2JVsA==","timeStorageClassUpdated":"2024-03-10T20:18:07.923Z","id":"hellograham.appspot.com/temp/9a3c60d3-d789-4997-96d5-422ce35994be/1e1f19ea-a9c8-4e37-8be3-57cd8a4c51da.jpg/1710101887923","selfLink":"http://127.0.0.1:9199/storage/v1/b/hellograham.appspot.com/o/temp%2F9a3c60d3-d789-4997-96d5-422ce35994be%2F1e1f19ea-a9c8-4e37-8be3-57cd8a4c51da.jpg","mediaLink":"http://127.0.0.1:9199/download/storage/v1/b/hellograham.appspot.com/o/temp%2F9a3c60d3-d789-4997-96d5-422ce35994be%2F1e1f19ea-a9c8-4e37-8be3-57cd8a4c51da.jpg?generation=1710101887923&alt=media"}}
[debug] [2024-03-10T20:18:07.965Z] <<< [apiv2][status] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast 200
[debug] [2024-03-10T20:18:07.965Z] <<< [apiv2][body] POST http://127.0.0.1:5001/functions/projects/hellograham/trigger_multicast {"status":"multicast_acknowledged"}
@aalej
Copy link
Contributor

aalej commented Mar 12, 2024

Hey @luckyape, thanks for reaching out and for providing a detailed report and description of the issue you’re encountering. I’m currently working on replicating this, but I’m unable to reproduce the same behavior. When uploading 4 files (~1MB in size each) at the same time, all of them get uploaded without any issues to the Storage emulator.

I created this repo to try and replicate this. Could you let me know if you’re able to reproduce the issue using the steps provided, or if I’m missing anything to reproduce the issue? Also, if you can provide a minimal reproducible example, please share it so that we can replicate the issue on our end.

@aalej aalej added the Needs: Author Feedback Issues awaiting author feedback label Mar 12, 2024
@google-oss-bot google-oss-bot added Needs: Attention and removed Needs: Author Feedback Issues awaiting author feedback labels Mar 12, 2024
@luckyape
Copy link
Author

luckyape commented Mar 13, 2024 via email

@aalej
Copy link
Contributor

aalej commented Mar 15, 2024

Thanks @luckyape for sharing a repo for this. I tried using the application you shared, but I’m still unable to reproduce the issue. The steps I took were:

  1. Run firebase emulators:start --project project_id
  2. Run yarn to install dependencies
  3. Add Firebase configs to https://github.com/luckyape/issue-6865-react-admin/blob/6a9695dc58ed9c4bf55ea29b10dbb6084cf4511d/src/App.tsx#L7-L9
  4. Run yarn dev to start the app
  5. Open "http://localhost:5173/#/firebase/create"
  6. Select files to upload
  7. Click the “UPLOAD” button
  8. Checking the Storage emulator shows that the files were uploaded:
storage-em-ss

I think I’m probably missing something here to reproduce the issue. To make sure that we're on the same page, could you let me know if the steps I took to reproduce the issue are correct?

@aalej aalej added Needs: Author Feedback Issues awaiting author feedback and removed Needs: Attention labels Mar 15, 2024
@luckyape
Copy link
Author

luckyape commented Mar 15, 2024 via email

@google-oss-bot google-oss-bot added Needs: Attention and removed Needs: Author Feedback Issues awaiting author feedback labels Mar 15, 2024
@joehan joehan added Needs: Author Feedback Issues awaiting author feedback and removed Needs: Attention labels Mar 19, 2024
@luckyape
Copy link
Author

luckyape commented Mar 23, 2024 via email

@google-oss-bot google-oss-bot added Needs: Attention and removed Needs: Author Feedback Issues awaiting author feedback labels Mar 23, 2024
@joehan joehan assigned tonyjhuang and unassigned aalej Mar 26, 2024
@aalej
Copy link
Contributor

aalej commented Mar 27, 2024

Hey @luckyape, thanks for all the information you shared. It is highly appreciated! I am now able to reproduce the issue. A few things to note, to repro the issue I did the ff:

  1. Increase the size of the images to ~2mb each. I can’t seem to reproduce the issue when using smaller images
  2. Increase the number of images being uploaded to around 6 at the same time.
  3. Upload multiple times. From what I can tell, the initial upload works just fine, all images are uploaded. However, succeeding uploads will only upload a few images (4/6 images would get uploaded)

Let me raise this to our engineering team so that they can take a look and provide some context on what could be causing this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tonyjhuang tonyjhuang

Labels
emulators: storage Needs: Attention reproducible type: bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@luckyape @tonyjhuang @joehan @google-oss-bot @aalej