RST_STREAM error keeps showing up #1023
Comments
product-auto-label
bot
added
the
api: firestore
jakeleventhal
changed the title
bcoe
added
the
type: bug
|
@jakeleventhal Thanks for filing this. Do you know at all what type of request/API call this is related to? |
|
The only place i use grpc is via firestore so it must be something from there. I don't really have more info unfortunately |
BenWhitehead
added
priority: p2
|
I also got this one with {"error":"Error: 13 INTERNAL: Received RST_STREAM with code 2
at Object.callErrorFromStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/call.js:30:26)
at Object.onReceiveStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client.js:175:52)
at Object.onReceiveStatus ... |
|
This is a weird jeopardize issue, thrown during |
|
All document modifications (e.g. |
|
Hi @schmidt-sebastian. |
|
After a while, errors, as mentioned by @jakeleventhal, are thrown and it’s not possible to persist data in firestore and an instance restart is required. That’s a real problem folks 😄 |
|
@Skonx To rule out that this is a problem with our backend, can you send me your project ID and the time at which you saw these repeated failures? My email is If it is not a backend issue, we likely need GRPC logs from to further diagnose these failures. These can be obtained by setting two environment variables: GRPC_TRACE=all, GRPC_VERBOSITY=DEBUG Note that this create a lot of logs, so hopefully this is not something we have to look at. |
|
Yep. I've replied to your google.com email address. |
|
Hello! We also experience the same error usually coupled with We noticed that it usually fails all at once and then recovers very quickly. |
|
We are also affected by this error. A month before, it was a sporadic error, maybe once a week, and now we are seeing it many times per day. |
|
I'm also experiencing the issue for over a month now, with a couple hundred to a few thousand errors per day:
@schmidt-sebastian I opened a support case if you need a project ID or other information: 23689958. |
|
Same problem here: Received RST_STREAM with code 2 |
|
We were able to resolve the
After the update, the various other DB related errors have also gone away, including We currently use Electron in a Google App Engine project and work in the standard runtime. Google has not updated its images to include some necessary libs for the new version of Electron. We were able to work around this as Google supports these libs in Puppeteer, so we installed Puppeteer and sent a deploy off to our production server (on a no-promote flag). After doing so, our best guess is that Google rebuilt our server images with the needed libs to run Puppeteer, which in turn allowed us to run the new version of Electron. I hope this information helps! We spent a lot of time diving into this, so if you have any questions, feel free to respond below. |
|
Our backend team has looked at the projects that were sent to us. There are no indications that your errors correlate with errors on the backend, which makes it likely that this is a client side issue. I will continue to investigate possible root causes. |
|
As of v3.8.5, we are now retrying RunQuery requests that fail with RST_STREAM. If this error shows up again, we can evaluate expanding our retry to other SDK methods as well. |
|
@schmidt-sebastian I've been running on v3.8.5 for the entire day, I still see RST_STREAM. I checked my code to see for which usage patterns does it occur:
Can we please reopen this issue for visibility? |
|
@merlinnot We continue to receive pushback on retrying RST_STREAM/INTERNAL for Commit RPCs. I hope your comment is enough to convince the respective folks that we should retry. |
|
hey all - on @grpc/grpc-js version 1.6.0 - i got errors like this -
with the NEWER version of @grpc/grpc-js 1.6.7 - for the same case i got the following errors.
So, basically I'm still getting errors with a highly parallel write scenario - but now I'm getting "Call Cancelled" with grpc v1.6.7, versus "DEADLINE_EXCEEDED" with v 1.6.0. whats the meaning behind Call Cancelled..? googling this - it looks like it is OK to retry these if the write is idempotent? |
|
I'm getting this error when using firebase from within cypress and connecting it to emulator for e2e testing. I can't make any requests to populate the database because of this error. |
|
an update from my end - we had to rearchitect our data layout to avoid more than one write to a document per second (This is described in the best practices for firestore) https://firebase.google.com/docs/firestore/best-practices#updates_to_a_single_document i've never heard of a data store that limits you to 1 write per second.... but whatever. the thing is, you can get away with having a few writes to a document for a second, but if you start doing it at scale and highly parallel, you are 1) more likely to get the issue 2) less likely to be able to just retry and have the write succeed. i put in a retry of 3, and was still unable to get a successful write when dealing with hundreds of concurrent writes. so, although i do think that firestore should really work on these issues and not have weird best practices like 1 write a second - we were able to rewrite our app to get things to succeed (with minimal retrying) |
|
Trying to write to collection. Protocol error. No info anywhere. What should be done ? "firebase-admin": "^11.5.0"
|
|
Happens to me with the emulator on every attempt, so maybe an easy case to reproduce package.json Under Node.js /sveltekit server endpoint. Here are my lib versions: My use case is pretty simple: My code using the initialized db variable: +server.js Just to be complete if you want to reproduce, here's a file specific to sveltekit allowing CORS |
Do you want the reproduction to be in a git repo? or is the code I shared here sufficient? |
Same for me |
|
Running finebase init again, things solved. I recommend doing it in case you forgot to correctly initialize something (I think I surely did, as I was discovering firebase) |
|
@ziedHamdi What was your solution? I ran firebase base init also again, but to no avail. |
|
Update: I ended up moving off of Firestore and moved to a Postgres DB, and ultimately everything from GCP to AWS because of this issue. |
Same here, we have millions of error spans with microseconds duration in our server due to this. Our system initializes a snapshot listeners to get changes on a specific document we have to control our feature flags, it should only instantiate once when a new instance of our NestJs server boots up and then only be called when changes happen, but we get millions of those small 10 micro second errors due to DNS errors, connection errors and now RST_STREAM. Our server is hosted in Render.com, ALL the normal Firestore calls work as expected, no requests ever failed because of this, the snapshot listener is just erroring out like this, all the time. I tried contacting GCP support on this, they have zero clues about what is the problem or how to try to help as they don't see any errors on their side. 
|
|
I just finished my backend local development and everything work well. Then I deployed and none of my functions work. I'm getting the same error. My code is pretty simple. this.#log.i(`create license`, JSON.stringify({license: license, id: id}));
const licenseDoc = this.ref.doc(id);
await licenseDoc.set(license); // <===== This is where the error originates from
return licenseDoc.id;Using Node.js 16 I don't get it once or twice, it literally happens everytime. My endpoint doesn't work at all. This is a brand new project on Firestore and my first deployment. Package.json dependencies "dependencies": {
"firebase-admin": "^11.10.1",
"firebase-backend": "^0.2.5",
"firebase-functions": "^4.4.1",
"uuid": "^9.0.1"
},Is there anything I can try to get past this? We need to launch our product and I'd hate to need another week to rewrite all these endpoints on a different platform. |
|
We've been facing this error in our production environment for the past 3 days and it's occurred roughly 10,600 times:
The error is triggered when executing the following code: Do we have any updates or workarounds for this? It's affecting our users and we'd appreciate your guidance. Note: Our Users collection has a significantly large number of documents. Could the volume of documents be a contributing factor to this issue? |
Have you found a solution to this? because i am facing the same exact error. |
|
Hello @CollinsVizion35, We haven't found a solution to this issue yet. We've attempted several methods, but none have resolved the problem:
Interestingly, everything operates flawlessly in our development project. The only difference is that the development project has a smaller User collection. I'm starting to suspect that this might be related to some undocumented limitation in Firestore... I will stay in touch about updates! |
|
Okay, thank you. I tried using batch commit and it still didn't work. |
|
I've got a workaround/solution in my situation |
|
Hey @maylorsan, i think i have found a solution from @edmilsonss I think it works with these changes former code: // Create a Firestore instance new code: // Create a Firestore instance |
|
@CollinsVizion35 Indeed, we haven't experimented with that solution just yet. As I mentioned in this comment, our primary approach was to optimize our algorithm logic between Firebase calls. Thankfully, this seems to have resolved the issue for now. It's certainly an unusual behavior 😄 |
|
Any update on this issue? Happens sporadically for us in production, using This seems to be occurring when instantiating too many transactions simultaneously, perhaps it initiates too many gRPC connections to the google API? |
|
@maylorsan Setting |
|
@sammyKhan, my apologies for the delayed reply! I wanted to clarify that we don't employ
In our case it appears that the issue we're encountering arises due to a significant delay occurring between the |
|
Hi @maylorsan Since your case is different to what is been reported in this issue, could you please open a new ticket and describe your problem in detail? |
|
Hi @cherylEnkidu, Thanks for the advice. I'll open a new ticket with all relevant details to address our specific Firestore issue. |
|
We still see this intermittently, no real pattern that I can see. The Firestore code that triggers it is simple and runs successfully 99% of the time: But every so often we'll see the error. I've been progressively adding more debugging logging to the function to see if I can work out what may be causing it but there is nothing of note that I can see. Using up-to-date dependencies and node version.
Stack trace: |
|
I think it's an issue with the GRPC library. We used to get this error
sometimes or a timeout and all our subsequent Firestore requests would fail
causing endless chaos.
googleapis/nodejs-datastore#1176
The recommended solution for the issue is the below - it falls back to
rest whenever GRPC is glitching.
const datastore = new Datastore({fallback:'rest'});
…On Thu, Feb 29, 2024 at 5:23 PM Adam Koch ***@***.***> wrote:
We still see this intermittently, no real pattern that I can see. The
Firestore code that triggers it is simple and runs successfully 99% of the
time:
await ref.set(updateData, {
merge: true,
});
But every so often we'll see the error. I've been progressively adding
more debugging logging to the function to see if I can work out what may be
causing it but there is nothing of note that I can see.
Using up-to-date dependencies and node version.
~/ $ node --version
v18.18.2
package.json dependencies:
"dependencies": {
***@***.***/firestore": "^7.3.0",
***@***.***/logging": "^11.0.0",
"firebase-admin": "^12.0.0",
"firebase-functions": "^4.7.0",
"googleapis": "^132.0.0",
...
},
Stack trace:
Error: 13 INTERNAL: Received RST_STREAM with code 2 (Internal server error) Error: 13 INTERNAL: Received RST_STREAM with code 2 (Internal server error)
at callErrorFromStatus ***@***.***/grpc-js/build/src/call.js:31:19)
at Object.onReceiveStatus ***@***.***/grpc-js/build/src/client.js:192:76)
at Object.onReceiveStatus ***@***.***/grpc-js/build/src/client-interceptors.js:360:141)
at Object.onReceiveStatus ***@***.***/grpc-js/build/src/client-interceptors.js:323:181)
at ***@***.***/grpc-js/build/src/resolving-call.js:99:78
at process.processTicksAndRejections (node:internal/process/task_queues:77:11)
for call at
at ServiceClientImpl.makeUnaryRequest ***@***.***/grpc-js/build/src/client.js:160:32)
at ServiceClientImpl.<anonymous> ***@***.***/grpc-js/build/src/make-client.js:105:19)
at ***@***.***/firestore/build/src/v1/firestore_client.js:231:29
at /workspace/node_modules/google-gax/build/src/normalCalls/timeout.js:44:16
at repeat (/workspace/node_modules/google-gax/build/src/normalCalls/retries.js:80:25)
at /workspace/node_modules/google-gax/build/src/normalCalls/retries.js:118:13
at OngoingCallPromise.call (/workspace/node_modules/google-gax/build/src/call.js:67:27)
at NormalApiCaller.call (/workspace/node_modules/google-gax/build/src/normalCalls/normalApiCaller.js:34:19)
at /workspace/node_modules/google-gax/build/src/createApiCall.js:84:30
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Caused by: Error
at WriteBatch.commit ***@***.***/firestore/build/src/write-batch.js:432:23)
at DocumentReference.set ***@***.***/firestore/build/src/reference.js:398:27)
at /workspace/lib/auth.js:201:19
at Generator.next (<anonymous>)
at fulfilled (/workspace/lib/auth.js:5:58)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
code: 13,
details: 'Received RST_STREAM with code 2 (Internal server error)',
metadata: Metadata { internalRepr: Map(0) {}, options: {} },
note: 'Exception occurred in retry method that was not classified as transient'
}
at console.error (/workspace/node_modules/firebase-functions/lib/logger/compat.js:31:23)
at /workspace/lib/auth.js:207:17
at Generator.throw (<anonymous>)
at rejected (/workspace/lib/auth.js:6:65)
—
Reply to this email directly, view it on GitHub
<#1023 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AIWKQUIUHFMMMAV3O4YRAF3YV637DAVCNFSM4MJANDO2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOJXGIYTGOJQGY3A>
.
You are receiving this because you commented.Message ID:
***@***.***>
--
Aditya Ramani
Ramani Data Science LLC
https://www.linkedin.com/in/adiramani/ 773-580-5665
|
|
any news on this issue? 2 years with the same problem here at the company, we tried
but it's too slow |
and others
Environment details
@google-cloud/firestoreversion: 3.7.4Steps to reproduce
This error keeps appearing over and over in my logs (not regularly reproducible):
The text was updated successfully, but these errors were encountered: