Error: 14 UNAVAILABLE: 403:Forbidden #1814
Comments
iAMkVIN-S
added
priority: p2
product-auto-label
bot
added
the
api: firestore
|
Thanks for reporting. Looks like this issue may have also been reported through a support request. Can you confirm if that's true so we can coordinate? Thanks |
|
Thank you for the prompt reply! @MarkDuckworth Correct. I am sorry if that was the wrong thing to do. Cheers |
|
I just wanted to confirm it is the same report, so that we can coordinate internally. Thank you. I spoke with an engineer from the backend about the error code, but we're not yet sure what's causing it. Can you double check the project ID you provided on the other support channel and send an update there if needed? Also, can you help us understand how often this call to Firestore is made successfully? |
|
I have sent the project ID along through the support email thread. This call to firestore is successfully made thousands of times inbetween getting the error. I've also looked at the USAGE and it looks like we've average 21 Million+ of these READ requests per month and have been running essentially this configuration for about 2 years without ever seeing this error prior.
First encounter of this error was on Jan 17th at 5:02am [EST] while running a version of our backend that had not received a deploy in 10 days. (meaning nothing changed) On the 18th of Jan the error happened 3-4 times. Today, on the 19th of Jan, we've gotten the error a little over once per hour. |
|
Important update: This is now happening on another backend of ours that is running The last deploy of this backend was from 6 weeks ago, with a non significant change. This has also been running for years without an issue prior. An important difference here, is rebooting doesn't seem to clear the error for a little bit, we've been unable to bring it back up and if this happens to our other backend, we're going to be in trouble! |
|
Are you using App Check, or is that enabled for your Firebase app? Also, can you tell me more about what happens after this first error? Are subsequent requests failing with the same error? |
|
Thanks for the reply @MarkDuckworth .
We have to restart the backend, which sometimes the error will happen at the first request, sometimes we'll be fine for 45 minutes. (Maybe there is a timeout or cooldown on your side that isn't reached when we reboot too quickly? or maybe it's random) |
|
@iAMkVIN-S, I'm recommending that you try to resolve this through the support ticket you have. That will likely be preferred moving forward because it will allow you to share more information privately. There seem to be more eyes on that conversation as well. For us, it reduces duplicate work. If you disagree, please feel free to re-open this conversation. |
|
@MarkDuckworth Thank you so much. I've responded via the support request and will continue to do so there since the fix is unlikely to be done by our side. I hope this thread is useful for others experiencing the same error to know that the team at Firebase/Google are taking the issue seriously. Cheers |
I have the same issue. Please help us fix this. |
|
Thanks for brining that to our attention @manoslive. I'll ping the internal thread and let them know it is affecting other users. @iAMkVIN-S, are you still experiencing this? |
|
@manoslive, can you tell us when this error started occurring for you? |
|
I first noticed it on 01/17/23 at 1:57pm. Also, I have issue starting it back up. Everytime I reboot i get the same error. Currently, we can't use our backend. Here's some logs from my node.js backend: |
|
@MarkDuckworth We are very much still getting the issue. This has become mission critical as its impacting our clients businesses with so much downtime. Thankfully our backend does reboot when we get the error and sometimes get away with a couple hundreds thousand OK requests before we randomly get the error again, though not always on the first go. Since Saturday Jan 21st in the early AM, we've logged everytime a reboot was necessary due to the error: 266 times in 96 hours. This is ridiculous. How did we process 250 Million + requests over the last 2 years from this one backend alone, and without making any changes, woke up on Jan 17th to non-stop crashing. (timestamps in milliseconds) |
|
Thanks for the update. I communicated this to the backend team. @manoslive, if you can provide a project ID, that will help them diagnose. You may want to create a support ticket where you can provide your project ID. In that ticket reference internal issue b/266097991, so we can link your info to the right place. |
Done. Thanks for the reply. |
|
Here are two things that you could try which might mitigate the issue until we find the root cause and/or implement a fix. Idea 1: Try the new "enable rest" option.The networking errors are coming from the grpc networking layer. We use grpc in the sdk because it provides the complex bi-directional communication infrastructure that is needed for streaming queries (e.g. @MarkDuckworth recently added a new "prefer rest" option to the sdk which I think you should try. By enabling this new option, the requests will be made to the backend using the "rest" protocol, until a request for bi-directional streaming (i.e. To use the new "prefer rest" option, use this code to initialize the Firestore object: Idea 2: Discard clients that fail with this errorThe SDK has some logic to stop using clients that fail with the nodejs-firestore/dev/src/pool.ts Lines 270 to 279 in 789d9eb Here is how you would do it:
It would be really useful to know if discarding the clients that experience this permissions error recover when a new connection is opened. |
|
Good morning @dconeybe - Thanks a ****ton for the very useful reply. We're so relieved that we weren't in fact doing anything particularly wrong and something had indeed changed on the Cloud side of things. Within the hour of you posting last night, we started going through both your ideas, given how critical it was for the projects. As for Idea 2, it's not as straight forward for us to quickly do, since we're deploying a docker container. This means that the code gets uploaded to a Build Machine, from there everything is handled automatically within a Dockerfile and built to serve in a container. As mentioned above though, since we don't use snapshots on the backend side, it doesn't seem like we'll need to go further into Idea 2 if we continue not to crash by bypassing GRPC I'll post an update later today |
|
@MarkDuckworth / @dconeybe - We've added another 24hrs without a crash on any of the projects from Idea 1. It sounds like we have our temporary fix until a permanent solution on the Firebase side! |
|
That's great news! Thanks for the update and for trying out the ideas. I'm glad we found something to get you unblocked. One thing, though, is that since we cannot reproduce this issue ourselves, we may need you to reproduce it for us in order to investigate to find the root cause. Maybe consider how you might go about reproducing this problem in a way that doesn't grind your business to a halt. We will get back to you with suggested next steps, either via this GitHub issue or via your support ticket. |
|
@dconeybe It's unfortunate to hear that you are unable to reproduce it. Moreso, it's very suprising to me that we aren't seeing more of the 1.1M weekly installers of We'll be more than happy to get any other data you need to figure out the exact root cause, please do let us know what you'd like us to do! The team is at your disposal. |
|
@iAMkVIN-S Count me as one of the 1.1M weekly installers. My cloud functions have been running smoothly for months, then yesterday I started receiving 403 Forbidden error messages with all of my functions that make calls to an outside API. This is a critical issue as the cloud functions are core to my application. Below is one of my functions that began failing yesterday. I first have to retrieve a token from my firestore and then make the request to the outside API (Finicity). I have confirmed the url and parameters are all generated and passed correctly in the logs of each failed attempt. Once the url is received, it is passed back to the end user. It's important to note that when running my functions from a firebase shell environment, everything runs smoothly and the calls succeed . However, when I invoke the functions from my application as an end user, I receive the 403 Forbidden error message. To that end, I also have a function that runs every 80 minutes making a call to an outside api that, again runs fine when I invoke it locally, but when it is called automatically, it just started failing yesterday nearly every time. Sample Function Error Message abbreviated package.json I feel I have exhausted my options at this point. I have updated all of my dependencies, tried variations of my functions, checked all of my parameters for the api call are set correctly, confirmed with Finicity that nothing has changed on their end, confirmed user access to my functions, tried using different function names, tried increasing the memory and instances allocated to each function...nothing has worked. I will note, however, that randomly every once in a while a call will go through successfully. No idea why. I am absolutely stuck at this point. Just hoping this issue gets resolved as soon as possible. |
|
Hey @PL47productions I'm not sure if your error is exactly the same as we were getting ( However, did you try Idea 1 from @dconeybe above? Where you Initialize Firebase Admin, you can pass in a param that tells the SDK to use the REST API when the request is not a snapshot. What we had before the change: What we had after the change: If your error is the same as ours, this will fix it temporarily, since I do not see a snapshot in your above request. P.S. The error isn't from an external request, it's from the |
|
@PL47productions, thanks for letting us know that this is also impacting you. I updated the internal conversation with this information. As @iAMkVIN-S mentioned, using |
|
@iAMkVIN-S Thanks and yes, not exactly the same issue...but in the same ball park. All functions running smoothly for months, no changes, then bam a bunch of 403 forbidden errors out of no where. Unfortunately your suggestion didn't do the trick for me. @MarkDuckworth Thanks, this is a frustrating one for sure. I'm testing the unirest library out instead of axios to see if that helps at all. |
|
@PL47productions Could you reproduce with debug logging enabled and provide the logs via a https://gist.github.com/ link? If you're not comfortable sharing the logs publicly, please open a support ticket at https://firebase.google.com/support where the logs can be submitted privately. First, enable debug logging by registering a logging function (you can probably just use https://cloud.google.com/nodejs/docs/reference/firestore/latest/overview#functions Second, enable verbose GRPC logging by setting https://github.com/grpc/grpc-node/blob/master/TROUBLESHOOTING.md |
|
https://gist.github.com/PL47productions/87422a54807d6280462920f7e0cb2d01 The above link is the error output I receive for the function below that was running fine for months. I'll try enabling verbose logging and send those along. |
|
@PL47productions The error message from your logs look like the problem is connecting to |
|
@dconeybe I don't think our error and @PL47productions are the same. First, the error log is different. Second, our failing requests were for a simple .get() from the Firebase Admin SDK. Without any external API in play. Third, we've had 0 errors since adding the We still are building up a sandbox backend to reproduce the error but with the additional logs requested in the support ticket. If time is on our side, we'll hopefully have that to you today. |
|
Apologies, our issues are different indeed...mine is related to errors within firebase functions, not with firestore. I opened a support ticket with firebase and in the meantime, I added a region code to all of my calls which seems to have fixed the issue...or it just randomly started working again just as randomly as it stopped. |
Describe your environment
(Used as a subdependency)
@google-cloud/firestoreversion: 6.4.2Describe the problem
Steps to reproduce:
Our NodeJS backend, running ExpressJS and using Firebase Admin SDK has been running smoothly for 2 years. All of a sudden, on the morning of the 17th of Jan, our service was interupted by a seemignly routine Firebase call made by our baseModel. This causes all requests to fail until we reboot the backend. Since then, the error has happened randomly every 2-6+ hours with thousands of OK requests in between. The error seems to stem from the Firebase Admin SDK package/its depencies.
Relevant Code:
Error Log:
Trickled down baseModel line from our code that does the Firebase fetch:
Our package.json:
The text was updated successfully, but these errors were encountered: