Can't deploy all functions consistently. Fails most of the time. #3919
Comments
|
Hey we're aware of issues when deploying over 100 functions, it's best to deploy the functions you changed specifically instead of the whole lot when you're at this scale. |
usually, we deploy only changed functions, but we need to find a way to set up CI/CD on this project, also when we deploy partially the config changes are still undeployed! |
|
@abeisgoat I'm also facing a quota issue, although I'm trying to deploy just 20 functions. Can you confirm if it's related or it's another issue? The output that I have in Functions dashboard is: Deploying the functions individually works as expected, but trying to deploy all of them at the same time outputs me the error above. |
|
I have the same issue. |
|
This is also happening for us pretty consistently, 3-5 will fail with quota exceeded when deploying 60 functions. We're using firebase-tools 9.23.0. |
|
HOW do you solve this issue? Tried to deploy one by one and still getting errors. To try redeploying those functions, run: To continue deploying other features (such as database), run: Error: Functions did not deploy properly. === Deploying to 'ichat-daac0'... i deploying functions Functions deploy had errors with the following functions: To try redeploying those functions, run: To continue deploying other features (such as database), run: Error: Functions did not deploy properly. === Deploying to 'ichat-daac0'... i deploying functions Functions deploy had errors with the following functions: To try redeploying those functions, run: To continue deploying other features (such as database), run: Error: Functions did not deploy properly. Having trouble? Try firebase [command] --help |
|
Hey all - looks like there are a few different issues here, so I'll take them one at a time:
|
|
@joehan The original problem that relates with the large number of functions is the same as I'm facing? Because I have a similar issue that outputs a quota error although I'm trying to deploy around 20 functions only. I'm facing:
|
|
For clarification, are these failures longstanding issues or regressions as of a recent version of |
|
My error was solved by upgrading my billing or plan to Blaze before I could deploy functions.
|
|
Will close soon if nobody is still experiencing issues. |
|
Unfortunately I am also experiencing these issue often (if not each time) trying to deploy all my GCF at once
I know it's best to deploy only the group of functions that have been updated and leave the other ones untouched, but sometimes when updating environment configuration it is required to deploy all cloud function to allow the new configuration to propagate properly. Please advise on this issue, it would be great to find a better workaround to allow better integration with CI too. Thanks! |
|
Experiencing same issue while deploying 62 functions. 2 functions fails to deploy, thus I assume there should be somewhere 60 function deployment limit? As @42ae mentioned previously, sometimes bulk deployment is needed in order for all functions to be updated. Looking forward hearing for solutions to this problem. |
|
We are also experiencing something similar with about 80 functions where 5-6 regularly fail. We are currently on version |
|
I am frequently experiencing this issue as well while deploying over 100 functions. We deploy in groups and have created some parsing logic for our CI system, however it would be very convenient to have these auto-retry if we pass the deployment quota, or at least spit out a command after failure to deploy the rest of the functions (currently it just lists them). The current text says "waiting to retry" while we exceed the quota, so that was the behavior I was expecting, however it is not what ended up happening. |
|
I can consistently get this error even when deploying a group of only 5 functions. The error message literally shows nothing useful and it has been a real pain. I can deploy each one individually which results in deployments for a full stack to take hours. Any ideas from folks on how to overcome this would be really appreciated. EDIT: 4/7/2022 This issue seemed to only be happening in Github Actions CI on a cloud runner. It seemed perhaps the runner was running out of memory. I modified the runner to a self-hosted runner and the issues seemed to go away. Hopefully this helps someone else. |
|
We will prioritize looking into this soon. There will be a three pronged strategy:
Hopefully this will help you all when we can get these improvements ready. |
|
This is encouraging to hear; thank you for the update! |
@inlined |
|
We've currently launched (but not well documented or added wizzard support for) codebases. With codebases you can have multiple directories of Cloud Functions. Simply create a firebase.json fragment that looks a bit like: "functions" : [
{
"source": "functions",
"codebase": "main"
}, {
"source": "otherfunctions",
"codebase": "other"
}
]With this, you can now call |
@inlined Wouldn't it be better if there was an option to deploy only modified functions? ( |
|
I'm getting "Invaild source token" every time for the same 2 functions when deploying 24 Node.js v16 functions at once. If I deploy then individually, there's no error. I'm using |
|
I've sent a bug report through https://firebase.google.com/support/troubleshooter/report/bugs and they answered pointing to this part of the documentation:
But @taeold said in another issue that:
They seem to contradict each other. What am I missing? |
|
So, I was trying to deploy 24 functions at once and 2 failed. I've experimented more and realized that if I deploy exactly those 2 problematic functions, deployment happens just fine. If I deploy the 2 problematic functions + another function, the 2 problematic functions fail again while the other function is successfully deployed. EDIT: Just upgraded to |
|
Scanning through some bugs and came across this one. I wanted to give a few updates:
@Kopunk Unfortunately, a more accurate implementation of Codebases is a compromise. If you agree to put firewalls between parts of your code (e.g. directories) we can instead do a file diff and notice nothing changes. Sometimes you may have shared code and it truly isn't possible to separate your code cleanly (e.g. there are shared interfaces). For the first several years of its existence, Firebase solved this problem using an internal-API node module. I've heard that Yarn workspaces can solve this quite eloquently as well. Many teams at Google use them. |
How do I disable this new default behavior? |
There isn't a feature to disable the behavior. (but that could be a worthwhile feature request to gather public opinion.) That said, if you invoke deploy on a function with the Example: ^ This will forcefully deploy the |
|
Hey @MarwanZeitoun. We need more information to resolve this issue but there hasn't been an update in 7 weekdays. I'm marking the issue as stale and if there are no new updates in the next 3 days I will close it automatically. If you have more information that will help us get to the bottom of this, just add a comment! |
taeold
removed
Needs: Author Feedback
|
Since there haven't been any recent updates here, I am going to close this issue. @MarwanZeitoun if you're still experiencing this problem and want to continue the discussion just leave a comment here and we are happy to re-open this. |
|
@taeold @inlined can we reopen pls? and any updates or recommended best practices here? Splitting up the codebase to take advantage of "codebases" is really clunky, especially when a team is just doing it for deploy optimization... leads to having to manage multiple package.json and yarn.lock files, changing how you build things, etc. |
Hey @taeold, did the team discuss this idea any further? 🥺🤞 It's great because it eliminates deployment complexity, saves on infra cost and improves the response times, all at once. |
I only want to deploy cloud functions so I need to run |
@pedrofsantoscom This was not working for me either, after digging through the code I found that the issue was with runtime configuration producing a new hash every time. I have a PR fix for this open. |
|
@pedrofsantoscom does |
|
@mvergarair same here with latest version. |
I haven't tested that as it will deploy everything, which i don't want. |
|
|
just updated to |
|
So updating to the latest version of firebase CLI worked for me now it is saying So what happened is
What am I doing wrong? |
|
skipdeployingnoopfunctions seems to be tripping over itself. As of firebase-tools 11.29.1,
|
|
I am also facing this issue and its starting to be a real problem for me. |
@taeold Any progress on this idea? |
|
I have since learned that there is a poory/un-documented syntax for manual-deploying some-but-not-all functions: |
|
These errors are appearing much more often than they used to: If I try to deploy again it usually works. So it is not a configuration issue. Is it because we are using 1gen functions? Please try to fix the root cause and please perhaps add a --force option so all functions can be re-deployed even if now changes are detected. |
I have a script that gets the updated files inside a |
|
Nice script! What if what changes is a method in a file outside the `modules` folder, that’s imported within a file inside the modules folder but this last file does not change? We also need to detect file dependencies changes, not only file changes, am I wrong?
Enviado desde Correo<https://go.microsoft.com/fwlink/?LinkId=550986> para Windows
De: Robert ***@***.***>
Enviado: lunes, 31 de julio de 2023 17:01
Para: ***@***.***>
CC: Daniel Reinón ***@***.***>; ***@***.***>
Asunto: Re: [firebase/firebase-tools] Can't deploy all functions consistently. Fails most of the time. (Issue #3919)
I have since learned that there is a poory/un-documented syntax for manual-deploying some-but-not-all functions: firebase deploy --only functions:myFunc1,functions:myFunc2,functions:myFunc3 Not a fix, but a very useful stopgap when an arbitrary 5 of your 50 functions fail to deploy for no actionable reason.
I have a script that gets the updated files inside a modules folder (where all of my firebase functions are exported) that might be helpful to some - https://gist.github.com/wobsoriano/3634d58ead2e1f80c1dc090d3f5dd8e6
—
Reply to this email directly, view it on GitHub<#3919 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AP7W3QMBCCQQ7AXSTIN645TXS7CGHANCNFSM5IRXJ6DQ>.
You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>
|
|
Would it be possible to re-open this issue? We are running into this issue quite frequently using firebase tools v13.1.0, and deploying around 70 cloud functions. Sometimes it works, sometimes it doesn't, and there doesn't really seem to be any other option than simply trying again. |
and others
Hello;
I have a project with 169 cloud functions and need to set up automatic deployment to firebase in GitLab, but the deployment is always failed.
I tried with various node versions, also with various access tokens, but always got the same error.
Locally one of the developers is able to deploy without any issue. But locally in my environment, it fails. Even when replicating the versions of node and firebase.
here is my .yml file
`
image: node:14.16.1
before_script:
script:
- npx firebase use way-backend
- npx firebase deploy --only functions --token $FIREBASE_TOKEN
only:
refs:
- master
changes:
- functions/**/*
`
first warnings like this with many functions
⚠ functions: got "Quota Exceeded" error while trying to update projects/xproject/locations/us-central1/functions/xxxxx. Waiting to retry...and then got this one
Functions deploy had errors with the following functions: function1(us-central1) function2(us-central1) .... etc Because there were errors creating or updating functions, the following functions were not deleted To delete these, use firebase functions:deleteand not all functions throw the error only a few functions
and finally got this one
i functions: cleaning up build files... Cleaning up project directory and file based variables 00:01 ERROR: Job failed: exit code 1Thanks in advance
The text was updated successfully, but these errors were encountered: