Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

firebase emulators:start always ignores .env file and throws “Attempted to write param-defined key” error #5219

Closed
pan93412 opened this issue Nov 5, 2022 · 24 comments · Fixed by #5330 or #5477

Comments

@pan93412
Copy link

pan93412 commented Nov 5, 2022

[REQUIRED] Environment info

firebase-tools: 11.16.0

Platform: macOS Ventura 13.0

[REQUIRED] Test case

https://github.com/pan93412/firebase-question-221105/tree/373cce9e7093f707198cfa0ff2fbba561c6b167c

[REQUIRED] Steps to reproduce

  1. Create a project that reads environment variables with firebase-functions/params, see “Test case.”
  2. Remove all the .env* files – ensure the following .env file is generated by firebase-tools.
  3. Run pnpm serve to build from TypeScript src and run firebase emulators:start
  4. firebase-tools prompts to type my DISCORD_TOKEN and GUILD – it is expected.
  5. Then, it generates a .env-[project name] file, and the function starts correctly.
  6. ⌘C, and re-run pnpm serve.
  7. It requires me to type the variables that has inputted in 4. and saved in 5. – it is unexpected.
  8. After typing the same value typed in 4. and press Enter, it throws the following message:
    Failed to load function definition from source: FirebaseError: Attempted to write param-defined key DISCORD_TOKEN to .env files, but it was already defined.
    
  9. After throwing, I can not see any functions starts, resulting me to remove the generated .env to let it work again.

The screencast:

firebase-tools-unexpected-behavior-221106.mp4

[REQUIRED] Expected behavior

  • It should read our .env file, and not prompt for typing again.
  • “.env write exception” should not block the whole start process.

[REQUIRED] Actual behavior

> functions@ serve /Volumes/Dev/Projects/ciscc-v3-backend/functions
> npm run build && firebase emulators:start --only functions "--debug"


> build
> tsc

[2022-11-05T16:28:45.339Z] > command requires scopes: ["email","openid","https://www.googleapis.com/auth/cloudplatformprojects.readonly","https://www.googleapis.com/auth/firebase","https://www.googleapis.com/auth/cloud-platform"]
[2022-11-05T16:28:45.339Z] > authorizing via signed-in user (pan93412@gmail.com)
i  emulators: Starting emulators: functions {"metadata":{"emulator":{"name":"hub"},"message":"Starting emulators: functions"}}
[2022-11-05T16:28:45.584Z] [logging] Logging Emulator only supports listening on one address (127.0.0.1). Not listening on ::1
[2022-11-05T16:28:45.584Z] assigned listening specs for emulators {"user":{"hub":[{"address":"127.0.0.1","family":"IPv4","port":4400},{"address":"::1","family":"IPv6","port":4400}],"ui":[{"address":"127.0.0.1","family":"IPv4","port":4000},{"address":"::1","family":"IPv6","port":4000}],"logging":[{"address":"127.0.0.1","family":"IPv4","port":4500}]},"metadata":{"message":"assigned listening specs for emulators"}}
[2022-11-05T16:28:45.591Z] [hub] writing locator at /var/folders/cn/c_n1ss3x7rx78k6mpw9t6bsm0000gn/T/hub-ciscc-18220.json
[2022-11-05T16:28:45.768Z] [functions] Functions Emulator only supports listening on one address (127.0.0.1). Not listening on ::1
[2022-11-05T16:28:45.768Z] [eventarc] Eventarc Emulator only supports listening on one address (127.0.0.1). Not listening on ::1
[2022-11-05T16:28:45.768Z] late-assigned ports for functions and eventarc emulators {"user":{"hub":[{"address":"127.0.0.1","family":"IPv4","port":4400},{"address":"::1","family":"IPv6","port":4400}],"ui":[{"address":"127.0.0.1","family":"IPv4","port":4000},{"address":"::1","family":"IPv6","port":4000}],"logging":[{"address":"127.0.0.1","family":"IPv4","port":4500}],"functions":[{"address":"127.0.0.1","family":"IPv4","port":5001}],"eventarc":[{"address":"127.0.0.1","family":"IPv4","port":9299}]},"metadata":{"message":"late-assigned ports for functions and eventarc emulators"}}
⚠  functions: The following emulators are not running, calls to these services from the Functions emulator will affect production: auth, firestore, database, hosting, pubsub, storage {"metadata":{"emulator":{"name":"functions"},"message":"The following emulators are not running, calls to these services from the Functions emulator will affect production: \u001b[1mauth, firestore, database, hosting, pubsub, storage\u001b[22m"}}
✔  functions: Using node@16 from host. {"metadata":{"emulator":{"name":"functions"},"message":"Using node@16 from host."}}
[2022-11-05T16:28:45.775Z] defaultcredentials: writing to file /Users/pan93412/.config/firebase/pan93412_gmail_com_application_default_credentials.json
[2022-11-05T16:28:45.779Z] Setting GAC to /Users/pan93412/.config/firebase/pan93412_gmail_com_application_default_credentials.json {"metadata":{"emulator":{"name":"functions"},"message":"Setting GAC to /Users/pan93412/.config/firebase/pan93412_gmail_com_application_default_credentials.json"}}
[2022-11-05T16:28:45.780Z] >>> [apiv2][query] GET https://firebase.googleapis.com/v1beta1/projects/ciscc-18220/adminSdkConfig [none]
[2022-11-05T16:28:46.214Z] <<< [apiv2][status] GET https://firebase.googleapis.com/v1beta1/projects/ciscc-18220/adminSdkConfig 200
[2022-11-05T16:28:46.214Z] <<< [apiv2][body] GET https://firebase.googleapis.com/v1beta1/projects/ciscc-18220/adminSdkConfig {"projectId":"ciscc-18220","storageBucket":"ciscc-18220.appspot.com"}
[2022-11-05T16:28:46.235Z] Ignoring unsupported arg: auto_download {"metadata":{"emulator":{"name":"ui"},"message":"Ignoring unsupported arg: auto_download"}}
[2022-11-05T16:28:46.235Z] Ignoring unsupported arg: port {"metadata":{"emulator":{"name":"ui"},"message":"Ignoring unsupported arg: port"}}
[2022-11-05T16:28:46.235Z] Starting Emulator UI with command {"binary":"node","args":["/Users/pan93412/.cache/firebase/emulators/ui-v1.11.1/server/server.js"],"optionalArgs":[],"joinArgs":false} {"metadata":{"emulator":{"name":"ui"},"message":"Starting Emulator UI with command {\"binary\":\"node\",\"args\":[\"/Users/pan93412/.cache/firebase/emulators/ui-v1.11.1/server/server.js\"],\"optionalArgs\":[],\"joinArgs\":false}"}}
i  ui: Emulator UI logging to ui-debug.log {"metadata":{"emulator":{"name":"ui"},"message":"Emulator UI logging to \u001b[1mui-debug.log\u001b[22m"}}
[2022-11-05T16:28:46.339Z] Web / API server started at 127.0.0.1:4000
 {"metadata":{"emulator":{"name":"ui"},"message":"Web / API server started at 127.0.0.1:4000\n"}}
[2022-11-05T16:28:46.340Z] Web / API server started at ::1:4000
 {"metadata":{"emulator":{"name":"ui"},"message":"Web / API server started at ::1:4000\n"}}
i  functions: Watching "/Volumes/Dev/Projects/ciscc-v3-backend/functions" for Cloud Functions... {"metadata":{"emulator":{"name":"functions"},"message":"Watching \"/Volumes/Dev/Projects/ciscc-v3-backend/functions\" for Cloud Functions..."}}
[2022-11-05T16:28:46.502Z] Validating nodejs source
[2022-11-05T16:28:50.025Z] > [functions] package.json contents: {
  "name": "functions",
  "scripts": {
    "lint": "eslint --ext .js,.ts .",
    "build": "tsc",
    "build:watch": "tsc --watch",
    "serve": "npm run build && firebase emulators:start --only functions",
    "shell": "npm run build && firebase functions:shell",
    "start": "npm run shell",
    "deploy": "firebase deploy --only functions",
    "logs": "firebase functions:log"
  },
  "type": "module",
  "engines": {
    "node": "16"
  },
  "main": "lib/index.js",
  "dependencies": {
    "@firebase/app-types": "^0.8.1",
    "discord-api-types": "^0.37.16",
    "discord.js": "^14.6.0",
    "firebase-admin": "^11.2.0",
    "firebase-functions": "^4.0.2",
    "nominal-types": "^0.1.2"
  },
  "devDependencies": {
    "@typescript-eslint/eslint-plugin": "^5.12.0",
    "@typescript-eslint/parser": "^5.12.0",
    "eslint": "^8.9.0",
    "eslint-config-google": "^0.14.0",
    "eslint-config-prettier": "^8.5.0",
    "eslint-plugin-import": "^2.25.4",
    "firebase-functions-test": "^3.0.0",
    "firebase-tools": "^11.16.0",
    "jest": "^29.2.2",
    "typescript": "5.0.0-dev.20221103"
  },
  "private": true
}
[2022-11-05T16:28:50.026Z] Building nodejs source
[2022-11-05T16:28:50.026Z] Analyzing nodejs backend spec
[2022-11-05T16:28:50.028Z] Could not find functions.yaml. Must use http discovery
[2022-11-05T16:28:50.194Z] Serving at port 9005

[2022-11-05T16:28:50.517Z] Got response from /__/functions.yaml {"endpoints":{"addnumbers":{"availableMemoryMb":null,"timeoutSeconds":null,"minInstances":null,"maxInstances":null,"ingressSettings":null,"concurrency":null,"serviceAccountEmail":null,"vpc":null,"platform":"gcfv2","labels":{},"callableTrigger":{},"entryPoint":"addnumbers"},"getstickers":{"availableMemoryMb":null,"timeoutSeconds":null,"minInstances":null,"maxInstances":null,"ingressSettings":null,"concurrency":null,"serviceAccountEmail":null,"vpc":null,"platform":"gcfv2","labels":{},"httpsTrigger":{},"entryPoint":"getstickers"}},"specVersion":"v1alpha1","requiredAPIs":[],"params":[{"name":"DISCORD_TOKEN","type":"string"},{"name":"GUILD","type":"string"}]}
[2022-11-05T16:28:50.524Z] shutdown requested via /__/quitquitquit

? Enter a string value for DISCORD_TOKEN: 
? Enter a string value for GUILD: 
i  functions: Loaded environment variables from .env.ciscc-18220. 
i  functions: Loaded environment variables from .env.ciscc-18220. 
⬢  functions: Failed to load function definition from source: FirebaseError: Attempted to write param-defined key DISCORD_TOKEN to .env files, but it was already defined. {"metadata":{"emulator":{"name":"functions"},"message":"Failed to load function definition from source: FirebaseError: Attempted to write param-defined key DISCORD_TOKEN to .env files, but it was already defined."}}

┌─────────────────────────────────────────────────────────────┐
│ ✔  All emulators ready! It is now safe to connect your app. │
│ i  View Emulator UI at http://127.0.0.1:4000/               │
└─────────────────────────────────────────────────────────────┘

┌───────────┬────────────────┬─────────────────────────────────┐
│ Emulator  │ Host:Port      │ View in Emulator UI             │
├───────────┼────────────────┼─────────────────────────────────┤
│ Functions │ 127.0.0.1:5001 │ http://127.0.0.1:4000/functions │
└───────────┴────────────────┴─────────────────────────────────┘
  Emulator Hub running at 127.0.0.1:4400
  Other reserved ports: 4500

Issues? Report them at https://github.com/firebase/firebase-tools/issues and attach the *-debug.log files.
 
^C[2022-11-05T16:28:59.201Z] Received signal SIGINT (Ctrl-C) 1
 
i  emulators: Received SIGINT (Ctrl-C) for the first time. Starting a clean shutdown. 
i  emulators: Please wait for a clean shutdown or send the SIGINT (Ctrl-C) signal again to stop right now. 
i  emulators: Shutting down emulators. {"metadata":{"emulator":{"name":"hub"},"message":"Shutting down emulators."}}
i  ui: Stopping Emulator UI {"metadata":{"emulator":{"name":"ui"},"message":"Stopping Emulator UI"}}
⚠  Emulator UI has exited upon receiving signal: SIGINT 
i  functions: Stopping Functions Emulator {"metadata":{"emulator":{"name":"functions"},"message":"Stopping Functions Emulator"}}
i  eventarc: Stopping Eventarc Emulator {"metadata":{"emulator":{"name":"eventarc"},"message":"Stopping Eventarc Emulator"}}
i  hub: Stopping emulator hub {"metadata":{"emulator":{"name":"hub"},"message":"Stopping emulator hub"}}
i  logging: Stopping Logging Emulator {"metadata":{"emulator":{"name":"logging"},"message":"Stopping Logging Emulator"}}
keywords: Attempted to write param-defined key, Failed to load function definition from source, environment variable, .env, params
@pan93412 pan93412 added the bug label Nov 5, 2022
@pan93412 pan93412 changed the title firebase emulators:start always ignores .env file and throws “Attempted to write param-defined key” error. firebase emulators:start always ignores .env file and throws “Attempted to write param-defined key” error Nov 5, 2022
@pan93412
Copy link
Author

pan93412 commented Nov 6, 2022

For people who want to workaround with this: Disable functionsparams experiment:

firebase experiments:disable functionsparams

Or downgrade firebase-tools to 11.15.0:

pnpm add -D firebase-tools@=11.15.0

@inlined After bisecting, I found that this issue was introduced in commit 87e8f0c, which switches functionsparams1 to true.

After I changed the functionsparams to false, it worked immediately:

image

image

Not really sure how to fix it as I'm not familiar to functionsparams. Could you help me take a look with this? Thanks!

Footnotes

  1. The source code of functionsparams feature on the latest master commit:
    https://github.com/firebase/firebase-tools/blob/98e23eded74e49f58d48be0417dc66bd85ea6e11/src/deploy/functions/build.ts#L284-L301

@colerogers
Copy link
Contributor

Hey @Berlioz I think you know this codebase a bit better than me, do you mind taking a look at this?

@juane1000
Copy link

Experiencing this exact same issue. Migrated to .env variable from functions.config method. When starting emulators it prompts for default values in the terminal even when a default is in the code.

@juane1000
Copy link

@pan93412 your two workarounds result in this error being thrown in the startup process

"functions: Failed to load function definition from source: FirebaseError: CEL identity expression '{{ params.MIN_INSTANCES }}' was not resolvable to a param"

Did you encounter this? Before trying your workarounds I did manually enter a value when prompted in the terminal but it would at least startup correctly. So I'm not sure what the cause is here.

@pan93412
Copy link
Author

@pan93412 your two workarounds result in this error being thrown in the startup process

"functions: Failed to load function definition from source: FirebaseError: CEL identity expression '{{ params.MIN_INSTANCES }}' was not resolvable to a param"

Did you encounter this? Before trying your workarounds I did manually enter a value when prompted in the terminal but it would at least startup correctly. So I'm not sure what the cause is here.

I never encountered it, sorry.

@RogerBetter
Copy link

Experiencing this exact same issue. Migrated to .env variable from functions.config method. When starting emulators it prompts for default values in the terminal even when a default is in the code.

same here

@siarheidudko
Copy link

same problem

@sceee
Copy link
Contributor

sceee commented Dec 12, 2022

To add here, please see my two linked issues that have been closed as duplicates: #5313 #5314

The current behavior is not in line with the docs. The docs here mention that When using a local Cloud Functions emulator, you can override environment variables for your project by setting up a .env.local file. Contents of .env.local take precedence over .env and the project-specific .env file..
Additionally, regarding secrets it is mentioned that Similar to Cloud Functions emulator support for environment variables, you can override secrets values by setting up a .secret.local file. This makes it easy for you to test your functions locally, especially if you don't have access to the secret value..

This not working as the emulators do not load the .env.local files as well as the .secret.local when executing firebase emulators:start or firebase emulators:exec.

@colerogers
Copy link
Contributor

Hey folks sorry for the late response, I've merged #5330 that should fix all of these issues. We'll try to cut a release asap! Thanks for your patience

@sceee
Copy link
Contributor

sceee commented Dec 21, 2022

Hi @colerogers , thanks for the fix but even after upgrading firebase-tools to v11.19.0, running my project in the local emulator does not pick up the .env.local file.
The output is:

i  functions: Loaded environment variables from .env, .env.default.

Contents of the .env.local are:

# .env.local file contains environment variables for the Functions Emulator.
FUNCTIONS_EMULATOR=true

So it seems, your fix did unfortunately not solve the issue.

And I have another related question:
Do we need to define "false" variables, too?
Taking the above example, I have the definition for FUNCTIONS_EMULATOR=true in the .env.local file. In the code this is referred to using a defineBoolean('FUNCTIONS_EMULATOR'). Do we then also need to put FUNCTIONS_EMULATOR=false in the .env file or will this default to false if it is not defined (in the given environment)?

@siarheidudko
Copy link

It still doesn't work when running command firebase emulators:exec.
For example, this is how I run automated testing firebase emulators:exec --only firestore,auth,storage './node_modules/.bin/nyc ./node_modules/.bin/mocha --exit test/functional/*.test.js test/unit/*.test.js', but emulator does not see .env.
In addition, it does not see .env if you run tests not in the emulator.
So your fix should be extended to the exec command, not just start, but also to the firebase-functions-test package.

@taeold taeold reopened this Dec 27, 2022
@taeold taeold assigned taeold and unassigned Berlioz and colerogers Dec 27, 2022
@oluckyman
Copy link

I noticed that process.env has variables from '.env' during function execution, but not during function definition:

// .env
WTF="hi there"
import * as functions from 'firebase-functions'

console.log(process.env.WTF) // undefined

export const lala = functions
  .pubsub.schedule('every 10 minutes')
  .onRun(async () => {
    console.log(process.env.WTF) // hi there
  })

@sceee
Copy link
Contributor

sceee commented Jan 12, 2023

Even in v11.20.0, it does not pick up the .env.local files when executing firebase emulators:exec or firebase emulators:start.
image

@taeold
Copy link
Contributor

taeold commented Jan 31, 2023

@oluckyman Please see https://firebase.google.com/docs/functions/config-env#params. Your environment variables are not visible during function definition.

@sceee Got a fix for this coming soon. Thanks for reporting.

taeold added a commit that referenced this issue Jan 31, 2023
…iggers. (#5477)

Fixes #5219.

Note that the actual fix is a one-liner. I'm adding basic integration test for dotenv and params support for the emulator.
@sceee
Copy link
Contributor

sceee commented Feb 1, 2023

@taeold thanks for the fix, now at least the log shows it picks up the file:
i functions: Loaded environment variables from .env, .env.default, .env.local.

Unfortunately, when using within function execution code, the values from the .env.local are still not populated when using firebase-tools 11.22.0.

Following setup:
.env.local:

# .env.local file contains environment variables for the Functions Emulator.
FUNCTIONS_EMULATOR=true

.env.:

.env.default:

SOME_OTHER_VAR=false

Now, having a file environment.ts

const environmentVariable_FunctionsEmulator = defineBoolean('FUNCTIONS_EMULATOR')

export function isRunningInEmulator(): boolean {
  console.log(`Process env: ${process.env.FUNCTIONS_EMULATOR}`)
  console.log(`defineBoolean: ${environmentVariable_FunctionsEmulator.value()}`)
  return environmentVariable_FunctionsEmulator.value() === true
}

When calling isRunningInEmulator() from a function execution code and running the code in the emulator using firebase emulators:exec, it always logs

Process env: undefined
defineBoolean: false

even though in the .env.local file, it is defined as true.

@taeold
Copy link
Contributor

taeold commented Feb 1, 2023

Hey @sceee. Do you mind sharing more about what you are trying to do with the emulator?

To my knowledge firebase emulators:exec simply spins up configured emulator instances and than creates a process to run the script passed in as the argument.

All of the features being used here - params, environment variables, etc - are only relevant to emulated functions. Functions Emulator will make sure functions defined using the Firebase Functions SDK are spun up with runtime environment similar to what you'd see when you deploy your functions to Google Cloud Functions. Script that is run as part of emulator:exec command isn't going to run in this emulated runtime environment.

@sceee
Copy link
Contributor

sceee commented Feb 2, 2023

Hey @taeold ,
Of course I will share more info.

I am having a lot of automated tests testing the Functions code that are executed using the emulator suite. And for some of these tests, I change some behavior when the functions are executed in the emulator suite compared to when the functions are executed in staging / production.
For this, I am using environment variables so I can determine in the functions whether it is executed in emulator or not.
Until now this worked perfectly by using something like this:

cross-env FUNCTIONS_EMULATOR=true firebase emulators:exec .............

Then I could just check in the functions code whether I am in the emulator like this:

export const isRunningInEmulator = process.env.FUNCTIONS_EMULATOR === 'true'

One particular example usecases for having this is testing a function that makes an external API call that I do want to "mock" out when being in the emulator. So when being in the emulator, it just returns true:

function someFunction(): boolean {
    if (isRunningInEmulator) {
        return true
    } else {
        return makeApiCall()
    }
}

someFunction() is called from a function code.

Now as Firebase encourages to use .env files for environment setups using params, I simply wanted to migrate the existing setup (within a few minutes) to the new params way.
Turned out it's unfortunately taking months instead of minutes to perform the migration ;)

The new setup is the one I mentioned in this comment:
#5219 (comment)

This is intended to just be a replacement for the cross-env / process.env solution.
So I thought I could just declare the FUNCTIONS_EMULATOR=true in the .env.local and update the isRunningInEmulator to the defineBoolean(...) as mentioned in the comment but I just can't get it to work so that the defineBoolean(...).value() emits the value from the .env.local` file.

@litehacker
Copy link

litehacker commented Mar 16, 2023

moved .env file to main root directory of the project instead of functions folder and it worked fine right after.

@sceee
Copy link
Contributor

sceee commented Mar 17, 2023

@litehacker did something similar to what I described in the last comment work for you? As it is still not working for me.

@taeold do you probably have any more insights based on the details I provided in my last comment?

@matometaru
Copy link

matometaru commented Mar 26, 2023

When I added dotenv, I was able to read the environment variables at startup.

import dotenv from 'dotenv';
dotenv.config();

@ishg
Copy link

ishg commented Mar 30, 2023

I was running into this issue as well, and tried a lot of things in this thread. The final step that helped me was upgrading firebase-tools.

$ npm install -g firebase-tools@latest
$ firebase --version
11.25.2

For good measure, I'm also using dotenv like @matometaru suggested.

import dotenv from 'dotenv';
dotenv.config();

Now, firebase emulators:start and firebase functions:shell both are able to pick up the local config from .env.local

i  functions: Loaded environment variables from .env.local

@vdiaz1130
Copy link

I spent a couple of hours on this. In my case, I had env variables starting with FIREBASE_. Renaming these variables fixed it for me. Found the tip here:
https://medium.com/rockedscience/fixing-the-firebase-functions-configuration-loading-issues-7c483b47e5f3

Give this author a clap if it helps you!

What to do if the settings are not loaded
Check the following:

  • The firebase-functions package MUST be version 3.18.0 or newer
  • The firebase-tools package MUST be version 10.2.0 or newer
  • You are not using any reserved name or prefix for the environment variables’ names

In my experience, the third case is the most common source of issues with the environment variables in Firebase Functions and not easy to detect.

Reserved env variables:
https://firebase.google.com/docs/functions/config-env#reserved-names

@jimmy-sandoval
Copy link

moved .env file to main root directory of the project instead of functions folder and it worked fine right after.

Thanks, works good!

@marthamwangi
Copy link

I spent a couple of hours on this. In my case, I had env variables starting with FIREBASE_. Renaming these variables fixed it for me. Found the tip here: https://medium.com/rockedscience/fixing-the-firebase-functions-configuration-loading-issues-7c483b47e5f3

Give this author a clap if it helps you!

What to do if the settings are not loaded
Check the following:

  • The firebase-functions package MUST be version 3.18.0 or newer
  • The firebase-tools package MUST be version 10.2.0 or newer
  • You are not using any reserved name or prefix for the environment variables’ names

In my experience, the third case is the most common source of issues with the environment variables in Firebase Functions and not easy to detect.

Reserved env variables: https://firebase.google.com/docs/functions/config-env#reserved-names

Thank you!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment