Skip to content

Commit

Permalink
Launch endpoints refactor (#3831)
Browse files Browse the repository at this point in the history 
* Trigger discovery now includes Endpoints (#3752)

* Fork translator implementations

* TriggerDiscovery now records Endpoints as well as Specs

* Remove debug print

* Start moving over cases of existingBackend (#3753)

* Fork translator implementations

* TriggerDiscovery now records Endpoints as well as Specs

* Remove debug print

* Start moving over existing use cases

* Migrate the deploy codebase aside from release code (#3754)

* Fork translator implementations

* TriggerDiscovery now records Endpoints as well as Specs

* Remove debug print

* Start moving over existing use cases

* Migrate the deploy codebase aside from release code

* PR feedback

* Use eventType to infer service from function trigger definition in the Emulator. (#3712)

* Refactor in preparation for adding multitenancy support (#3746)

This change introduces the refactor discussed in go/emulator-multi-tenancy. Main changes include:

* Updating how `ProjectState` is indexed in `getProjectStateById()`
* Extending `ProjectState` with `AgentProjectState` and `TenantProjectState`

Corresponding internal bug: b/192387245

* Add MFA support for Auth Emulator (#3732)

* Update API schema from discovery.

* WIP: Add MFA support.

* Implement withdraw and token fields.

* Add error handling for most cases.

* Clean up some comments.

* Support MFA in batchCreate.

* Properly support MFA for emailLink and IDP.

* Address review feedback.

* Clean up more TODOs.

* Update gen-auth-api-spec.ts to use flatPath instead of path (#3761)

Add preprocessing logic to use flatPaths instead of paths in discovery object.

Some endpoints/methods have the same path, which led to endpoints being overwritten and not appearing in apiSpec.js (e.g. see identitytoolkit.projects.inboundSamlConfigs.delete and identitytoolkit.projects.defaultSupportedIdpConfigs.delete in v2 discovery docs). To work around this, use the flatPath which are distinct between overloaded paths. This change largely impacts paths that have params with slashes in them, which is currently not supported by OpenAPI.

Corresponding internal bug: b/199768026

* Adding noninteractive support to ext:dev:publish (#3745)

* Adding noninteractive support to ext:dev:publish

* Add --non-interactive and --force support to ext:update (#3749)

* ext:update now supports --non-interactive and --force

* fixed test

* add changelog

* formats

* Extract environment variables initialization out of Functions Emulator Runtime. (#3707)

As a step towards standardizing Functions Runtime environment, we extract all logic for initializing environment variables for a n emulated function out of the `FunctionsEmulatorRuntime` and place them in the caller (`FunctionsEmulator`).

All behaviors should remain the same minus the following details:

1. Environment variable values will not change once the runtime has started. Env vars like `FUNCTIONS_TARGET` might look like it should change per function invocation, but since each worker is associated with a trigger,`FUNCTION_TARGET` won't actually ever change. This may be a problem though if other emulators (e.g. storage emulator) is started _after_ the functions emulator is started and a function is invoked. (I don't think this is a common use case?).

2. We no longer check user's Functions SDK version to optionally override realtime emulator path. I think it's actually better to fail the emulator IF user has configured realtime database emulator and using an unsupported version of the functions SDK rather than silently talking to the prod database. I can be convinced otherwise though.

* security(deps): proxy-agent@5.0.0 (CVE-2021-23406) (#3757)

https://snyk.io/vuln/SNYK-JS-PACRESOLVER-1564857

firebase-tools@9.17.0 requires pac-resolver@^4.1.0 via a transitive dependency on pac-proxy-agent@4.1.0.

upgrading to proxy-agent@5.0.0 causes pac-resolver to get upgraded to 5.0.0, resolving the issue.

Co-authored-by: Bryan Kendall <bkend@google.com>

* adding externalServices to extensions types (#3766)

* fix login:use and account resolution issue (#3773)

* Minor fixes for Auth Emulator IDP widget. (#3774)

* Minor fixes for Auth Emulator IDP widget.

* Changelog.

* Move ref parsing code to a separate file. (#3779)

* migrating parseRef to ref

* moving ref parsing into its own file

* add jsdocs

* renaming to refs

* forgot to save

* Generate CloudEvents in the PubSub Emulator (#3767)

Update PubSub Emulator to emit events in CloudEvent format given a trigger whose function signature is `cloudevent`.

Previously, the Functions Emulator considered 2 types of functions - `https` and `background`. With the introduction of [CloudEvents](https://cloud.google.com/functions/docs/writing/cloudevents) in Google Cloud Functions,  we now have 2 types of `background` functions - `legacy` vs `cloudevent`. For example:

```js
/* Legacy Events */
exports.helloPubSub = functions.pubsub.topic('topic-name').onPublish((message) => {
  // ...
});

/* CloudEvents */
exports.helloPubSub = functions.v2.pubsub.topic('topic-name').onPublish((cloudevent) => {
  const message = cloudevent.data.message;
});
```

To accommodate, we adopt concept of "function signature" introduced in Google Cloud Function's [Functions Framework](https://github.com/GoogleCloudPlatform/functions-framework-nodejs) to categorize each function trigger as one of `http`, `event`, or`cloudevent`.

When registering a function trigger with the Pubsub emulator, the functions's signature type will be included in the registration request, and the Pubsub emulator will use the info to emit an event in the format expected by the underlying function trigger. We also make minor changes to the Functions Emulator to selectively massage the incoming events in-route to the function trigger.

As an added bonus, we are now able to remove `triggerType` field in the `FunctionsRuntimeArg` as that info is now outdated and not needed when invoking an emulated function.

* Add Endpoints version of DeploymentPlanner (#3756)

* Fork translator implementations

* TriggerDiscovery now records Endpoints as well as Specs

* Remove debug print

* Start moving over existing use cases

* Migrate the deploy codebase aside from release code

* Migrate deploymentPlanner

* Make code easier to review

* PR feedback

* Start inferring wantBackend details from existingBackend (#3790)

* Start inferring wantBackend details from existingBackend

* Update src/test/deploy/functions/prepare.spec.ts

Co-authored-by: Daniel Lee <danielylee@google.com>

Co-authored-by: Daniel Lee <danielylee@google.com>

* Adds Fabricator to release code (#3807)

* Adds Fabricator to release code

Fabricator is the repalcement for the old Tasks codebase, though
it will not handle the portion of tracking or error reporting
internally (those will be future utilities for better testing).

Some differences between tasks.ts and fabricator.ts:
* Queues are used for narrow transactions, which means we don't
  accidentally have an error bubble up and cause a retry on a bulk
  operation. There's a design pattern of executor.run.catch(rethrowAs)
* Function triggers (not EventArc triggers, which are deployed as
  part of the function, but external triggers like topics, and in
  the future intents or task queues) are only created after a
  successful function creation and functions are only deleted
  after a successful trigger deletion
* We do not delete functions if an upsert failed (this helps catch
  cases where a user renamed a function)
* This code is actually unit tested (hooray!)

* Delete scratch space code

* PR feedback + tests & fixes for SourceTokenScraper

* Merge master; backport recent changes (#3816)

Merge master; backport @colerogers' changes to Endpoint code.

* Add metrics and error reporting to release code (#3815)

Adds Fabricator to release code

Fabricator is the repalcement for the old Tasks codebase, though
it will not handle the portion of tracking or error reporting
internally (those will be future utilities for better testing).

Some differences between tasks.ts and fabricator.ts:
* Queues are used for narrow transactions, which means we don't
  accidentally have an error bubble up and cause a retry on a bulk
  operation. There's a design pattern of executor.run.catch(rethrowAs)
* Function triggers (not EventArc triggers, which are deployed as
  part of the function, but external triggers like topics, and in
  the future intents or task queues) are only created after a
  successful function creation and functions are only deleted
  after a successful trigger deletion
* We do not delete functions if an upsert failed (this helps catch
  cases where a user renamed a function)
* This code is actually unit tested (hooray!)

* Move firebase functions:delete to use Endpoints code (#3821)

* Move functions:delete to endpoints code

* Remove old functions delete code

* Make container contract parsing use Endpoints (#3826)

* Inlined.endpoints 9.6.discovery (#3830)

* Make container contract parsing use Endpoints

* Finish the endpoints refactor! (#3827)

* Delete dead code; make a few fixes as I discover them

* Free up 12LOC to meet budget

* Fixes from bugbashing (#3832)

* Don't obscure helpful error messages

* Functions should have deployment-tool label and functions:delete shouldn't need it

* Lost commit? Record URI after deploys for future printing

* Lint fixes

* Alternative deploymentTool fix

* get rid of extra space in logOpStart

* Don't cache trigger regions if the trigger changed

* functions list should list ID not entrypoint

* Fix compiler erro

* Rewrite trigger region copy code

* Enablement of APIs should have 'functions' prefix

* Remember old trigger region. Recreate Fn when it changes

* Run formatter

Co-authored-by: Daniel Lee <danielylee@google.com>
Co-authored-by: Lisa Jian <lisajian@google.com>
Co-authored-by: Yuchen Shi <yuchenshi@google.com>
Co-authored-by: joehan <joehanley@google.com>
Co-authored-by: Avi Vahl <avi.vahl@wix.com>
Co-authored-by: Bryan Kendall <bkend@google.com>
  • Loading branch information
@inlined @taeold
@lisajian @yuchenshi @joehan @AviVahl @bkendall
7 people committed Oct 19, 2021
1 parent 925bdc3 commit 6529c97
Show file tree
Hide file tree
Showing 55 changed files with 4,405 additions and 4,306 deletions.
132 changes: 69 additions & 63 deletions src/commands/functions-delete.ts
View file
@@ -1,16 +1,21 @@
import { Command } from "../command";
import * as clc from "cli-color";
import * as functionsConfig from "../functionsConfig";
import { deleteFunctions } from "../functionsDelete";

import { Command } from "../command";
import { FirebaseError } from "../error";
import { Options } from "../options";
import { needProjectId } from "../projectUtils";
import { promptOnce } from "../prompt";
import * as helper from "../deploy/functions/functionsDeployHelper";
import { reduceFlat } from "../functional";
import { requirePermissions } from "../requirePermissions";
import * as utils from "../utils";
import * as args from "../deploy/functions/args";
import * as helper from "../deploy/functions/functionsDeployHelper";
import * as utils from "../utils";
import * as backend from "../deploy/functions/backend";
import { Options } from "../options";
import { FirebaseError } from "../error";
import * as planner from "../deploy/functions/release/planner";
import * as fabricator from "../deploy/functions/release/fabricator";
import * as executor from "../deploy/functions/release/executor";
import * as reporter from "../deploy/functions/release/reporter";

export default new Command("functions:delete [filters...]")
.description("delete one or more Cloud Functions by name or group name.")
Expand All @@ -26,72 +31,73 @@ export default new Command("functions:delete [filters...]")
return utils.reject("Must supply at least function or group name.");
}

const context = {
const context: args.Context = {
projectId: needProjectId(options),
} as args.Context;
filters: filters.map((f) => f.split(".")),
};

const [config, existingBackend] = await Promise.all([
functionsConfig.getFirebaseConfig(options),
backend.existingBackend(context),
]);
await backend.checkAvailability(context, /* want=*/ backend.empty());
const appEngineLocation = functionsConfig.getAppEngineLocation(config);

// Dot notation can be used to indicate function inside of a group
const filterChunks = filters.map((filter: string) => {
return filter.split(".");
if (options.region) {
existingBackend.endpoints = { [options.region]: existingBackend.endpoints[options.region] };
}
const plan = planner.createDeploymentPlan(/* want= */ backend.empty(), existingBackend, {
filters: context.filters,
deleteAll: true,
});
const allEpToDelete = Object.values(plan)
.map((changes) => changes.endpointsToDelete)
.reduce(reduceFlat, [])
.sort(backend.compareFunctions);
if (allEpToDelete.length === 0) {
throw new FirebaseError(
`The specified filters do not match any existing functions in project ${clc.bold(
context.projectId
)}.`
);
}

try {
const [config, existingBackend] = await Promise.all([
functionsConfig.getFirebaseConfig(options),
backend.existingBackend(context),
]);
await backend.checkAvailability(context, /* want=*/ backend.empty());
const appEngineLocation = functionsConfig.getAppEngineLocation(config);
const deleteList = allEpToDelete.map((func) => `\t${helper.getFunctionLabel(func)}`).join("\n");
const confirmDeletion = await promptOnce(
{
type: "confirm",
name: "force",
default: false,
message:
"You are about to delete the following Cloud Functions:\n" +
deleteList +
"\n Are you sure?",
},
options
);
if (!confirmDeletion) {
throw new FirebaseError("Command aborted.");
}

const functionsToDelete = existingBackend.cloudFunctions.filter((fn) => {
const regionMatches = options.region ? fn.region === options.region : true;
const nameMatches = helper.functionMatchesAnyGroup(fn, filterChunks);
return regionMatches && nameMatches;
});
if (functionsToDelete.length === 0) {
throw new Error(
`The specified filters do not match any existing functions in project ${clc.bold(
context.projectId
)}.`
);
}
const functionExecutor: executor.QueueExecutor = new executor.QueueExecutor({
retries: 30,
backoff: 20000,
concurrency: 40,
maxBackoff: 40000,
});

const schedulesToDelete = existingBackend.schedules.filter((schedule) => {
functionsToDelete.some(backend.sameFunctionName(schedule.targetService));
});
const topicsToDelete = existingBackend.topics.filter((topic) => {
functionsToDelete.some(backend.sameFunctionName(topic.targetService));
try {
const fab = new fabricator.Fabricator({
functionExecutor,
executor: new executor.QueueExecutor({}),
appEngineLocation,
});

const deleteList = functionsToDelete
.map((func) => {
return "\t" + helper.getFunctionLabel(func);
})
.join("\n");
const confirmDeletion = await promptOnce(
{
type: "confirm",
name: "force",
default: false,
message:
"You are about to delete the following Cloud Functions:\n" +
deleteList +
"\n Are you sure?",
},
options
);
if (!confirmDeletion) {
throw new Error("Command aborted.");
}
return await deleteFunctions(
functionsToDelete,
schedulesToDelete,
topicsToDelete,
appEngineLocation
);
const summary = await fab.applyPlan(plan);
await reporter.logAndTrackDeployStats(summary);
reporter.printErrors(summary);
} catch (err) {
throw new FirebaseError("Failed to delete functions", {
original: err,
original: err as Error,
exit: 1,
});
}
Expand Down
22 changes: 11 additions & 11 deletions src/commands/functions-list.ts
View file
Expand Up @@ -5,7 +5,6 @@ import { needProjectId } from "../projectUtils";
import { Options } from "../options";
import { requirePermissions } from "../requirePermissions";
import * as backend from "../deploy/functions/backend";
import { listFunctions } from "../functions/listFunctions";
import { previews } from "../previews";
import { logger } from "../logger";
import Table = require("cli-table");
Expand All @@ -18,7 +17,8 @@ export default new Command("functions:list")
const context = {
projectId: needProjectId(options),
} as args.Context;
const functionList = await listFunctions(context);
const existing = await backend.existingBackend(context);
const endpointsList = backend.allEndpoints(existing).sort(backend.compareFunctions);
const table = previews.functionsv2
? new Table({
head: ["Function", "Version", "Trigger", "Location", "Memory", "Runtime"],
Expand All @@ -28,23 +28,23 @@ export default new Command("functions:list")
head: ["Function", "Trigger", "Location", "Memory", "Runtime"],
style: { head: ["yellow"] },
});
for (const fnSpec of functionList.functions) {
const trigger = backend.isEventTrigger(fnSpec.trigger) ? fnSpec.trigger.eventType : "https";
const availableMemoryMb = fnSpec.availableMemoryMb || "---";
for (const endpoint of endpointsList) {
const trigger = backend.endpointTriggerType(endpoint);
const availableMemoryMb = endpoint.availableMemoryMb || "---";
const entry = previews.functionsv2
? [
fnSpec.entryPoint,
fnSpec.platform === "gcfv2" ? "v2" : "v1",
endpoint.id,
endpoint.platform === "gcfv2" ? "v2" : "v1",
trigger,
fnSpec.region,
endpoint.region,
availableMemoryMb,
fnSpec.runtime,
endpoint.runtime,
]
: [fnSpec.entryPoint, trigger, fnSpec.region, availableMemoryMb, fnSpec.runtime];
: [endpoint.id, trigger, endpoint.region, availableMemoryMb, endpoint.runtime];
table.push(entry);
}
logger.info(table.toString());
return functionList;
return endpointsList;
} catch (err) {
throw new FirebaseError("Failed to list functions", {
exit: 1,
Expand Down

0 comments on commit 6529c97

Please sign in to comment.